Haven't solved the problem yet, but I have more data. I get the same behavior in chrome browser, so it doesn't seem like a browser problem.
The popup is called from inside a togglepanel that implements a wizard. This is a fairly complicated input wizard with four pages of inputs. Most of the inputs have associated input validations.
I've noticed that if I comment out or otherwise don't render some other parts of the form, then the popup works correctly. I narrowed it down to a set of about eight inputs.
It also works if I change the togglepanel to client mode, but then the wizard doesn't work.
However, I still can't see any association between those inputs and popup that should affect the popup.
The fact that the correct dynamically-rendered content for the popup panel is coming back in the ajax request, but not making it into the page still seems like a bug in the components.