similar facelet works for me well
you should have both popup and "content" in separate forms
I have separate forms for my popup and page content, nevertheless it's not working for me.
Code for page is like:
// some content
I would try to execute @form in popup's a4j:commandButton
e.g. delete execute attribute of a4j:commandButton since @form is the default
try using this
onkeydown='if(event.keyCode == 13) document.getElementById("searchPopupForm:searchButton").click();'
it will work.