This involves the use of a <rich:modalPanel> as the popUp window. In the modalPanel, you can embed a h:form with any other JSF/RF components you want in there.
So you should do a simple POC, with a main JSF form and a button that displays the modalPanel. Embed a form and button in the modalPanel which executes an action method on your backing bean (JavaBean or SFSB) that sets an instance variable like a String and then hides the modalPanel. The trick will be to reRender the main JSF form so that the String value displays now.
There is good coverage on how to do this in Practical RF book.