Do you have a form inside your modalPanel? I meant that if you have only one form on the page you are probably getting validation error anyway(even if modalPanle is hidden). Add rich:messages somewhere on the page(but not in modalPanel) and see if there are any error messages when you select data in the seconf time.
A lot depends on how you "display" the modalPanel. I've had the best results when I conditionally render the modalPanel (using the rendered attribute) and wrap the modalPanel inside an <a4j:outputPanel that is always rendered. I also usually set my modalPanels to showWhenRendered="true". Code whatever action you call that would potentially display the modalPanel to set the appropriate rendered condition and reRender the outer panel.