In any way if you need perform submits from modal - it should has its own form and be outside the external one. (read doc on it)
So moving the panel outside the main form and insert form in panel body it will resolve both problems.
which form should I use then - a4j:form or h:form?
My problem was that when I moved it to another form I couldn't redisplay components that were outside the form of modalPanel. Do you have any ideas how to bypass that?