We just bumped into this problem recently. Searching this forum doesn't seem to work real well, so I thougtht I'd post my workaround in case it helps someone else... All you have to do is re-render the modalPanel itself, as it will be closed when re-rendered. If you are not already calling an AJAX action when you want to close it, you can add an ajax action that does nothing and/or calls a null method. Alternatively, you can upgrade to 3.3.1, as the problem is reported resolved in that release.
Check out this thread.
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=4243440
Matt