That can happen if browser has reached the limit for simultaneous HTTP connections opened. You can try to change settings for IE by following this article: http://www.pctools.com/guides/registry/detail/536/ and check whether that is the source of the problem.
I use chrome for daily browsing and modal panel works ok when I explore livedemo site. Does this work: http://livedemo.exadel.com/richfaces-demo/richfaces/modalPanel.jsf?c=modalPanel&tab=usage for you?
Thanks nbelaevski for the tip about max connections, that was the solution.
I still can't get chrome or safari to work but it looks more like a problem with my ajax forms and buttons.