In your case:
a) the modalPanel should have its own h:form
b) modalPanel should NOT be inside other h:form.
thank u I guess thats how I will need to implement..
so now I have the form inside the rich:pannel but any validation error is being displayed on the back layer on not the rich:modelpannel