The JIRA answer pretty much answers it. A class loaded twice in different class loaders will appear different to the VM. In your case what most likely happens is that your class loads Validator from your application library while the identity portlet loads it from MyFaces, thus causing the ClassCastException. Try to remove the JSF libs from your application.
JBoss Portal identity application is using MyFaces
MyFaces jar libraries are in
portal-identity.sar\portal-identity.war\WEB-INF\lib2 (sic!) folder
this is where portal-core-identity-ui-lib.jar files is which contains build in CaptachValidator class.
I placed my jar file into jboss-portal.sar\portal-identity.sar\lib\ which makes it available to the whole identity application.
So does the anwser mean I need to put my jar file (which contains just validator class) into portal-identity.sar\portal-identity.war\WEB-INF\lib2?
to everybody: the problem is solved
I put the jar file containing MyValidator.class into lib2 folder, and it worked. Apparently, myfaces uses some tricks to load jar files from "lib2" (not "lib") folder. You were right if I put my class in lib2 folder the same Class Loader that loads myfaces libs loads my validator too and makes it castable to Validator interface.