I wanted to try bumping this up again. I'm really having trouble with this and I can't find any good JBoss Portal docs on how to create Custom Portlet Modes.
I've tried calling request.getPortalContext().getSupportedPortletModes(), but it only returns View, Edit and Help even though I should have a couple more modes set up in the portlet.xml and the Portlet class.
As per the spec, porlet modes can only be defined by the portal vendor.
JBoss Portal could decide to add a specific porlet mode (And all portlets using that special mode would not work in any other portal) but so far we don't. We only support the modes that are part of the specification VIEW, EDIT, HELP
Whatever you specified in portlet.xml are custom modes that you want to use, but since those modes don't exist, they are not used and JBoss Portal as it follows the spec ignores them.
See section 8.4 of the spec.