User exception sometimes wrapped with UndeclaredThrowableException

[mikasch]

I am using JBoss AS 4.2.3.GA (a short time ago migrated from 4.0.5) and a separate Tomcat.

There are cases where the JBoss is throwing a ValidationException in a Stateless Session Bean (EJB2) to show the user, that he has to fill the forms on the GUI correctly.

So now when working with the tomcat running on the same machine as the jboss it occurs very often - but not always! - that the thrown ValidationException is wrapped by an UndeclaredThrowableException. At the GUI a can save the same (wrongly filled) form 10 times and get about 7 times an UndeclaredThrowableException and about 3 times the unwrapped ValidationException.

I noticed that the communication between tomcat and jboss has changed between 4.0.5 and 4.2.3. Where in 4.0.5 it was RMI over port 4444 it is now Remoting Service over port 4446.

What puzzles me: how does it come that the ValidationException is sometimes wrapped and sometimes not by the UndeclaredThrowableException, and does it have anything to do with the migration to 4.2.3?

Additionally what is also interesting: I don't see this problem when the tomcat and the jboss run on different machines.

I saw many postings regarding the UndeclaredThrowableException, but nowhere I could read "sometimes".

First I thought that this could be a ClassLoader issue. I checked the jars that are deployed in the tomcat and in the jboss. There is only one jar file that contains the ValidationException.class and both jar files (one in jboss and on in tomcat) do have the same md5 check sum.

The ValidationException implements Serializable but not RemoteException.

Does someone has an idea, what is going wrong?

A few lines from the logs:

handleUnexpectedException() - got unexpected exception:
java.lang.reflect.UndeclaredThrowableException
        at $Proxy23.saveFeaturePackage(Unknown Source)
        at sun.reflect.GeneratedMethodAccessor1344.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
[...]

Caused by: [...].ValidationException

[brian.stansberry]

If you post the whole stack trace, that's more helpful.

[mikasch]

java.lang.reflect.UndeclaredThrowableException
    at $Proxy23.saveXyz(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor1344.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at [...].invoke([...])
    at $Proxy21.saveXyz(Unknown Source)
    at [...]
    at [...]
    at [...]
    at [...]
    at [...]
    at sun.reflect.GeneratedMethodAccessor1331.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at [...]
    at [...]
    at [...]
    at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:425)
    at org.apache.struts.faces.application.FacesRequestProcessor.processActionPerform(FacesRequestProcessor.java:241)
    at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:228)
    at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1913)
    at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:462)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at [...]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at [...]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at [...]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:174)
    at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
    at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
    at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    at java.lang.Thread.run(Thread.java:595)
Caused by: [...].common.validation.ValidationException, resultMessages=[...]
    at [...]checkXyz([...])
    at [...]
    at [...]
    at [...]
    at [...]
    at sun.reflect.GeneratedMethodAccessor854.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
    at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
    at org.jboss.ejb.plugins.SecurityProxyInterceptor.invoke(SecurityProxyInterceptor.java:190)
    at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
    at org.jboss.wsf.container.jboss42.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:59)
    at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
    at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
    at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
    at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350)
    at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
    at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
    at org.jboss.ejb.plugins.AbstractInterceptor.invoke(AbstractInterceptor.java:111)
    at [...].AccessControlInterceptor.invoke(AccessControlInterceptor.java:116)
    at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
    at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
    at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
    at org.jboss.ejb.Container.invoke(Container.java:960)
    at org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java:430)
    at org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java:103)
    at $Proxy223.doTransaction(Unknown Source)
    at [...]
    at [...]
    at [...]
    at [...].saveXyz([...])
    at sun.reflect.GeneratedMethodAccessor2476.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
    at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:237)
    at org.jboss.ejb.plugins.SecurityProxyInterceptor.invoke(SecurityProxyInterceptor.java:190)
    at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
    at org.jboss.wsf.container.jboss42.ServiceEndpointInterceptor.invoke(ServiceEndpointInterceptor.java:59)
    at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:169)
    at org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
    at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
    at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:450)
    at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:181)
    at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:168)
    at org.jboss.ejb.plugins.AbstractInterceptor.invoke(AbstractInterceptor.java:111)
    at [...].AccessControlInterceptor.invoke(AccessControlInterceptor.java:116)
    at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:205)
    at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:138)
    at org.jboss.ejb.SessionContainer.internalInvoke(SessionContainer.java:648)
    at org.jboss.ejb.Container.invoke(Container.java:960)
    at sun.reflect.GeneratedMethodAccessor564.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at org.jboss.invocation.unified.server.UnifiedInvoker.invoke(UnifiedInvoker.java:231)
    at sun.reflect.GeneratedMethodAccessor519.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)
    at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
    at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
    at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
    at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
    at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
    at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
    at $Proxy15.invoke(Unknown Source)
    at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:809)
    at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:608)
    at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:420)
    at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:173)
    at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:163)
    at org.jboss.remoting.Client.invoke(Client.java:1634)
    at org.jboss.remoting.Client.invoke(Client.java:548)
    at org.jboss.invocation.unified.interfaces.UnifiedInvokerProxy.invoke(UnifiedInvokerProxy.java:184)
    at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)
    at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)
    at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
    at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
    at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:112)
    at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
    ... 46 more

Unfortunately I had to censor sth.

[adrian.brock]

[e-mail] wrote:

java.lang.reflect.UndeclaredThrowableException
    at $Proxy23.saveXyz(Unknown Source)
    at sun.reflect.GeneratedMethodAccessor1344.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:585)

<snip/>

Caused by: [...].common.validation.ValidationException, resultMessages=[...]
    at [...]checkXyz([...])

<snip/>

It's getting wrapped because your saveXyz() method isn't declared to throw the ValidationException on the interface.
The "sometimes" will actually be "some places" in that some methods will have declared that they throw that exception.

The javadoc explains it better:

http://java.sun.com/j2se/1.4.2/docs/api/java/lang/reflect/UndeclaredThrowableException.html

[brian.stansberry]

1) Is ValidationException a checked exception?

2) Is it or a superclass declared in the throws clause of saveXyz() ?

3) Does your application have any configuration to allow EJB calls to be invoked on remote machines, even if the EJB is deployed locally? (By default JBoss will route the call locally.)

A vague theory I have is this only happens when the call is routed locally, and your app is routing ~ 70% of requests remotely. If you have a 4 node cluster perhaps ~ 3/4 calls go remote...

[brian.stansberry]

Oh, I missed Adrian's response. Yes, for sure if it's different methods that are being invoked when the exception gets wrapped or doesn't, then the likely explanation is differences in method signatures.

[mikasch]

Thanks to both of you, Adrian and Brian for your responses.

Adrian:

This exception is also "sometimes" thrown when calling exactly the same EJB method a number of times and is never thrown when the exact same method is called from a tomcat that runs on a distant node. The "sometimes only occurs, when the tomcat on the same machine as the JBoss runs on does the call.

Of course I read the API for the UndeclaredThrowableException and checked out if the involved methods declare the ValidationException.

Brian:

1) Is ValidationException a checked exception?

2) Is it or a superclass declared in the throws clause of saveXyz() ?

3) Does your application have any configuration to allow EJB calls to be invoked on remote machines, even if the EJB is deployed locally? (By default JBoss will route the call locally.)

A vague theory I have is this only happens when the call is routed locally, and your app is routing ~ 70% of requests remotely. If you have a 4 node cluster perhaps ~ 3/4 calls go remote...

I can answer all your questions with yes.

We are working with a separate Tomcat, so there are two Java VMs, thus all calls are remote calls. The JBoss is configured as single node server (no HA).

Here comes the declaration of the method in the actual Stateless Session Bean implementation ("XyzSessionFacadeBean.java"):

     [...]

    /*
     * @ejb.interface-method view-type = "both"
     * @ejb.transaction type = "Never"
     * @ejb.permission role-name = "[...]"
     */
    public Long saveXyz(Xyz obj) throws ProcessingException, ValidationException {

     [...]

and the appropriate generated (xdoclet) remote interface ("XyzSessionFacade.java"):

     [...]

   public java.lang.Long saveXyz( Xyz obj )
      throws com.[...].exceptions.ProcessingException, com.[...].validation.ValidationException, java.rmi.RemoteException;

     [...]

The ValidationException extends a superclass that itself extends Exception and implements Serializable.

[adrian.brock]

[e-mail] wrote:

Adrian:

This exception is also "sometimes" thrown when calling exactly the same EJB method a number of times and is never thrown when the exact same method is called from a tomcat that runs on a distant node. The "sometimes only occurs, when the tomcat on the same machine as the JBoss runs on does the call.

Of course I read the API for the UndeclaredThrowableException and checked out if the involved methods declare the ValidationException.

But you've already posted two stracktraces that show the UndeclaredThrowableException is raised in $ProxyNNN.saveXyz()

This is code generated by the JDK to handle cases when you throw exceptions that aren't declared on the interface.

Some obvious things that could go wrong:

1) You've declared the wrong interface as the local interface - for remote tomcat you would use the remote interface

which sounds like it is correct.

2) You're not using the version of the interface you think you are using, e.g. you have a copy of the class deployed somewhere else

that is different and you are not using classloader isolation, or you included the wrong .class in the jar.

Try running:

javap -c -classpath my.jar com.acme.MyClass

[mikasch]

Adrian,

I still have this problem.

I tried to change some settings in the start script of the jboss.

Currently the jboss is started with the following options:

-Dsolid_appinfo=jboss_ooo \

-Dprogram.name=run_ooo.sh \

-server -Xms1024m -Xmx1024m -XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=256m \

-Djava.endorsed.dirs=/[...]/jboss/lib/endorsed \

-classpath /[...]/jboss/bin/run.jar:/usr/java/lib/tools.jar \

org.jboss.Main -b hostname_ext -c ooo

before that the jboss was started that way:

-Dsolid_appinfo=jboss_ooo \

-Dprogram.name=run_ooo.sh \

-server  -Xms1024m -Xmx1024m -XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=256m  \

-Djava.endorsed.dirs=/[...]/jboss/lib/endorsed \

-Djava.rmi.server.hostname=hostname_ext  \
-Djava.rmi.useLocalHostname=false \
-classpath  /[...]/jboss/bin/run.jar:/usr/java/lib/tools.jar \

org.jboss.Main  -b hostname_ext -c ooo

By removing the red lines I observed that the UndeclaredThrowableException comes less frequent (but maybe this is just accidentally).

I reproduced this issue and got two stack traces for the very same use case. The traces are exactly identical with the only difference that the "good" one has just got the wanted ValidationException pure and the other one wraps the very same ValidationException in an UndeclaredThrowableException. (I don't post the second stack trace here because it really doesn't differ from the one I already posted, just look at caused by and that's the stack trace...) - By the way: the stack trace is taken from the Tomcat log not jboss log.

Regarding your suggestions:

Some obvious things that could go wrong:

1) You've declared the wrong interface as the local interface - for remote tomcat you would use the remote interface

which sounds like it is correct.

2) You're not using the version of the interface you think you are using, e.g. you have a copy of the class deployed somewhere else

that is different and you are not using classloader isolation, or you included the wrong .class in the jar.

Try running:

javap -c -classpath my.jar com.acme.MyClass

1. If this would be the case then every call / exception would fail / be wrapped with the UndeclaredThrowableException.
2. I already checked the ValidationException - everything was fine. So which classes shall I check now? All classes that are involved in the call stack? It might be a Tomcat class that interferes with a jbossall-client class. What do you think?

I'll go on and try to check up the application classes and interfaces for consistency.

[mikasch]

In the meantime the behavior has changed (I don't know why/how):

There  is no mixed behavior during the runtime of the tomcat. Means that once  the tomcat has started, there is always the UndeclaredThrowableException  which seems to wrap all our application's exceptions or the  UndeclaredThrowableException never appears.

The ratio  seems to be at about 3 to 1: If you start the tomcat 4 times then 3  times the tomcat wraps all our exceptions with the  UndeclaredThrowableException.

I now need some ideas,  how to track which classes from which jars are loaded by the tomcat to  find out if this is really a ClassLoader/incompatible classes issue.  Does someone have a suggestion how to track this issue?

Regards

Fares

[mikasch]

Now I found something new which is somewhat disturbing.

I added the following code to the tomcat:

try

{

     remote.saveXyz(xyz);

}

catch( UndeclaredThrowableException e1 )
{
     Throwable t = e1.getUndeclaredThrowable();
     if(t instanceof ValidationException)
     {
          ValidationException e2 = (ValidationException)t;
          log.info( "A ValidationException: ", e2 );
          e2.getResultMessages().add( new ResultMessage( "standardProcessingError.internalError" ) );
          throw new ProcessingException(e2.getResultMessages(), e2);
     }
     else if(t instanceof ProcessingException)
     {
          log.info( "A ProcessingException: ", t );
          throw (ProcessingException)t;
     }
     else
     {
          log.info( "Neigther a Validation-  nor a ProcessingException: ", t );
          throw e1;
     }      

}

Guess what - I got the else branch:

Neigther a Validation-  nor a ProcessingException: [...]ValidationException [...]

So the ValidationException is not a ValidationException, although the fully qualified class name (including the package) is exactly the same.

I am 99% sure that there are not different versions of the same class in the system. The class may be duplicate in different jars but this should not be a problem.

So what is this? Any ideas, what to do?

Regards

Fares

[mikasch]

I got some progress and I am now sure that it is a ClassLoading issue.

In the case where the ValidationException is wrapped by the UndeclaredThrowableException, the WebappClassLoader instance (tomcat) of the current thread and the instance of the WebappClassLoader of the caught (and wrapped) ValidationException are different. As a result the ValidationException that is caught is not the same as the one that is expected and thus violates the interface => UndeclaredThrowableException.
In the following code the "instanceof ValidationException" returns false:
    […]
            key = saveSomething();
        }
        catch (ValidationException e)
        {
    […]
        }
        catch( UndeclaredThrowableException e1 )
        {
            Throwable t = e1.getUndeclaredThrowable();
           
            if(t instanceof ValidationException)
            {
        […]
            }
            else
            {
                log.info( "Not a ValidationException: ", t );
                log.info( "  thrown Exception class: " + t.getClass() + ": " + getUniqueID( t ) );
                log.info( "expected Exception class: " + ValidationException.class + ": " + getUniqueID( new ValidationException() ) );
                throw e1;
            }
        }

It seems that during unmarshalling of the ValidationException it is not predictable which ValidationException (identified by its ClassLoader) is constructed. It seems that the WebApp may get a ValidationException that doesn't fit to its own ClassLoader but to a ClassLoader of another WebApp.
Is this a bug of the jboss? It is the jboss that does the unmarshalling and I mean it should not happen that the unmarshaller uses a ClassLoader for the ValidationException that does not fit to the ClassLoader that belongs to the WebApp that invoked the RMI call?