10 Replies Latest reply on May 20, 2009 4:23 PM by zradick

    JBoss 5.1.0.CR1 classloader isolation issue

    zradick

      We are migrating a host of services from JBoss 4.0.5 to JBoss 5.1.0.CR1 (we attempted to use 5.0.1.GA but encountered a bug with using double values as parameters to EJB3 Beans which is fixed in 5.1.0). We have had to make a number of minor modifications to our services (mostly XML configuration changes), but we have migrated a good number of services already. However, we have just encountered an issue that we have not been able to find a good solution for yet.

      The gist of the issue is that it appears that the classloader isolation for one of our .ear files is not fully working. We are using JMS via JBoss Messaging to deliver messages on a Topic to interested applications. One of these recipient applications lives within the same JBoss server as the service that publishes the messages (it very well might not in our production environments, but for development purposes it does). The recipient application uses a set of internally developed libraries to connect and receive messages from the topic (which worked properly in 4.0.5 and continues to work properly for my test clients that operate outside of JBoss). When I run the client process (in JBoss5.1.0) through the debugger I can see that the payload of the ObjectMessage received is the same class, but has come from the ClassLoader of the publishers .ear file.

      The publisher has the following XML in the jboss-app.xml file:

      <?xml version="1.0"?>
      <!DOCTYPE jboss-app
       PUBLIC "-//JBoss//DTD Java EE Application 5.0//EN"
       "http://www.jboss.org/j2ee/dtd/jboss-app_5_0.dtd">
      <jboss-app>
       <loader-repository>
       StargateServer:loader=StargateServer.ear
       <loader-repository-config>
       java2ParentDelegation=true
       </loader-repository-config>
       </loader-repository>
      </jboss-app>
      


      I have tried it without the java2ParentDelegation and with the delegation set to false, but no combination changes this behavior. The client in question lives within a .war file deployed within a .ear file. In early testing I had isolated the .war file to the point that it could connect to a JBoss 4.0.5 server using the JBoss 4.0.5 client libraries (as none of our services were yet ported to JBoss5), but even that does not seem to prevent the pollution from occurring (and if I understand correctly the issue is that the publisher is somehow exporting the class despite having it's own class loader domain and isolation of the recipient is a separate issue).

      The exact same set of XML in the jboss-app.xml files (DOCTYPE aside)used to work under 4.0.5 and I cannot find anything indicating how I should modify these files for JBoss5 to resolve the issue. I have seen reference to a "jboss-classloading.xml" file and a JIRA indicating that it should have documentation, but I'm not sure if it is something I need to use (and less sure how to use it if I should).

      Of particular interest, the client initially synchronizes with the remote service (within the same JBoss5 AS) at the point of subscribing to the topic and receives a generified List of the exact same Objects that are used as the payload of the ObjectMessages. This List of Objects is obtained via a Stateless EJB exposed by the publisher and is not polluted (the Object class was loaded using the ClassLoader of the client). This would seem to imply that the issue is related to the fact that the Objects are being exchanged via JMS (JBoss Messaging). Is there configuration in the destination-service that controls this?

      This seems somewhat similar to a couple of JIRA tickets (e.g. https://jira.jboss.org/jira/browse/JBMESSAGING-1277), but they have been closed and the fix versions cited are several back from the messaging implementation included in JBoss 5.1.0.CR1 (1.4.3).

      We are using isolation because we don't have a guarantee about our deployment environments and we may need to have multiple client applications connecting to different versions of our services within the same JBoss instance, however we do share common database drivers and the JBoss libraries.

      Any advice would be appreciated.
      Thanks in advance,
      --Zack

        • 1. Re: JBoss 5.1.0.CR1 classloader isolation issue
          jaikiran

          Long one! :)

          Can you please post the exception stacktrace? Also let us know where in the client application, the offending class shown in the exception resides. We can start discussing it from there.

          • 2. Re: JBoss 5.1.0.CR1 classloader isolation issue
            zradick

            I don't actually get a stacktrace because I test the Object before attempting to cast it:

             public void onMessage ( final Message message )
             {
             if( message instanceof ObjectMessage )
             {
             ObjectMessage objectMessage = (ObjectMessage)message;
             Serializable payload;
             try
             {
             payload = objectMessage.getObject();
             if( payload instanceof TypedData )
             {
             m_incomingQueue.add( (TypedData)payload );
             }
             }
             catch( JMSException e )
             {
             m_handler.onConnectionException( new ConnectionException( e ) );
             }
             }
             }
            


            The issue is that the payload is a TypedData Object, but loaded using the ClassLoader of the topic producer, not the subscriber. In this particular instance my receiver is instantiated inside of a .war file (nested in a .ear file), but it appears to also happen when the receiver is loaded outside of the .war as well (e.g. instantiated from a Service Bean in the EJB context).

            After modifying the code to naively cast the payload to the TypedData class without checking for type I receive the following:
            13:20:14,624 ERROR [ClientConsumer] RuntimeException was thrown from onMessage, 20345880406507537 will be redelivered
            java.lang.ClassCastException: crc.stargate.common.TypedData cannot be cast to crc.stargate.common.TypedData
             at crc.stargate.common.ReceiverFactory$TypedDataConnectionImpl.onMessage(ReceiverFactory.java:167)
             at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:229)
             at org.jboss.jms.client.container.ClientConsumer$ListenerRunner.run(ClientConsumer.java:1043)
             at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:120)
             at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
             at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
             at java.lang.Thread.run(Thread.java:619)
            


            Let me know if there's anything else that would be helpful.
            Thanks,
            --Zack

            • 3. Re: JBoss 5.1.0.CR1 classloader isolation issue
              zradick

              Also, within the same client application, the following line DOES work and returns TypedData Objects loaded via the client's ClassLoader and remoted from an EJB within the .ear file that is also responsible for publishing the topic:

               final List<TypedData> currentData = server.getAllCurrentData( filters );
              


              Currently in the JMX console under the jboss.classloader view both the classloader for the .ear providing the topic and the .war that subscribes to it list their PARENT_POLICY_NAME as AFTER_BUT_JAVA_BEFORE. I have also had them both set to BEFORE and combinations of the two.

              Both instances report their own classloader when invoking the "findClassLoaderForClass" operation with the class that is being polluted (TypedData). Attempting to find the TypedData class under the "DefaultDomain" results in no classloader being found (as expected).

              • 4. Re: JBoss 5.1.0.CR1 classloader isolation issue
                zradick

                In further testing, we installed the Topic producer on a remote JBoss5 server and verified that the client (in a different JBoss5) would receive the message payloads correctly. This works as expected.

                We then put a copy of the Topic producer on the same JBoss5 as the client, but continued to have the client subscribe to the producer on the remote JBoss5 instance. The results of this seem deeply strange to me... The initial connection works (as before) and the client proceeds to receive 1 or 2 messages (messages are produced by 2 separate threads within the application system roughly every 60 seconds a piece) with payloads that work (this seems pretty consistent across runs), before the classloader pollution seems to take effect and all further messages received fail. This is different than the behavior that occurs when the client connects to the local Topic, as in that case all of the messages are polluted.

                During one call we received an error on the client:

                2009-05-08 15:13:28,484 ERROR [org.jboss.jms.client.container.ClientConsumer] (Thread-24) Failed to deliver message
                org.jboss.jms.exception.MessagingJMSException: Failed to invoke
                 at org.jboss.jms.client.delegate.DelegateSupport.handleThrowable(DelegateSupport.java:271)
                 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:205)
                 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:160)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate.org$jboss$jms$client$delegate$ClientSessionDelegate$acknowledgeDelivery$aop(ClientSessionDelegate.java:186)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate$acknowledgeDelivery_N5825751487881460811.invokeTarget(ClientSessionDelegate$acknowledgeDelivery_N5825751487881460811.java)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
                 at org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:92)
                 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
                 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate.acknowledgeDelivery(ClientSessionDelegate.java)
                 at org.jboss.jms.client.container.SessionAspect.ackDelivery(SessionAspect.java:873)
                 at org.jboss.jms.client.container.SessionAspect.handlePostDeliver(SessionAspect.java:352)
                 at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect_z_handlePostDeliver_9861267.invoke(SessionAspect_z_handlePostDeliver_9861267.java)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
                 at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
                 at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:86)
                 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
                 at org.jboss.jms.client.delegate.ClientSessionDelegate.postDeliver(ClientSessionDelegate.java)
                 at org.jboss.jms.client.container.ClientConsumer.callOnMessage(ClientConsumer.java:253)
                 at org.jboss.jms.client.container.ClientConsumer$ListenerRunner.run(ClientConsumer.java:1043)
                 at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:120)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
                 at java.lang.Thread.run(Thread.java:619)
                Caused by: java.lang.IllegalStateException: Cannot find object in dispatcher with id xe-3vnwfhuf-1-8fprehuf-g8atyv-j133o4c5
                 at org.jboss.jms.wireformat.SessionAcknowledgeDeliveryRequest.serverInvoke(SessionAcknowledgeDeliveryRequest.java:75)
                 at org.jboss.jms.server.remoting.JMSServerInvocationHandler.invoke(JMSServerInvocationHandler.java:143)
                 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
                 at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
                 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
                 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:524)
                 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
                 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:211)
                 at org.jboss.remoting.Client.invoke(Client.java:1724)
                 at org.jboss.remoting.Client.invoke(Client.java:629)
                 at org.jboss.remoting.Client.invoke(Client.java:617)
                 at org.jboss.jms.client.delegate.DelegateSupport.doInvoke(DelegateSupport.java:189)
                 ... 25 more
                

                After the error the client re-subscribed to the topic and received three messages that were apparently waiting for it (all of which worked -- they loaded the TypedData class with the local copy), and then all subsequent messages failed.

                I am rather confused how it would use the right classloader initially (and also in one case after a failure), but then proceed to use a different one after that...

                Any ideas would be helpful.
                Thanks,
                --Zack

                • 5. Re: JBoss 5.1.0.CR1 classloader isolation issue
                  zradick

                  We also tried setting the following properties (to no avail):

                  server/default/deployers/ear-deployer-jboss-beans.xml
                  true
                  true

                  server/default/deployers/ejb-deployer-jboss-beans.xml
                  true

                  server/default/conf/jboss-service.xml
                  true

                  • 6. Re: JBoss 5.1.0.CR1 classloader isolation issue
                    zradick

                    We also tried setting the following properties (to no avail):

                    server/default/deployers/ear-deployer-jboss-beans.xml
                    <property name="callByValue">true</property>
                    <property name="isolated">true</property>
                    
                    server/default/deployers/ejb-deployer-jboss-beans.xml
                    <property name="callByValue">true</property>
                    
                    server/default/conf/jboss-service.xml
                    <attribute name="CallByValue">true</attribute>


                    • 7. Re: JBoss 5.1.0.CR1 classloader isolation issue
                      zradick

                      After more searching and testing we came upon a Thread that suggested that the problem was related to having the onMessage() method within the WebContainer (in the application we are currently porting the .war file creates a singleton via the ServletContextListener which receives the JMS message notifications).
                      http://www.jboss.org/index.html?module=bb&op=viewtopic&t=144634

                      To test this out, we extracted a simple test case that creates a JMS subscriber and registers itself as a MessageListener on the topic. This MessageListener is a simple service created with the @Service annotation and it also experiences the same class cast exceptions when running within the same JBoss5 instance as the Topic to which it subscribes.

                      I realize that we could use MDB's for this process and have them pass the data they receive into an injected service/EJB, etc. However, not all of our client applications run in JBoss (some run in Tomcat and others are standalone Java processes) and we share client libraries across them. Ideally we would prefer not to have to do something different for our JBoss applications (especially since it works if the clients are on different JBoss servers already).

                      • 8. Re: JBoss 5.1.0.CR1 classloader isolation issue
                        zradick

                        Based on the aforementioned forum topic we have a hack work-around that seems to work. I feel funny about mucking around with the ClassLoaders directly, but I suspect that the JBossMQ previously did something to this effect anyway since the old code worked properly with JBoss 4.0.5 using JBossMQ.

                         public void onMessage ( final Message message )
                         {
                         if( message instanceof ObjectMessage )
                         {
                         // todo: remove this classloader hack if it becomes un-necessary
                         final ClassLoader original = Thread.currentThread().getContextClassLoader();
                         try
                         {
                         Thread.currentThread().setContextClassLoader( this.getClass().getClassLoader() );
                         final ObjectMessage objectMessage = (ObjectMessage)message;
                         final Serializable payload = objectMessage.getObject();
                         if( payload instanceof TypedData )
                         {
                         m_incomingQueue.add( (TypedData)payload );
                         }
                         }
                         catch( JMSException e )
                         {
                         m_handler.onConnectionException( new ConnectionException( e ) );
                         }
                         finally
                         {
                         Thread.currentThread().setContextClassLoader(original);
                         }
                         }
                         }
                        


                        Has anyone else had to do something like this? If so did you encounter any gotchas that we should be aware of? It looks safe enough, but it seems hackish and has a slightly unpleasant code smell about it...

                        Thanks,
                        --Zack

                        • 9. Re: JBoss 5.1.0.CR1 classloader isolation issue
                          timfox

                          Guys-

                          Please post potential JBoss Messaging issues on the JBM forum.

                          I wasn't aware of this thread until some one pointed me in this direction, we don't monitor this forum.

                          • 10. Re: JBoss 5.1.0.CR1 classloader isolation issue
                            zradick

                            Sorry Tim, I was not initially aware that this was a Messaging issue, I assumed it was a classloader isolation problem. However, I did post a link to this thread in the JBoss Messaging forums (in the Thread I mentioned above) after determining what the issue really was (although there was no further response to it there). Please feel free to move this thread to the Messaging forums if it would be helpful.
                            Thanks,
                            --Zack