13 Replies Latest reply on Jun 9, 2003 12:19 PM by Adrian Brock

    Classloader Problems

    Darran Lofthouse Master

      Hi,

      I am currently having a problem redeploying the application the application that I am developing, between each deployment I am having to restart JBoss to clear the class loaders.

      My application consists of a couple of jars containing EJBs, some sars containing services a rar, the required configuration files and a couple of jars containing classes used by some of the other components. Everything is packages up into a single ear file (Darran.ear).

      After deploying the application I run a client application which puts a number of instances of ClassX into a hashtable and passes the hashtable into a session bean that in turn passes it onto a resource adapter. The resource adapter then checks using instanceof that each object contained in the hashtable is an instance of ClassX.

      When I run the application after the first deployment everything is fine and the ClassX instances passed in are correctly identified.

      Next I remove the Darran.ear from the deploy folder, I then recompile the application and deploy the newly generated 'Darran.ear'.

      Now when I run the application that passes in instances of ClassX the comparison fails.


      Just before performing the comparison I have output the results of obj.getClass().getClassLoader().toString() where obj is the instance being tested.

      - org.jboss.mx.loading.UnifiedClassLoader3@a3f178{
      url=file:/C:/jboss-3.2.1_tomcat-4.1.24/server/all/tmp/deploy/
      server/all/deploy/Darran.ear/53.Darran.ear ,addedOrder=35}


      I have also output the results of new ClassX().getClass().getClassLoader().toString()

      - org.jboss.mx.loading.UnifiedClassLoader3@837645{
      url=file:/C:/jboss-3.2.1_tomcat-4.1.24/server/all/tmp/deploy/
      server/all/deploy/Darran.ear/56.Darran.ear ,addedOrder=38}


      Even though I completely removed the first instance of the application before deploying the second it appears that the first one is being cached and used still.

      I know that the client application is definately using the correct version of all classes as the build script I am using replaces the jars used by the client at the same time as createing the ear.

      Has anyone else seen anything like this before?

      Thanks in advance for any help.

      --
      Darran.

        • 1. Re: Classloader Problems
          Adrian Brock Master

          Where is the client?

          Show the getClassLoader() output for the passed
          objects, rather than new ClassX()

          Regards,
          Adrian

          • 2. Re: Classloader Problems
            Darran Lofthouse Master

            Thanks for the reply.

            The client (although running on the same machine) is remote to the JBoss server i.e. It is running in it's own JVM.

            The first of the two getClassLoader() outputs was from getting the ClassX instance from the Hashtable that was passed in.

            The second one that was from the new ClassX() was to show which class loader was being used to create a new instances of ClassX.

            • 3. Re: Classloader Problems
              Adrian Brock Master

              Can you explain the deployment
              more clearly, is everything redeployed?

              Which version of jboss are you using?

              Regards,
              Adrian

              • 4. Re: Classloader Problems
                Darran Lofthouse Master

                Hi,

                I am using jboss-3.2.1_tomcat-4.1.24.

                Originally I was deploying each component individually when I first noticed the problems so I thought that might have been the cause of the problem.

                Now every component of my application is contained within a single ear.

                When my application is not deployed the only components running on JBoss are those that are part of the default 'all' configuration.

                Between deployments I delete the ear file and wait for the JBoss console to say that it has been undeployed before deploying the new ear file.

                • 5. Re: Classloader Problems
                  Darran Lofthouse Master

                  I have now had a look at putting together a simple test case to demonstrate this. I have implemented a simple serializable data transfer object and a session bean that with a method that takes an Object as it's parameter.

                  My client then passes the data transfer object into the session bean and the bean prints the class loader details and tests the object by using instanceof and casting. This code works perfectly even after redeployment.

                  I think that this means that something in my ear is keeping the classloader used to deploy the ear around even after the ear has been undeployed. Is there any way that I can find out what is keeping the class loader active?

                  I have seen other posts in this forum from people saying that they think JBoss has got memory leaks when performing lots of redeployments, maybe they are experiencing the same problems as me - something in their deployment keeps the class loader active so it is not removed when their application is redeployed.

                  • 6. Re: Classloader Problems
                    Adrian Brock Master

                    The classloader will be kept active if you
                    are holding onto a class. i.e. you are holding
                    an Object that is from a class that is loaded
                    by the classloader.

                    This should not cause your classcast,
                    unless the old Object is reachable from your
                    new deployment.
                    It is a memory leak. Are you using something
                    like Struts. I remember somebody saying that
                    had problems in 1.0

                    You will need a tool lets you examine the heap,
                    or just add bits of the old application until
                    you hit the problem.

                    Regards,
                    Adrian

                    • 7. Re: Classloader Problems
                      Darran Lofthouse Master

                      I have found a profiling tool that I have been using to check what objects are in memory.

                      I did find a that one of my objects was still bound to JNDI after undeployment so I have corrected it and now the object is correctly removed.

                      I am still however left with a lot of other objects created from the ear in memory. The objects that are left are referenced statically. I do not have any control over these objects as one set of objects are from the JDBC driver and the other set are from classes generated by Castor.

                      Does the ClassLoader check to see that only static references remain or are these references going to stay until the next time JBoss is restarted?

                      The instanceof failure still remains so I am assuming that it is these objects keeping the ClassLoader active.

                      • 8. Re: Classloader Problems
                        Adrian Brock Master

                        It is the job of the undeployer to remove
                        these static references.

                        How are you deploying the jdbc driver?
                        That maybe a bug in JBoss/JCA or it might be a
                        problem with the JDBC driver.

                        Castor I don't know much about.

                        Regards,
                        Adrian

                        • 9. Re: Classloader Problems
                          Darran Lofthouse Master

                          The JDBC driver (For MySQL) is being deployed by
                          - putting the jar containing the driver inside the ear
                          - putting a java reference to the jar in the application.xml
                          - putting the mysql-ds.xml in the ear and putting a service declaration in the jboss-app.xml file

                          I will try deploying the MySQL driver and mysql-ds.xml files on their own and undeploy them and see if they are left behind then.

                          The Castor classes are nothing special, the only thing that makes them Castor classes is because Castor generated them - apart from that they are standard java classes.

                          Basically it is a class with a private constructor that declares static references to itself and statically creates the instances that are referred to by the static variables.

                          The purpose of it is to create an Enumeration that is a lot more type safe than using int values.


                          • 10. Re: Classloader Problems
                            Adrian Brock Master

                            Does the class with the static references get
                            redeployed? Or are the static references cleared?

                            Regards,
                            Adrian

                            • 11. Re: Classloader Problems
                              Darran Lofthouse Master

                              The classes with the static references are deployed within the ear and undeployed and redeployed along with the ear.

                              The static references are not cleared and there is no way that they can be cleared as they are declared final.

                              However I am not sure if it actually the static references that are the cause of the problem, I added a class with some static final properties to my simple test projects and it deployed and undeployed with no problems. I watched the instances being garbage collected without any problems from the profiling tool that I have.

                              I am now going to try wrapping my simple app in an ear to see if that affects it in any way.

                              • 12. Re: Classloader Problems
                                Darran Lofthouse Master

                                Ok, I have investigate this problem quite a bit more.

                                First the static references to classes contained within the ear / jar.

                                As I said earlier using the profiling tool I can see these references being cleared automatically when there is nothing else keeping the class loader active, however they only seem to get cleared at the time the ear is redeployed. At the moment it looks as if they would stay in memory permenantly if I didn't deploy the application again - this probably needs some further testing to make sure that garbage collection does run during the time that I am waiting for them to be cleared.

                                The JDBC driver if it is deployed by putting the driver in lib and the mysql-ds.xml in deploy or deployed by putting both in the ear seems to stay in memory permenantly after undeployment - again this needs some further testing to make sure that the garbage collector has had long enough to run.

                                I have checked how many UnifiedClassLoader3 instances are in memory and they do appear to be getting garbage collected (As long as there are no JDBC drivers in memory).

                                I have now tracked down the cause of my problem to one of my MBeans, it is a clustered MBean that registers itself as an RPC handler for the cluster that it is in.

                                My client passes in ClassX to a session bean, the session bean passes ClassX to the MBean that in turn calls callAsynchMethodOnCluster to distribute the ClassX instance to each node in the cluster.

                                If the method being called by callAsynchMethodOnCluster is declared to take an object of type ClassX everything works fine even after redeployment. If the method being called by callAsynchMethodOnCluster is declared to take an object of type Object it uses an old class loader to identify the object. My ClassX instances need to be passed around as Objects as they are contained within a Hashtable.

                                When running my test code the class loader is eventually garbage collected, once this happens the call to callAsynchMethodOnCluster results in the following message being logged

                                13:44:57,617 WARN [DefaultPartition] RpcProtocol.Handle(): java.lang.NullPointerException


                                I will post a message to the JCA forum now to discuss the unloading of JDBC drivers as that is probably a more appropiate place to discuss that than here.

                                I will post a message to the clustering formum to discuss how the appropriate class loader is selected for callAsynchMethodOnCluster.

                                Which forum is the appropriate place to discuss when static references should be cleared after undeployment?

                                • 13. Re: Classloader Problems
                                  Adrian Brock Master

                                  The clustering forum.

                                  Regards,
                                  Adrian