8 Replies Latest reply on Sep 2, 2002 10:07 AM by sanderbos

    JBoss classloaders vs. the default JVM classloader

    sanderbos

      Hi there,

      [First off, my apologies for crossposting. I posted this same message in the performance tuning forum yesterday, but now I see it has been 10 days since anybody got an answer in that group. I hope nobody gets upset that I now try my luck here, since it is also a configuration issue.]


      Does anybody know about the JBoss classloaders being significantly slower than the default JVM classloaders?

      I am using JBoss 3.0.0, with Sun JDK 1.3.1 on both Linux and Windows 2000. I used Xerces 2.0.1 for my testcase, but I suspect the same problem occurs for any Xerces.

      I've been hunting down a performance problem, and I have found it now. I made a reproducible test case for it. I attached that test-case, if you have ant it is reasonably easy to compile/ run.

      What the test does is do 1000 client-server requests to parse a small file with Xerces (that I do not use jaxp is intentional, I want 'my' jars to be used for the parsing). I time how long this takes from the client perspective.

      I ran the test with two configurations:
      - Scenario 1: sample-bean jar in server/default/deploy, and the Xerces jars in server/default/lib. This is how I believe you are meant to deploy (I am kind of a newbie to EJB).
      - Scenario 2: sample-bean jar in server/default deploy, and Xerces jars added to the JBoss classpath, like so:
      set JBOSS_CLASSPATH=d:\cl\jars\xercesImpl.jar;d:\cl\jars\xmlParserAPIs.jar
      and not in any JBoss directory.


      Here are the times for the parsing of 1000 docs:
      - Scenario 1: 36 seconds
      - Scenario 2: 13 seconds...

      I respect that the JBoss classloaders do some extra checks or something, but not that it becomes 3 times slower. Has anybody else experienced this? Is this what should be expected? Can something be configured to make it faster with the jars in the JBoss directory?
      Most importantly for me is: What should we advise our customers (we have an EJB binding for our product, currently we advise people to configure it according to scenario 1....).

      (It has nothing to do with Xerces. I just made a test-case with Xerces so you or somebody else may reproduce it, the problem was found when testing our own software)

      Kind regards,

      --Sander.

      P.S.: The forum does not seem to like attachments, so you can now actually download it from
      http://support.xhive.com/downloads/extras/testcase.zip

        • 1. Re: JBoss classloaders vs. the default JVM classloader

          I suspect you are experiencing a problem with
          loadClass() misses. i.e. classes that don't exist.
          The JBoss classloaders cache a found class for future
          use.
          There is an optimization for this in 3.0.1+

          Regards,
          Adrian

          • 2. Re: JBoss classloaders vs. the default JVM classloader
            sanderbos

            Dear Adrian,

            thank you for your reply.

            I looked at the problem a little more in the mean-time. In JBoss 3.0.0, I added a debug statement in
            org.jboss.mx.loading.UnifiedLoaderRepository::loadClass(String name, boolean resolve, ClassLoader cl)
            Would this show the non-existing classes? Because after one request (in the first request I see lots of classes being loaded), for further requests I only see java.lang.Integer/Long (no idea why) and org.apache.xerces.parsers.StandardParserConfiguration (because of some crazy way Xerces initializes that thing directly on the classloader) being loaded.

            I made a test-program in which I extended the URLClassloader, because I wanted to see whether all custom classloaders would lead to a performance degradation (that one turned out to be slightly faster than the JVM one...), in which I also had debug output to see what classes were loaded. I did not notice it attempting to load non-existing classes. Could you indicate what kind of classes it may not be able to find (the parsing succeeds, there is not a functionality problem).


            Anyway, I have download JBoss 3.0.1, the times for my test-case are:
            - With Xerces in server\default\lib, 29 seconds
            - With Xerces in classpath outside JBoss, 12 seconds.
            So closer, but not close enough to ignore the difference in my opinion.


            The forum screen indicates you have posted 1561 messages in this forum, so I feel it is safe to regard you as an authority on JBoss? Can I read from your answer that it is not a known issue then that JBoss classloaders are slower than the JVM ones?
            I would also find it very strange if I would be the first to notice this. The problem I have that running our product in JBoss is a lot slower currently than running it 'directly' in a Java program. If you (or anybody else) have any idea on how I should proceed to find what's wrong I would love to hear it (in my opinion it is not related to our product, that is why I made the testcase for Xerces...).

            Kind regards,

            --Sander.

            • 3. Re: JBoss classloaders vs. the default JVM classloader
              marcin

              Looks like i have the same problem. We wrote quite big application based on Cocoon 2, and running it on JBoss 2.4.3 is about 3 times faster that on JBoss 3.0.2.

              Optimization software indicates that most of the time is spent in java.net.URLClassLoader which is used by org.jboss.mx.loading.UnifiedClassLoader.getResource() method. The last one method is used by xalan and xpath.

              This issue blocks our project from moving to jboss 3 and running it on the cluster.

              Someone can explain why it happens and if can we expect fixing of this problem?

              Regards
              Marcin

              • 4. Re: JBoss classloaders vs. the default JVM classloader
                marcin

                well,

                i found why cocoon works so slow with jboss 3.0 but i do not know if my solution will work for you.

                Each time the request was handled by cocoon the pipeline is set up, this means initializing xalan, and xalan initializes the module called dtm. As i read on xml.apache.org/xalan-j/ it is the technology for efficient xpath queries for sax processing.
                So back to the point, initializing this dtm module was done by requesting factory - DTMManager (org.apache.xml.dtm.DTMManager) for an instance of the DTMManager. This was done by findFactory method. Each time this is done the searching algorithm is applied. Doesnt matter how this algorithm works but it is important that everytime classloader is requested for a resource (uppss!) and it takes about 150-300ms (2xPIII 700, 1GB). The whole process is repeated for 4-20 times for each request - too bad!.

                But looking at this algorithm i saw that if i provide system property org.apache.xml.dtm.DTMManager=org.apache.xml.dtm.ref.DTMManagerDefault it will not call classloader. So this is the solution.

                Now my application works only about 30% slower that on jboss 2.

                Regards
                Marcin

                • 5. Re: JBoss classloaders vs. the default JVM classloader
                  sanderbos

                  Hi there,

                  this thing is starting to give me a severe headache.

                  I picked Xerces to make the test with, because I figured 'well everybody can download that'.
                  Because Marcin mentioned profiling the the software I did that too. Turns out Xerces does some very peculiar classloading thingies during the intialization of the parser.

                  In my test EJB method code, I had the lines:

                  DOMParser parser = new DOMParser();
                  parser.parse(fileName);
                  Document doc = parser.getDocument();

                  If I change that to

                  DOMParser parser = new DOMParser(new org.apache.xerces.parsers.StandardParserConfiguration());
                  parser.parse(fileName);
                  Document doc = parser.getDocument();

                  the times will be the same for both configuration methods (both 10 seconds, so also faster than before).

                  Turns out Xerces has some crazy code, where if you don't pass a configuration, it will try to read a configuration file through the classloader (or something like that, I don't even want to know anymore). It's in
                  ObjectFactory::createObject(String,String,String)
                  for those who are interested.

                  All of this led to a lot of time being spend org.jboss.mx.loading.UnifiedClassLoader2.getResource, like with Marcin. Maybe Xalan has the same tricks going on.

                  Now we use both Xalan and Xerces, so maybe that is the problem. But actually I think the problem also occurred in code that uses neither. Our own software calls Class.forName() a lot (a lot), I thought that was the problem. But we use simply Class.forName, for objects that all exist.
                  I will have to do a little more testing to see whether our problem is really something else.

                  Kind regards,

                  --Sander.

                  • 6. Re: JBoss classloaders vs. the default JVM classloader

                    This is horrible!

                    On my machine your test case is spending half
                    its time looking for
                    META-INF/services/org.apache.xerces.xni.parser.XMLParserConfiguration

                    The logic goes something like:

                    Is the resource cached for sample.jar? No

                    Ok, look through all the jars in server/default/lib and
                    server/default/deploy

                    Ah! xercesImpl.jar has it, cache the result against
                    xercesImpl.jar

                    Repeat everytime we construct a DomParser.

                    I noticed the Log4jService has the same problem when
                    checking for a modified log4j.xml

                    I'll take a look at some optimizations.

                    Regards,
                    Adrian

                    • 7. Re: JBoss classloaders vs. the default JVM classloader

                      FYI

                      I committed an optimization last night that makes
                      your testcase run twice as fast.

                      Regards,
                      Adrian

                      • 8. Re: JBoss classloaders vs. the default JVM classloader
                        sanderbos

                        Dear Adrian,

                        thank you for your patch and all your help. We have now found another problem in another third-party jar, and in combination with the Xerces problem it accounts for most of the difference in performance we had, so I consider this problem solved with respect to the JBoss integration now.

                        Thanks again, also for the prompt replies,

                        --Sander.