1 2 Previous Next 15 Replies Latest reply on Feb 1, 2005 9:05 AM by Robert Sklar

    JBoss vs. WebLogic tuning results

    Steven Grimm Newbie

      Hi

      I've been uning JBoss 3.2.1 for some time with the JBoss.net axis webservice integration. And this worked ok. We want to upgrade to 3.2.6 because of a path to the UnifiedClassloader.

      However when we try to run the same webservice on 3.2.6 we get a NullpointerException in org.jboss.net.axis.Deployment.oldGetService.

      Full stactrace:

      java.lang.NullPointerException
      at org.jboss.net.axis.Deployment.oldGetService(Deployment.java:255)
      at org.jboss.net.axis.Deployment.getService(Deployment.java:275)
      at org.apache.axis.configuration.FileProvider.getService(FileProvider.java:290)
      at org.apache.axis.AxisEngine.getService(AxisEngine.java:302)
      at org.apache.axis.transport.http.AxisServlet.doPost(AxisServlet.java:900)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
      at org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:372)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
      at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:75)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:186)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:214)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
      at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
      at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:66)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
      at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:158)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
      at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
      at org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104)
      at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520)
      at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:929)
      at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160)
      at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799)
      at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705)
      at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577)
      at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
      at java.lang.Thread.run(Thread.java:534)


      I looked at the code where the Nullpointer originates, and it looks like the MessageContext isn't initialised at the time of the call. When I look at the code in AxisServlet.doPost is seems that the initialisation of this context is done after the call to getService(). To me it looks like this cannot work. Does anyone know if this is a bug, or if there is a workarond for it?

      I also noticed that the WSDL generated from the 3.2.6 differs fra the one from 3.2.1.

      Snippet from 3.2.1:
      <wsdl:operation name="execute">
      <wsdlsoap:operation soapAction="Manager"/>
      <wsdl:input name="executeRequest">
      <wsdlsoap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="http://localhost:8080/jboss-net/services/Manager" use="encoded"/>
      </wsdl:input>
      <wsdl:output name="executeResponse">
      <wsdlsoap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="http://localhost:8080/jboss-net/services/Manager" use="encoded"/>
      </wsdl:output>
      </wsdl:operation>

      Snippet fra 3.2.6
      <wsdl:operation name="execute">
      <wsdlsoap:operation soapAction="Manager"/>
      <wsdl:input name="executeRequest">
      <wsdlsoap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="http://service.tjener.myapplication.myorganization.no" use="encoded"/>
      </wsdl:input>
      <wsdl:output name="executeResponse">
      <wsdlsoap:body encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="http://localhost:8080/jboss-net/services/Manager" use="encoded"/>
      </wsdl:output>
      <wsdl:fault name="MyAppException">
      <wsdlsoap:fault encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" namespace="http://localhost:8080/jboss-net/services/Manager" use="encoded"/>
      </wsdl:fault>
      </wsdl:operation>

      The marked namespace in the 3.2.6 version is completely bogus, and I have no idea where i came from.

      Any ideas to what the problem is? I would greatly appretiate any input.

      Best regards
      Øystein Aadland

        • 1. Re: JBoss vs. WebLogic tuning results
          Remy Maucherat Novice

          Some tips for the web tier based on the information you submitted:
          - mod_jk 2 is actually officially abandoned: you should use mod_jk 1.2.x instead (or use Apache HEAD branch with mod_proxy and mod_proxy_ajp)
          - if there's no functional need for Apache, I don't see much gain from using it, so you could try using a simpler setup with the Tomcat HTTP connector instead

          Maybe you can try enabling verbose GC to see if there are indeed too many long GC. If you want better telemetry, I've had a good experience with jconsole and Java 5, as well as with JFluid (but it's going to be difficult to install that on your servers ;) ).

          • 2. Re: Image galery module dev
            Steven Grimm Newbie

             

            "cnovara" wrote:
            I use another way to achieve this.
            To serve big amount of downloads (static content), it's better to use Apache instead of Tomcat.
            This way I manage a complete directory out of Nukes, with classic Apache Locations/Directories, that I synchronize with Nukes DB wich contains links (this could be automatic).
            Anyway, Apache is always in front of Tomcat, because it's more versatile.

            See enfanceetcroissance.com


            Could you please explain some more regarding how you done this!? What is the setup on Apache side and on JBoss/Tomcat/Nukes side?

            So, the result of this is that when you in Nukes selects a link to some static stuff this is served by Apache and not Tomcat?

            Thanks,
            /L

            • 3. I/O leak in jboss application -
              Scott Stark Master

              hi all,

              we have recently deployed a jboss based application to a production server -
              after 7 days of perfect performance - the sys admin tried to logon
              using jass- role based auth (user/role properties file)
              and failed every time.

              after watching the logs i have noticed an I/O Exception- too many open files -
              now thats odd -
              and after further tests i noticed that the thread count is always rising .
              after 3 days the thread count is 400 !!

              so first how can i isolate the leak to find if its in the WEB/EJB container ?

              considering the fact that i cant physically access the server only through a secured shell - how can i find the source of all the threads?

              we are using Jboss 4.0 on Linux fedora 2 with jdk 5(tiger)

              if anyone has any advise or idea ...
              thanks

              Alon.

              • 4. Re: JBoss vs. WebLogic tuning results
                sfrind Newbie

                Have a look at
                http://www-106.ibm.com/developerworks/java/library/j-jtp11253/
                and try to setup your JVM parameters more properly.

                Regards,
                Stephan

                • 5. Re: JBoss vs. WebLogic tuning results
                  Alexey Loubyansky Master

                  Have you tried 'cmp2.x jdbc2 pm' container?
                  Here is a brief description of it
                  http://www.jboss.org/wiki/Wiki.jsp?page=CMP2xJDBC2PM

                  • 6. Re: JBoss vs. WebLogic tuning results
                    Steven Grimm Newbie

                     

                    Have you tried 'cmp2.x jdbc2 pm' container?

                    Just did, and it seems to not like redeploying my EJBs -- do I have to do something special to get redeployment to work? It spews exceptions like
                    org.jboss.deployment.DeploymentException: Failed to register table cache for IP_AUC; - nested throwable: (javax.management.InstanceAlreadyExistsException: jboss.cmp:ejbname=Auction,service=tablecache,table=IP_AUC already registered.)

                    and I have to restart the server. Which is a bit frustrating when I want to do a lot of redeployment to test my changes!

                    • 7. Re: JBoss vs. WebLogic tuning results
                      Steven Grimm Newbie

                       

                      Have a look at
                      http://www-106.ibm.com/developerworks/java/library/j-jtp11253/
                      and try to setup your JVM parameters more properly.

                      Thanks for that pointer. I've tried fiddling with different sizes for different parts of the heap and switching garbage collectors. I can shave 5-10 seconds off the test run time, which is good but still nowhere near WebLogic.

                      Turning on very verbose GC logging as described in that article was informative, however. It tells me that WebLogic is being a lot more frugal about its object creation. Here is the GC output from two consecutive test runs under WL8 using the default garbage collectors (this is on just one of the two app servers):

                      397.132: [GC 397.133: [DefNew: 209791K->24046K(235968K), 2.8994055 secs] 258827K->73082K(1022400K), 2.8995789 secs]
                      420.994: [GC 420.994: [DefNew: 233838K->16438K(235968K), 2.3361720 secs] 282874K->78856K(1022400K), 2.3365907 secs]
                      459.288: [GC 459.288: [DefNew: 226230K->18275K(235968K), 0.4941213 secs] 288648K->84648K(1022400K), 0.4942972 secs]
                      479.701: [GC 479.702: [DefNew: 228067K->14275K(235968K), 0.4610617 secs] 294440K->84463K(1022400K), 0.4612431 secs]

                      Totaling up the differences in the "A->B" figures, that's a total of 1595MB garbage-collected.

                      On JBoss, with the tweaks that got me the biggest performance gain, namely heap size increased to 1536MB and the MaxNew size set to 768MB (vs. 1024MB and 256MB for WebLogic), the output looks like this:
                      417.876: [GC 417.876: [DefNew: 147873K->6848K(157248K), 0.1584065 secs] 254811K->113787K(1555392K), 0.1585619 secs]
                      418.741: [Full GC 418.742: [Tenured: 106938K->104471K(1398144K), 5.8910570 secs] 130645K->104471K(1555392K),
                       [Perm : 32410K->32375K(32512K)], 5.8912741 secs]
                      431.097: [GC 431.097: [DefNew: 139775K->3583K(157248K), 0.1146439 secs] 244247K->108054K(1555392K), 0.1148095 secs]
                      438.575: [GC 438.575: [DefNew: 143358K->4848K(157248K), 0.1276344 secs] 247829K->109319K(1555392K), 0.1278011 secs]
                      445.096: [GC 445.096: [DefNew: 144624K->5849K(157248K), 0.8005188 secs] 249095K->110320K(1555392K), 0.8006881 secs]
                      456.110: [GC 456.110: [DefNew: 145624K->7329K(157248K), 0.1943690 secs] 250095K->111800K(1555392K), 0.1945323 secs]
                      463.571: [GC 463.571: [DefNew: 147071K->9693K(157248K), 0.2391462 secs] 251542K->114165K(1555392K), 0.2393058 secs]
                      470.461: [GC 470.461: [DefNew: 149469K->11348K(157248K), 0.2775235 secs] 253941K->117005K(1555392K), 0.2776854 secs]
                      477.302: [GC 477.302: [DefNew: 151124K->9799K(157248K), 0.9607772 secs] 256781K->119013K(1555392K), 0.9609508 secs]
                      484.992: [Full GC 484.992: [Tenured: 109214K->120788K(1398144K), 4.1267928 secs] 248755K->120788K(1555392K),
                       [Perm : 32500K->32500K(32512K)], 4.1270046 secs]
                      495.718: [GC 495.718: [DefNew: 139772K->1598K(157248K), 0.0804109 secs] 260560K->122386K(1555392K), 0.0806087 secs]
                      502.776: [GC 502.776: [DefNew: 141364K->3113K(157248K), 0.1058699 secs] 262152K->123901K(1555392K), 0.1060297 secs]
                      509.985: [GC 509.985: [DefNew: 142889K->5255K(157248K), 0.1436009 secs] 263677K->126043K(1555392K), 0.1437650 secs]
                      520.092: [GC 520.092: [DefNew: 145031K->5883K(157248K), 0.1592386 secs] 265819K->126672K(1555392K), 0.1593999 secs]
                      526.785: [GC 526.785: [DefNew: 145659K->7906K(157248K), 0.8480749 secs] 266448K->128694K(1555392K), 0.8482264 secs]
                      533.990: [GC 533.990: [DefNew: 147682K->9275K(157248K), 0.2309098 secs] 268470K->130063K(1555392K), 0.2310682 secs]
                      541.335: [GC 541.335: [DefNew: 149051K->9970K(157248K), 0.2338242 secs] 269839K->131308K(1555392K), 0.2339857 secs]
                      547.981: [GC 547.981: [DefNew: 149746K->9285K(157248K), 0.2470304 secs] 271084K->132613K(1555392K), 0.2471913 secs]
                      549.211: [Full GC 549.211: [Tenured: 123328K->130785K(1398144K), 3.6915951 secs] 145917K->130785K(1555392K),
                       [Perm : 32523K->32523K(32768K)], 3.6918101 secs]
                      562.457: [GC 562.458: [DefNew: 139775K->5966K(157248K), 0.2013474 secs] 270561K->136752K(1555392K), 0.2015033 secs]

                      That's a total of 4736MB of objects garbage-collected.

                      I'm the first to admit I'm no expert on Java memory management! But I have to wonder if any amount of tweaking of GC settings is going to make up for a 3X difference in the amount of garbage that needs to be collected.

                      If I'm interpreting those numbers incorrectly, I look forward to being set straight.

                      • 8. Re: JBoss vs. WebLogic tuning results
                        Scott Stark Master

                        So what are the vm settings being used here? Without knowing what objects are being created that are triggering the full garbage collections, you can't say if there is a tuning setting at the server level which can eliminate the full gc pauses.

                        • 9. Re: JBoss vs. WebLogic tuning results
                          Steven Grimm Newbie

                          On the test run that produced those numbers, the VM arguments I passed into run.sh were:

                          -XX:MaxNewSize=768m -XX:SurvivorRatio=8 -Xms1536m -Xmx1536m -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails

                          I had tried setting MinNewSize as well but it didn't seem to make any big difference in the run times. SurvivorRatio is from the BEA tuning guide and seemed to help slightly, but adjusting the MaxNewSize had more impact.

                          If there's some setting I can use, or some monitoring tool I can install, that will tell me what kinds of objects are being created and destroyed, I'll be happy to try that! I want JBoss to be a viable option for us -- I'm kind of the main free-software advocate in my company and the closer I can get it to WebLogic's numbers, the easier it'll be to convince management to go with the more open software.

                          Thanks for the responses!

                          • 10. Re: JBoss vs. WebLogic tuning results
                            Scott Stark Master

                            You are also not using any of the parallel garbage collection settings that will spread out the gc cost and possibly avoid the full gc pauses. There is likely some set of objects that can be optimized to avoid this though.

                            • 11. Re: JBoss vs. WebLogic tuning results
                              Steven Grimm Newbie

                              I did try -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, and -XX:+UseParallelGC, the first two both by themselves and together. -XX:+UseParNewGC shaved maybe 5 seconds off the test run time. -XX:+UseConcMarkSweepGC actually made my test about 10 seconds slower, and -XX:+UseParNewGC didn't make much difference.

                              My sense was that some of those were aimed at multiprocessor systems with larger numbers of processors than the 2 in each of my machines. I'd easily believe that if I had 4 or 8 or 16 CPUs, I'd see a lot more benefit.

                              I will check out those profilers. Thanks for answering my newbie questions -- poking and prodding the JVM is something I've never had a need to learn much about until now.

                              • 12. 3731656
                                Alexey Loubyansky Master

                                 

                                "koreth" wrote:
                                Have you tried 'cmp2.x jdbc2 pm' container?

                                Just did, and it seems to not like redeploying my EJBs -- do I have to do something special to get redeployment to work? It spews exceptions like
                                org.jboss.deployment.DeploymentException: Failed to register table cache for IP_AUC; - nested throwable: (javax.management.InstanceAlreadyExistsException: jboss.cmp:ejbname=Auction,service=tablecache,table=IP_AUC already registered.)

                                and I have to restart the server. Which is a bit frustrating when I want to do a lot of redeployment to test my changes!


                                It should not happen. Do you see any other errors during first deployment?

                                • 13. Re: JBoss vs. WebLogic tuning results
                                  Steven Grimm Newbie

                                  First some good news...

                                  I spent some time with YourKit Java Profiler yesterday (thanks again for that pointer) and discovered a major source of short-lived garbage objects. It is not really JBoss's fault, though it is in some sense a consequence of how JBoss works. Our logging code logs a file and line number with each message. It figures that out by throwing a dummy exception and walking up the stack trace as returned by Throwable.getStackTrace(). In our utility and EJB code, each log message has a unique tag that is used to cache the file and line number, so the stack trace is only done the first time a particular message is logged and it's no big deal. But in the JSPs, since we have logging calls in JSP fragment files that get included in multiple places, unique tags in the source code won't work and we let the logging code recompute the file and line for each instance of each message.

                                  JBoss's call stacks are a lot deeper than WebLogic's, so this was causing many more short-lived StackTraceElement objects to be created on each log call from a JSP. Enough that the logging system accounted for over 30% of the objects that were garbage-collected during a test run -- and we're not even doing all that much logging from our JSPs.

                                  Commenting out the stack-trace-scanning code cut the number of GCs nearly in half. Now that I know it's a big deal, I will put something in place to do unique tags for the JSP log calls.

                                  That's the good news. The bad news is that it didn't have too much impact on the test execution time, which is now down around 1:00. I suppose the fact that each of the two servers in the cluster is doing its garbage collection at different times is somewhat mitigating the effect on throughput. Plus those objects were mostly getting cleaned up by the relatively fast scans of the "new" heap anyway.

                                  I will experiment some more with the profilers and see if their CPU usage analyses turn up any interesting bottlenecks.

                                  • 14. Re: JBoss vs. WebLogic tuning results
                                    David Tauzell Newbie

                                    Is your Weblogic installation using the JRockit JVM?

                                    1 2 Previous Next