14 Replies Latest reply on Apr 25, 2013 9:33 AM by mausbull

    JBoss AS 7.1.0.Final - performance on RHEL6

    mausbull

      Hi,

       

      I've got a huge performance problem running JBoss AS 7.1.0.Final on RHEL 6. Startup is extremely slow even when nothing is deployed.

      I used the exact same JBoss on other systems (Ubuntu, Debian, Mint) without the mentioned problem.

       

      Is there a fix how to speed up AS 7 on RHEL 6?

       

      Thanks, Stephan

       

      JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms512m -Xmx2048m -XX:PermSize=512M -XX:MaxPermSize=1024m -Djava.net.preferIPv4Stack=true 
      -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=standalone.xml -XX:+UseCompressedOops
       -XX:+DisableExplicitGC -noverify -Djboss.socket.binding.port-offset=10000
      
        • 1. Re: JBoss AS 7.1.0.Final - performance on RHEL6
          jaikiran

          What kind of performance problem? On a related note, please use a latest released version instead of 7.1.0.Final.

          • 2. Re: JBoss AS 7.1.0.Final - performance on RHEL6
            mausbull

            It takes 8 minutes from start to final deployment.

            On an Ubuntu system, the same deployment takes 30 seconds.

             

            I can't use the latest version as our application is not compatible with the latest JBoss version.

            • 3. Re: JBoss AS 7.1.0.Final - performance on RHEL6
              jaikiran

              Do you have logs or thread dumps which suggest where it's taking time? Also, what's the output of:

               

              java -version

               

              on that system?

              • 4. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                ctomc

                Hi,

                 

                what is hardware like? Virutal? How much ram? what cpu, what kind of disks.

                i asume 64bit, but can you be more precise.

                what does uname -a say?

                what is exact version of your jvm?

                 

                looking at your jvm settings. they seem bit strange, but without hardware info i cannot comment if they are wrong.

                 

                 

                --

                tomaz

                • 5. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                  mausbull

                  It takes 53804ms to start Thunder.

                   

                   

                  Then 5mins to do:

                  10:10:59,710 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-21) JBAS010400: Bound data source [java:jboss/DefaultDS]

                  10:15:27,355 WARN  [org.jboss.as.ejb3.deployment.processors.SessionBeanComponentDescriptionFactory]

                   

                   

                  Another 6 mins:

                  10:15:33,785 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-47) Initializing Mojarra 2.1.5 (SNAPSHOT 20111202) for context '/SeqBench'

                  10:21:52,712 INFO  [org.primefaces.webapp.PostConstructApplicationEventListener] (MSC service thread 1-47) Running on PrimeFaces 3.4.1

                   

                   

                  Java: java version "1.7.0_21"

                  Uname: "Linux 2.6.32-220.17.1.el6.x86_64 #1 SMP Thu Apr 26 13:37:13 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux"

                  MemTotal: 132141840 kB

                  Proc: 24 processors a X7542  @ 2.67GHz

                  JVM Settings: I played around a bit with them to see if I can fix the problem. Didn't improved the performance compared to the standard/shipped settings.

                   

                   

                  Log is at https://dl.dropboxusercontent.com/u/14071835/jboss_log_2013_04_23.txt

                   

                   

                  Thanks Stephan

                  • 6. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                    jaikiran

                    This doesn't match with the initial post in this thread which said:

                     

                    Startup is extremely slow even when nothing is deployed.

                    It looks like the server starts in 53 odd seconds when nothing is deployed and it's the deploying of the deployment which is taking time.

                     

                    Having said that, from what you say about the timings, I now have the same questions as Tomaz about the hardware.

                    • 7. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                      mausbull

                      "53 sec startup" vs "30 sec startup & deployment" -- in my view that's slow. What bugs me is that it's the exact same system that runs on a multitude of OS really well, just on this one it doesn't.

                       

                      In addition to what I posted in the previous answer, what do you need to know about the hardware?

                       

                      The server is a HP Proliant DL580 G7

                      • 4 CPUs with in total 24 cores (Intel X7542, 2.66 GHz)
                      • 128 GB RAM (5.3 GB/core)
                      • 1.2 TB local high performance storage (SAS disks, 10k rpm)
                      • 24 TB local storage (HP-StorageWorks P2000) with 6GBit/sec SATA
                      • 8. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                        sfcoy

                        This sounds like an under resourced virtual machine, and that hardware smells a lot like a VM hosting box.

                        • 9. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                          mausbull

                          Well, it's neither a virtualbox where JBoss is running in, nor is the server a VM hosting box. It's a server where we're running successfully (other JEE) applications on.

                           

                          Any updates about the performance issue?

                          • 10. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                            ctomc

                            Hi,

                             

                            that is bizzare to put it midly....

                             

                            how long does boot take without deployments? Given that log it also takes about 10 sec+ which is huge.

                             

                             

                            The only issue about slow(er) boot in 7.1.0 i know about is that it could take up to few extra seconds if there are many network interfaces on system.

                            but that should not result in such difference.

                             

                            Can you try two things.

                             

                            running server with 32bit jvm (with smaller heap sizes)

                            and can you try with 7.2 http://www.jboss.org/jbossas/downloads/ (EAP6.1.*)

                            i am curius what is going on.

                             

                            I am running many instances on RHEL6 on much much slower hardware and have boot times under 3sec... (without deployments)

                            • 11. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                              ctomc

                              Also could you do thread dump when this slowness is happening.

                               

                              It would be nice to get it in operations that take long time, like when web server is starting

                              inbetween this two log messages:

                               

                              10:10:08,705 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) JBAS015400: Bound mail session [java:jboss/mail/Default]

                              10:10:58,151 INFO  [org.apache.coyote.ajp.AjpProtocol] (MSC service thread 1-47) Starting Coyote AJP/1.3 on ajp--127.0.0.1-18009

                               

                              you can get thread dump by calling jstack <java process id> (jstack is present in bin folder of jdk)

                               

                              --

                              tomaz

                              • 12. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                                sfcoy

                                I agree with Tomaz that it's quite bizzare.

                                 

                                It certainly seems environmental.

                                 

                                This 50 second gap for example:

                                 

                                {code}

                                10:10:08,705 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-2) JBAS015400: Bound mail session [java:jboss/mail/Default]

                                10:10:58,151 INFO  [org.apache.coyote.ajp.AjpProtocol] (MSC service thread 1-47) Starting Coyote AJP/1.3 on ajp--127.0.0.1-18009

                                {code}

                                 

                                and the fact that it's up to MSC service thread 1-47 suggests maybe CPU starvation?

                                • 13. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                                  nickarls

                                  Does the server have free access to the outside internet? Surely the CPU can't show 100% when it's booting, right?

                                  • 14. Re: JBoss AS 7.1.0.Final - performance on RHEL6
                                    mausbull

                                    32bit, 7.2, jstack, "access to outside" --- didn't resolve the issue.

                                     

                                    We finally applied solution number 1 -> Reboot.

                                    JBoss is now behaving normally, no performance issues so far.

                                    We believe it could have been the "Leap second bug" [1] that caused it.

                                     

                                    Thanks for your help!

                                    Stephan

                                     

                                    [1]

                                    http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/