1 2 3 4 Previous Next 81 Replies Latest reply on Aug 3, 2014 4:18 AM by Wulf Rowek Go to original post
      • 17. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
        C. T. Newbie

        This situation is still reoccuring. It is not necessarly 24 hours every time but just a general slow-down depening AFAICT on usage. At server start, the bubble sort takes 20 seconds. Now the bubble sort (servlet deployed into JBoss) is timing out before completing. We are in contact with the sysops who are analyzing the VMWare metrics - the VM has dedicated memory, so this should not be the issue. The same code, running as a java JAR at the command line is still fast:

         

        C:\Users\apl-xxx\Desktop>java -jar PerfTest.jar

        63ms took to generate array

        19032ms took to bubblesort array

        0ms took to generate array

        18970ms took to bubblesort array

        0ms took to generate array

        18829ms took to bubblesort array

         

        So, if it is environmental, then I am unclear why the same code is running so fast from the command line and why simply restarting the JBoss server resolves the issue. Everything else (e.g. non-java) on the server is also running at completely normal speeds.

        • 18. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
          Scott Marlow Master

          You said earlier:

           

          RAM and CPU usage is ok

           

          How do you know that RAM is okay?  Have you taken a Java heap dump?  You could use the jmap tool to take a snapshot of Java memory and then examine it with Eclipse Memory Anaylzer.

           

          How do you know that CPU usage is okay?  Are you watching system level statistics that inform you of the virtual machine stats?  Is the cpu usage the same during the first hour as it is 24 hours later?

           

          If looking at the above doesn't help, you could look at java thread dumps of the app server process, while it is doing the test.  Capture a few thread dumps in the middle of the test and look for unexpected activity or signs that the tested operations are blocked on some unexpected resource.  Do this during the first hour and again after the slow down.

          • 19. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
            Daniel Wehrle Newbie

            Hi Scott,

             

            we have made this. We have inspected the running Jboss with vmvisual looking into running threads in real time.

            • 20. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
              jaikiran pai Master

              Chris Turchin wrote:

               

               

              So, if it is environmental, then I am unclear why the same code is running so fast from the command line and why simply restarting the JBoss server resolves the issue.

              Have you reproduced this locally on a JBoss installation? That'll tell for sure if it's environmental.

              • 21. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                C. T. Newbie

                thanks for the comments, jaikiran and scott.

                 

                as daniel noted, we have done many heap dumps and used jvisualvm to investigate the threads.

                we are not seeing anything specific to our application or environment, but maybe we just have blinders on, i dunno...

                 

                sysops replied and sent us the statistics from the vmware host for the client OS, which can be seen here:

                https://www.dropbox.com/sh/hp6ihlj94j5m5de/e20A5r62L3/jboss

                the cpu peaks this morning are the bubble sorts, which should be only taking 20 seconds but are taking - literally - hours...

                the rest of the memory and cpu usage over the past days looks "normal" to us, e.g. not much going on at all...

                 

                recently we have had jboss on nightly restarts which seems to keep the problem at bay.

                before moving to jboss7, we had month(s) long uptimes - we will keep investigating, but looks  like we are going back to nightly restarts for the time being.

                if you have any more hints where we could look, we are very eager to hear it!  thanks again. :-)

                cheers

                -chris

                • 22. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                  jaikiran pai Master

                  Chris Turchin wrote:

                   

                  thanks for the comments, jaikiran and scott.

                   

                  as daniel noted, we have done many heap dumps and used jvisualvm to investigate the threads.

                  we are not seeing anything specific to our application or environment, but maybe we just have blinders on, i dunno...

                   

                  Can you share those thread dumps (that are taken a few seconds apart) when this issue occurs?

                  • 23. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                    Daniel Wehrle Newbie

                    We have now some data from jmap and a thread dump with running bubblesort and slow system.

                     

                    What is slow:

                    - bubblesort in separat web application in a servlet

                    - our webapplication

                     

                    What is still fast:

                    - bubblesort startet as hornetQ-Thread

                     

                    Could there be a problem with web threads?

                     

                    C:\server\jdk1.7.0_15\bin>jmap -J-d64 -heap 3384

                    Attaching to process ID 3384, please wait...

                    Debugger attached successfully.

                    Server compiler detected.

                    JVM version is 23.7-b01

                     

                    using thread-local object allocation.

                    Parallel GC with 4 thread(s)

                     

                    Heap Configuration:

                       MinHeapFreeRatio = 40

                       MaxHeapFreeRatio = 70

                       MaxHeapSize      = 8589934592 (8192.0MB)

                       NewSize          = 1310720 (1.25MB)

                       MaxNewSize       = 17592186044415 MB

                       OldSize          = 5439488 (5.1875MB)

                       NewRatio         = 2

                       SurvivorRatio    = 8

                       PermSize         = 21757952 (20.75MB)

                       MaxPermSize      = 268435456 (256.0MB)

                       G1HeapRegionSize = 0 (0.0MB)

                     

                    Heap Usage:

                    PS Young Generation

                    Eden Space:

                       capacity = 1093009408 (1042.375MB)

                       used     = 151490448 (144.47254943847656MB)

                       free     = 941518960 (897.9024505615234MB)

                       13.859939986902656% used

                    From Space:

                       capacity = 35323904 (33.6875MB)

                       used     = 28473432 (27.154380798339844MB)

                       free     = 6850472 (6.533119201660156MB)

                       80.606696247391% used

                    To Space:

                       capacity = 35913728 (34.25MB)

                       used     = 0 (0.0MB)

                       free     = 35913728 (34.25MB)

                       0.0% used

                    PS Old Generation

                       capacity = 398196736 (379.75MB)

                       used     = 372196360 (354.95410919189453MB)

                       free     = 26000376 (24.79589080810547MB)

                       93.47046983328362% used

                    PS Perm Generation

                       capacity = 215220224 (205.25MB)

                       used     = 157663368 (150.35950469970703MB)

                       free     = 57556856 (54.89049530029297MB)

                       73.25676233846872% used

                     

                    62290 interned Strings occupying 6550280 bytes.

                    • 24. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                      jaikiran pai Master

                      Daniel Wehrle wrote:

                       

                      We have now some data from jmap and a thread dump with running bubblesort and slow system.

                       

                      What is slow:

                      - bubblesort in separat web application in a servlet

                      - our webapplication

                       

                      You mean after the server has been up for a while, right?

                       

                       

                      Daniel Wehrle wrote:

                       

                       

                      What is still fast:

                      - bubblesort startet as hornetQ-Thread

                       

                      Was the HornetQ thread test done when you started noticing the slow response for the bubblesort in a web application? What I'm trying to understand is, whether this test was done before the system started showing the slowness. If so, then this isn't a good candidate to base any theory on.

                       

                       

                      Daniel Wehrle wrote:

                       

                       

                      Could there be a problem with web threads?

                       

                      I don't know. HornetQ threads are pooled too. The only thing that I can note is that the web threads have perhaps been pooled for a longer time than the HornetQ threads. Does that make any difference in the virual environment? I have no idea.

                       

                      You are testing against EAP 6.1 right? Can you attach the standalone configuration xml being used?

                      • 25. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                        Daniel Wehrle Newbie

                        You mean after the server has been up for a while, right?

                        Yes.

                        Was the HornetQ thread test done when you started noticing the slow response for the bubblesort in a web application? What I'm trying to understand is, whether this test was done before the system started showing the slowness. If so, then this isn't a good candidate to base any theory on.

                        Yes.

                        You are testing against EAP 6.1 right? Can you attach the standalone configuration xml being used?

                         

                        First we had 7.1.1, then we changed to 7.1.3, then to 6.1.0 alpha and then back to 7.1.3. With all releases same behaviour. See standalone.xml attached.

                         

                        Here also the parameters of the jvm:

                         

                        -XX:+TieredCompilation
                        -XX:+UseCompressedOops
                        -Dprogram.name=standalone.bat
                        -Dmc.root=..\..\..\server\namespaces\client
                        -Xmx8192m
                        -Djboss.socket.binding.port-offset=0
                        -Xrs
                        -Xms64M
                        -XX:MaxPermSize=256M
                        -Dsun.rmi.dgc.client.gcInterval=3600000
                        -Dsun.rmi.dgc.server.gcInterval=3600000
                        -Djava.net.preferIPv4Stack=true
                        -Dorg.jboss.resolver.warning=true
                        -Djboss.modules.system.pkgs=org.jboss.byteman
                        -Djboss.server.default.config=standalone.xml
                        -Dorg.apache.coyote.http11.Http11Protocol.COMPRESSION=on
                        -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
                        -Dorg.jboss.boot.log.file=C:\mediacockpit\server\jboss-eap.krones\standalone\log\boot.log
                        -Dlogging.configuration=file:C:\mediacockpit\server\jboss-eap.krones\standalone/configuration/logging.properties
                        
                        • 26. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                          jaikiran pai Master

                          -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

                          This isn't a good thing to have in production or when you are testing performance. Some past discussions in this forums have shown that it can impact the performance terribly and it doesn't matter if no debugger is attached. Here's a relevant reference http://stackoverflow.com/questions/2195720/why-does-java-code-slow-down-in-debugger. I would recommend that you remove that from the JVM options and keep a watch on the behaviour.

                          • 27. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                            Daniel Wehrle Newbie

                            jaikiran pai schrieb:

                             

                             

                            -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

                             

                            This isn't a good thing to have in production or when you are testing performance. Some past discussions in this forums have shown that it can impact the performance terribly and it doesn't matter if no debugger is attached. Here's a relevant reference http://stackoverflow.com/questions/2195720/why-does-java-code-slow-down-in-debugger. I would recommend that you remove that from the JVM options and keep a watch on the behaviour.

                             

                            We will give this a try. Thanx.

                            • 28. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                              robert ritchie Newbie

                              The project I work on has recently made a similar transition to AS 7.2.0 Alpha Snapshot and then EAP 6.1.0 Alpha.  At a recent customer test event, early April, we had to talk around and mitigate the performance issues we were seeing by doing nightly restarts of JBoss on our project as well.

                               

                              For the second week of the test event we took out the jdwp listener as described in this post and did not require any JBoss restarts that week, although load was significantly lower during the second week of testing.

                               

                              However, after 48-72 hours we are still seeing performance degredation so naturally our customer/management does not believe the issue to be resolved.

                               

                              I am very interested in hearing about the performance improvements yielded by Daniel and Chris after applying this change and curious if they will see the issue as "fixed" and whether or not they still see the slowdown just after a longer period of time as I am seeing on my project or if they will once again go back to month(s) of uptime as they described in their posts.

                               

                              In any event, this forum thread has provided me at least with enough proof that the configuration change we had made during the test event did in fact fix A performance issue and that what they are seeing now is more than likely a seperate issue(s).

                              • 29. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
                                Daniel Wehrle Newbie

                                Daniel Wehrle schrieb:

                                 

                                jaikiran pai schrieb:

                                 

                                 

                                -Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

                                 

                                This isn't a good thing to have in production or when you are testing performance. Some past discussions in this forums have shown that it can impact the performance terribly and it doesn't matter if no debugger is attached. Here's a relevant reference http://stackoverflow.com/questions/2195720/why-does-java-code-slow-down-in-debugger. I would recommend that you remove that from the JVM options and keep a watch on the behaviour.

                                 

                                We will give this a try. Thanx.

                                 

                                We have another production system where the jdwp listener was NOT activated. There are the same problems. So I can exclude this listener bringing the performance problems.

                                 

                                What about the web threads? Could it be that threads in the pool are corrupt but will still be used?

                                1 2 3 4 Previous Next