[daniel.wehrle]

added some trace times

[jaikiran]

To me this still looks environmental. Try getting the metrics as explained here http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1017926 for analysis.

[turchinc]

This situation is still reoccuring. It is not necessarly 24 hours every time but just a general slow-down depening AFAICT on usage. At server start, the bubble sort takes 20 seconds. Now the bubble sort (servlet deployed into JBoss) is timing out before completing. We are in contact with the sysops who are analyzing the VMWare metrics - the VM has dedicated memory, so this should not be the issue. The same code, running as a java JAR at the command line is still fast:

C:\Users\apl-xxx\Desktop>java -jar PerfTest.jar

63ms took to generate array

19032ms took to bubblesort array

0ms took to generate array

18970ms took to bubblesort array

0ms took to generate array

18829ms took to bubblesort array

So, if it is environmental, then I am unclear why the same code is running so fast from the command line and why simply restarting the JBoss server resolves the issue. Everything else (e.g. non-java) on the server is also running at completely normal speeds.

[smarlow]

You said earlier:

RAM and CPU usage is ok

How do you know that RAM is okay?  Have you taken a Java heap dump?  You could use the jmap tool to take a snapshot of Java memory and then examine it with Eclipse Memory Anaylzer.

How do you know that CPU usage is okay?  Are you watching system level statistics that inform you of the virtual machine stats?  Is the cpu usage the same during the first hour as it is 24 hours later?

If looking at the above doesn't help, you could look at java thread dumps of the app server process, while it is doing the test.  Capture a few thread dumps in the middle of the test and look for unexpected activity or signs that the tested operations are blocked on some unexpected resource.  Do this during the first hour and again after the slow down.

[daniel.wehrle]

Hi Scott,

we have made this. We have inspected the running Jboss with vmvisual looking into running threads in real time.

[jaikiran]

Chris Turchin wrote:

So, if it is environmental, then I am unclear why the same code is running so fast from the command line and why simply restarting the JBoss server resolves the issue.

Have you reproduced this locally on a JBoss installation? That'll tell for sure if it's environmental.

[turchinc]

thanks for the comments, jaikiran and scott.

as daniel noted, we have done many heap dumps and used jvisualvm to investigate the threads.

we are not seeing anything specific to our application or environment, but maybe we just have blinders on, i dunno...

sysops replied and sent us the statistics from the vmware host for the client OS, which can be seen here:

https://www.dropbox.com/sh/hp6ihlj94j5m5de/e20A5r62L3/jboss

the cpu peaks this morning are the bubble sorts, which should be only taking 20 seconds but are taking - literally - hours...

the rest of the memory and cpu usage over the past days looks "normal" to us, e.g. not much going on at all...

recently we have had jboss on nightly restarts which seems to keep the problem at bay.

before moving to jboss7, we had month(s) long uptimes - we will keep investigating, but looks  like we are going back to nightly restarts for the time being.

if you have any more hints where we could look, we are very eager to hear it!  thanks again. :-)

cheers

-chris

[jaikiran]

Chris Turchin wrote:

thanks for the comments, jaikiran and scott.

as daniel noted, we have done many heap dumps and used jvisualvm to investigate the threads.

we are not seeing anything specific to our application or environment, but maybe we just have blinders on, i dunno...

Can you share those thread dumps (that are taken a few seconds apart) when this issue occurs?

[daniel.wehrle]

We have now some data from jmap and a thread dump with running bubblesort and slow system.

What is slow:

- bubblesort in separat web application in a servlet

- our webapplication

What is still fast:

- bubblesort startet as hornetQ-Thread

Could there be a problem with web threads?

C:\server\jdk1.7.0_15\bin>jmap -J-d64 -heap 3384

Attaching to process ID 3384, please wait...

Debugger attached successfully.

Server compiler detected.

JVM version is 23.7-b01

using thread-local object allocation.

Parallel GC with 4 thread(s)

Heap Configuration:

   MinHeapFreeRatio = 40

   MaxHeapFreeRatio = 70

   MaxHeapSize      = 8589934592 (8192.0MB)

   NewSize          = 1310720 (1.25MB)

   MaxNewSize       = 17592186044415 MB

   OldSize          = 5439488 (5.1875MB)

   NewRatio         = 2

   SurvivorRatio    = 8

   PermSize         = 21757952 (20.75MB)

   MaxPermSize      = 268435456 (256.0MB)

   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:

PS Young Generation

Eden Space:

   capacity = 1093009408 (1042.375MB)

   used     = 151490448 (144.47254943847656MB)

   free     = 941518960 (897.9024505615234MB)

   13.859939986902656% used

From Space:

   capacity = 35323904 (33.6875MB)

   used     = 28473432 (27.154380798339844MB)

   free     = 6850472 (6.533119201660156MB)

   80.606696247391% used

To Space:

   capacity = 35913728 (34.25MB)

   used     = 0 (0.0MB)

   free     = 35913728 (34.25MB)

   0.0% used

PS Old Generation

   capacity = 398196736 (379.75MB)

   used     = 372196360 (354.95410919189453MB)

   free     = 26000376 (24.79589080810547MB)

   93.47046983328362% used

PS Perm Generation

   capacity = 215220224 (205.25MB)

   used     = 157663368 (150.35950469970703MB)

   free     = 57556856 (54.89049530029297MB)

   73.25676233846872% used

62290 interned Strings occupying 6550280 bytes.

[jaikiran]

Daniel Wehrle wrote:

We have now some data from jmap and a thread dump with running bubblesort and slow system.

What is slow:

- bubblesort in separat web application in a servlet

- our webapplication

You mean after the server has been up for a while, right?

Daniel Wehrle wrote:

What is still fast:

- bubblesort startet as hornetQ-Thread

Was the HornetQ thread test done when you started noticing the slow response for the bubblesort in a web application? What I'm trying to understand is, whether this test was done before the system started showing the slowness. If so, then this isn't a good candidate to base any theory on.

Daniel Wehrle wrote:

Could there be a problem with web threads?

I don't know. HornetQ threads are pooled too. The only thing that I can note is that the web threads have perhaps been pooled for a longer time than the HornetQ threads. Does that make any difference in the virual environment? I have no idea.

You are testing against EAP 6.1 right? Can you attach the standalone configuration xml being used?

[daniel.wehrle]

You mean after the server has been up for a while, right?

Yes.

Was the HornetQ thread test done when you started noticing the slow response for the bubblesort in a web application? What I'm trying to understand is, whether this test was done before the system started showing the slowness. If so, then this isn't a good candidate to base any theory on.

Yes.

You are testing against EAP 6.1 right? Can you attach the standalone configuration xml being used?

First we had 7.1.1, then we changed to 7.1.3, then to 6.1.0 alpha and then back to 7.1.3. With all releases same behaviour. See standalone.xml attached.

Here also the parameters of the jvm:

-XX:+TieredCompilation
-XX:+UseCompressedOops
-Dprogram.name=standalone.bat
-Dmc.root=..\..\..\server\namespaces\client
-Xmx8192m
-Djboss.socket.binding.port-offset=0
-Xrs
-Xms64M
-XX:MaxPermSize=256M
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000
-Djava.net.preferIPv4Stack=true
-Dorg.jboss.resolver.warning=true
-Djboss.modules.system.pkgs=org.jboss.byteman
-Djboss.server.default.config=standalone.xml
-Dorg.apache.coyote.http11.Http11Protocol.COMPRESSION=on
-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
-Dorg.jboss.boot.log.file=C:\mediacockpit\server\jboss-eap.krones\standalone\log\boot.log
-Dlogging.configuration=file:C:\mediacockpit\server\jboss-eap.krones\standalone/configuration/logging.properties

[jaikiran]

-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

This isn't a good thing to have in production or when you are testing performance. Some past discussions in this forums have shown that it can impact the performance terribly and it doesn't matter if no debugger is attached. Here's a relevant reference http://stackoverflow.com/questions/2195720/why-does-java-code-slow-down-in-debugger. I would recommend that you remove that from the JVM options and keep a watch on the behaviour.

[daniel.wehrle]

jaikiran pai schrieb:

-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

This isn't a good thing to have in production or when you are testing performance. Some past discussions in this forums have shown that it can impact the performance terribly and it doesn't matter if no debugger is attached. Here's a relevant reference http://stackoverflow.com/questions/2195720/why-does-java-code-slow-down-in-debugger. I would recommend that you remove that from the JVM options and keep a watch on the behaviour.

We will give this a try. Thanx.

[robert.d.ritchie]

The project I work on has recently made a similar transition to AS 7.2.0 Alpha Snapshot and then EAP 6.1.0 Alpha.  At a recent customer test event, early April, we had to talk around and mitigate the performance issues we were seeing by doing nightly restarts of JBoss on our project as well.

For the second week of the test event we took out the jdwp listener as described in this post and did not require any JBoss restarts that week, although load was significantly lower during the second week of testing.

However, after 48-72 hours we are still seeing performance degredation so naturally our customer/management does not believe the issue to be resolved.

I am very interested in hearing about the performance improvements yielded by Daniel and Chris after applying this change and curious if they will see the issue as "fixed" and whether or not they still see the slowdown just after a longer period of time as I am seeing on my project or if they will once again go back to month(s) of uptime as they described in their posts.

In any event, this forum thread has provided me at least with enough proof that the configuration change we had made during the test event did in fact fix A performance issue and that what they are seeing now is more than likely a seperate issue(s).

[daniel.wehrle]

Daniel Wehrle schrieb:

jaikiran pai schrieb:

-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n

This isn't a good thing to have in production or when you are testing performance. Some past discussions in this forums have shown that it can impact the performance terribly and it doesn't matter if no debugger is attached. Here's a relevant reference http://stackoverflow.com/questions/2195720/why-does-java-code-slow-down-in-debugger. I would recommend that you remove that from the JVM options and keep a watch on the behaviour.

We will give this a try. Thanx.

We have another production system where the jdwp listener was NOT activated. There are the same problems. So I can exclude this listener bringing the performance problems.

What about the web threads? Could it be that threads in the pool are corrupt but will still be used?