JBossDeveloper

Issues that disappear magically have the tendency the re-appear just as magically. When you least can afford it

Hi

we exactly have the same problem: jboss / java is running fast directly after start of jboss. After some time (mostly after roughly one day) jboss decreases in runtime performance, and is up to 100 times slower than before. Every operation is just taking more (CPU) time.

We do not have any abnormal disk IO, nor network IO, no memory paging, etc. Also, our java heap is not full - garbage collection is running at a normal rate and GC-time does not increase over time. Even testing different GC-strategies does not help ("best" solution here is to use the brand new G1GC - the delay between jboss start and decrease of performance is a few hours longer).

Our environment is:

Host: RHEL 6.3 64bit with KVM

Guest running the jboss: CentOS 6.3 64bit

Java: 32bit Oracle VM 1.7.0_10, server mode, 2 GB heap

Everytime the decrease in performace started, we see roughly 100% CPU utitlization for the jboss process in "top". As we can not track this high CPU utilization to our own code, or to a high GC activity, we used the command "perf top" to have a deeper look into the the actual CPU-eating function:

perf top

{code}
70,11%  libjvm.so           [.] CodeHeap::largest_free_block() const
{code}

Using google does not help us to resolve the "why is java constantly searching for large free blocks"

- we do have plenty of free memory in HOST, and in GUEST,

- we have roughly 50% heap used, and 50% heap free, with "normal" GC-activity in java

- there could be some memory fragmentation inside java itself, but this should cleaned every time GC is running (high fragmentation / a lot of small objects should increase GC time - but jvisualvm or other tools do not show a higher GC-activtity)

We also experimented using linux large memory pages: same behavior...

Next we will try jboss-7.1.3, and if this does not help, 7.1.4-SNAPSHOT. A proper solution to this problem would be highly appreciated, nevertheless.

If there is more debugging informations required to analyze this, i would like to help.

With kind regards,

Stefan

Stefan Schueffler wrote:

 

 

Next we will try jboss-7.1.3, and if this does not help, 7.1.4-SNAPSHOT. A proper solution to this problem would be highly appreciated, nevertheless.

 

I don't think we have been able to figure out what the problem really is. Please do try against 7.1.3/7.1.4-SNAPSHOT. If it's fixed there then that would be good news. Else it will require more analysis and debugging.

After running a few days with 7.1.3.Final (checked out the source code from the corresponding tag), everything seems to work fine. We could not reproduce the slow down in the last few days anymore using this jboss version.

Perhaps that bug report to JSF is linked with your problems : http://java.net/jira/browse/JAVASERVERFACES-2176 (fixed in 2.1.13, so using JBoss 7.2.0.Final will fix it as it uses 2.1.18).

Anyone every figure out a resolution to this? We see the exact same problem. Two different instances of JBoss on the same box can run drastically different even though they are sitting idle except for a little loop of java code I wrote to test performance. Restarting the app always gets rid of the problem . . . (at least until it happens again). Currently JBOSS jvm one takes 450ms to run a loop, another JVM running jboss on the same box takes 44 SECONDS . . .

You should start a new thread for your question and not post to an old thread

Most time the problem is related to memory (GC) issues, so you should enable GC logging for the JVM first.

I posted here because it sounds exactly like this problem. It's not GC related. We've been troubleshooting this problem for months.

Jeremiah Pierucci wrote:

 

I posted here because it sounds exactly like this problem. It's not GC related. We've been troubleshooting this problem for months.

Exactly the same problem? Also using 7.1.x version of AS? Did you try upgrading?

You should always give more information than just "me too" this is beyond unhelpful in situations like this.

System details are important, jvm parameters are important, that all is a factor.

Look, I'm not asking anyone to analyze our problem. I'm just wondering if Stefan learned anything more because, yes, it sounds like the EXACT same problem, and yes, we are on the same version.

Hi Jeremiah,

did you try to use an earlier version of Java, i.e., 6u45 (if that is possible in your case)?

Best regards

No, we usually go the other direction . Have you had some experience that shows that java 6 fixed something?

I'm asking because we experienced a similar behavior when upgrading to Java 7. The slow down didn't occur under Java 6, but after changing the Java version. Would be intersting to know if the problem is the same.

Hi,

We are having the same problem: Jboss 7.1.1 becomes very slow over time. We don't have any memory or cpu problem.

Where can I download Jboss 7.1.4 to test with this version ?