-
30. Re: JBoss 7.1.1.Final/JVM slows over time
nickarls Jan 15, 2013 2:27 AM (in response to lukascz)Issues that disappear magically have the tendency the re-appear just as magically. When you least can afford it
-
31. Re: JBoss 7.1.1.Final/JVM slows over time
schueffi Jan 16, 2013 3:19 AM (in response to lukascz)Hi
we exactly have the same problem: jboss / java is running fast directly after start of jboss. After some time (mostly after roughly one day) jboss decreases in runtime performance, and is up to 100 times slower than before. Every operation is just taking more (CPU) time.
We do not have any abnormal disk IO, nor network IO, no memory paging, etc. Also, our java heap is not full - garbage collection is running at a normal rate and GC-time does not increase over time. Even testing different GC-strategies does not help ("best" solution here is to use the brand new G1GC - the delay between jboss start and decrease of performance is a few hours longer).
Our environment is:
Host: RHEL 6.3 64bit with KVM
Guest running the jboss: CentOS 6.3 64bit
Java: 32bit Oracle VM 1.7.0_10, server mode, 2 GB heap
Everytime the decrease in performace started, we see roughly 100% CPU utitlization for the jboss process in "top". As we can not track this high CPU utilization to our own code, or to a high GC activity, we used the command "perf top" to have a deeper look into the the actual CPU-eating function:
perf top
{code}
70,11% libjvm.so [.] CodeHeap::largest_free_block() const
{code}
Using google does not help us to resolve the "why is java constantly searching for large free blocks"
- we do have plenty of free memory in HOST, and in GUEST,
- we have roughly 50% heap used, and 50% heap free, with "normal" GC-activity in java
- there could be some memory fragmentation inside java itself, but this should cleaned every time GC is running (high fragmentation / a lot of small objects should increase GC time - but jvisualvm or other tools do not show a higher GC-activtity)
We also experimented using linux large memory pages: same behavior...
Next we will try jboss-7.1.3, and if this does not help, 7.1.4-SNAPSHOT. A proper solution to this problem would be highly appreciated, nevertheless.
If there is more debugging informations required to analyze this, i would like to help.
With kind regards,
Stefan
-
32. Re: JBoss 7.1.1.Final/JVM slows over time
jaikiran Jan 16, 2013 3:26 AM (in response to schueffi)Stefan Schueffler wrote:
Next we will try jboss-7.1.3, and if this does not help, 7.1.4-SNAPSHOT. A proper solution to this problem would be highly appreciated, nevertheless.
I don't think we have been able to figure out what the problem really is. Please do try against 7.1.3/7.1.4-SNAPSHOT. If it's fixed there then that would be good news. Else it will require more analysis and debugging.
-
33. Re: JBoss 7.1.1.Final/JVM slows over time
schueffi Jan 21, 2013 4:33 AM (in response to jaikiran)After running a few days with 7.1.3.Final (checked out the source code from the corresponding tag), everything seems to work fine. We could not reproduce the slow down in the last few days anymore using this jboss version.
-
34. Re: JBoss 7.1.1.Final/JVM slows over time
lukascz Jan 31, 2013 3:32 PM (in response to schueffi)Ok, so it's not that great.
We were running the server (7.1.4 SNAPSHOT) for more than 3 weeks without restart and it was fine but today the slow down showed up again. But this time it was much less severe than with 7.1.1.Final (approximately 3-4 times slower, with 7.1.1. it was up to 20 times slower). Also it took much longer for this to happen (with 7.1.1 it usually showed up after one day).
Lukas
-
35. Re: JBoss 7.1.1.Final/JVM slows over time
aogier Mar 27, 2013 3:02 PM (in response to lukascz)Perhaps that bug report to JSF is linked with your problems : http://java.net/jira/browse/JAVASERVERFACES-2176 (fixed in 2.1.13, so using JBoss 7.2.0.Final will fix it as it uses 2.1.18).
-
36. Re: JBoss 7.1.1.Final/JVM slows over time
climbbike Mar 14, 2014 1:27 PM (in response to lukascz)Anyone every figure out a resolution to this? We see the exact same problem. Two different instances of JBoss on the same box can run drastically different even though they are sitting idle except for a little loop of java code I wrote to test performance. Restarting the app always gets rid of the problem . . . (at least until it happens again). Currently JBOSS jvm one takes 450ms to run a loop, another JVM running jboss on the same box takes 44 SECONDS . . .
-
37. Re: JBoss 7.1.1.Final/JVM slows over time
wdfink Mar 15, 2014 1:20 PM (in response to climbbike)You should start a new thread for your question and not post to an old thread
Most time the problem is related to memory (GC) issues, so you should enable GC logging for the JVM first.
-
38. Re: JBoss 7.1.1.Final/JVM slows over time
climbbike Mar 17, 2014 5:42 PM (in response to wdfink)I posted here because it sounds exactly like this problem. It's not GC related. We've been troubleshooting this problem for months.
-
39. Re: JBoss 7.1.1.Final/JVM slows over time
ctomc Mar 18, 2014 6:35 AM (in response to climbbike)Jeremiah Pierucci wrote:
I posted here because it sounds exactly like this problem. It's not GC related. We've been troubleshooting this problem for months.
Exactly the same problem? Also using 7.1.x version of AS? Did you try upgrading?
You should always give more information than just "me too" this is beyond unhelpful in situations like this.
System details are important, jvm parameters are important, that all is a factor.
-
40. Re: JBoss 7.1.1.Final/JVM slows over time
climbbike Mar 18, 2014 11:34 AM (in response to ctomc)Look, I'm not asking anyone to analyze our problem. I'm just wondering if Stefan learned anything more because, yes, it sounds like the EXACT same problem, and yes, we are on the same version.
-
41. Re: JBoss 7.1.1.Final/JVM slows over time
cejay Mar 19, 2014 10:42 AM (in response to climbbike)Hi Jeremiah,
did you try to use an earlier version of Java, i.e., 6u45 (if that is possible in your case)?
Best regards
-
42. Re: JBoss 7.1.1.Final/JVM slows over time
climbbike Mar 19, 2014 2:26 PM (in response to cejay)No, we usually go the other direction . Have you had some experience that shows that java 6 fixed something?
-
43. Re: JBoss 7.1.1.Final/JVM slows over time
cejay Mar 20, 2014 4:49 AM (in response to climbbike)I'm asking because we experienced a similar behavior when upgrading to Java 7. The slow down didn't occur under Java 6, but after changing the Java version. Would be intersting to know if the problem is the same.
-
44. Re: JBoss 7.1.1.Final/JVM slows over time
icemaker Mar 20, 2014 7:27 AM (in response to lukascz)Hi,
We are having the same problem: Jboss 7.1.1 becomes very slow over time. We don't have any memory or cpu problem.
Where can I download Jboss 7.1.4 to test with this version ?