-
30. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Jul 1, 2013 5:22 AM (in response to daniel.wehrle)Daniel Wehrle wrote:
We have another production system where the jdwp listener was NOT activated. There are the same problems. So I can exclude this listener bringing the performance problems.
I would still like to make sure that at least on this instance removing these parameters has fixed the issue. As for the other system, you'll have to provide (again) the entire set of details including the JVM configurations, the standalone configuration and the thread dump.
-
31. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 1, 2013 8:24 AM (in response to daniel.wehrle)For the first system we have disabled debug listener and also nightly restarts.
Jaikiran, please look also at http://stackoverflow.com/questions/14876447/what-could-cause-global-tomcat-jvm-slowdown. There is also described a problem with tomcat and some kind of latency.
-
32. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 2, 2013 4:22 AM (in response to daniel.wehrle)We have a result: disabling the jdwp listener have NO effect. After running 2d 11 h (over weekend with no traffic) the bubblesort is slow.
Bubblesort result:
09:43:36,561 INFO [stdout] (http-/172.16.13.53:8080-21) 281ms took to generate array
10:01:23,436 INFO [stdout] (http-/172.16.13.53:8080-21) 1066875ms took to bubblesort array
-
33. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
robert.d.ritchie Jul 3, 2013 12:44 PM (in response to daniel.wehrle)My project is experimenting with EAP 6.1.0 Final (vice 6.1.0 Alpha) and have been running for 24 hours without a slowdown. I will monitor it for the next few days and post back on Friday whether or not we see a slowdown after 72 hours.
Comparing the standalone.bat from Final vs Alpha there is an interesting delta/inconsistency but looks like it would only impact if you passed in --debug as a command line argument:
In Alpha there is the following code block:
rem Set debug settings if not already set
if "%DEBUG_MODE%" == "true" (
echo "%JAVA_OPTS%" | findstr /I "\-agentlib:jdwp" > nul
if errorlevel == 1 (
echo Debug already enabled in JAVA_OPTS, ignoring --debug argument
) else (
set "JAVA_OPTS=%JAVA_OPTS% -agentlib:jdwp=transport=dt_socket,address=%DEBUG_PORT%,server=y,suspend=n"
)
)
and in the Final version the block is changed by flipping the contents of the if and else:
rem Set debug settings if not already set
if "%DEBUG_MODE%" == "true" (
echo "%JAVA_OPTS%" | findstr /I "\-agentlib:jdwp" > nul
if errorlevel == 1 (
set "JAVA_OPTS=%JAVA_OPTS% -agentlib:jdwp=transport=dt_socket,address=%DEBUG_PORT%,server=y,suspend=n"
) else (
echo Debug already enabled in JAVA_OPTS, ignoring --debug argument
)
)
...This is probably not the cause of anything, but I found the discrepancy interesting between Alpha and Final.
-
34. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
robert.d.ritchie Jul 5, 2013 8:54 AM (in response to robert.d.ritchie)63 hours since startup and still have not seen a slowdown in performance with EAP 6.1.0 Final.
bubble sort time: 17.665 seconds
Daniel, have you experimented at all with 6.1.0 Final?
-
35. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 5, 2013 10:12 AM (in response to robert.d.ritchie)Robert, not yet. I have waited for your result. I will try it.
-
36. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
robert.d.ritchie Jul 5, 2013 11:39 AM (in response to daniel.wehrle)Excellent,
I will keep my experiment running as well and I plan to setup two dedicated machines side by side (one with Alpha and one with Final) and conduct load testing on them.
For consistency in the experiments I am running with the remote debugger listener disabled as well.
Will post more Monday with some results.
-
37. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 8, 2013 3:29 AM (in response to robert.d.ritchie)Result from me: JBoss slows down also with EAP 6.1
-
38. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
robert.d.ritchie Jul 9, 2013 9:36 AM (in response to daniel.wehrle)I am up to day 6 of uptime now and am experiencing the same slowdown with EAP 6.1.0 Final as well.
-
39. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Jul 9, 2013 9:38 AM (in response to robert.d.ritchie)I don't know what else to suggest, but have you guys taken a look at the GC rates and tried to compare its timings?
-
40. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 9, 2013 9:46 AM (in response to jaikiran) -
41. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
viragegroup Jul 9, 2013 12:25 PM (in response to daniel.wehrle)Hi all,
We had the exactly same issue with a webapp on jboss 7 on a VM from vmware: the overall performance of the JVM were degraded over time
Did you check on a real machine ? We notice a very good improvement.
Did you check VMvare best practices ?
-
42. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 9, 2013 12:39 PM (in response to viragegroup)Yes. We checked the best practices. There is enough reserved memory for the vm and the jvm. We cant see any memory or cpu problems profiling the application. But what we see is a slowdown of same method calls after a while.
A real machine is little bit difficult. Whole hosting environement is virtual and also our client hosting enviroments are virtual.
Did you solve the problem on your vm or did you switch to real machine? Do you use windows or linux?
-
43. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
robert.d.ritchie Jul 9, 2013 12:49 PM (in response to daniel.wehrle)All of my performance issues have been witnessed on real machines (Windows Server 2008 and Windows 7 machines).
-
44. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
viragegroup Jul 9, 2013 12:59 PM (in response to daniel.wehrle)Did you solve the problem on your vm or did you switch to real machine?
In some environnements, we switched on a real machine (when possible). In others, we set up nightly restarts.
Do you use windows or linux?
Mainly windows 2008 server. We also encountered performance problem on linux but I'm not sure that it was the same problem.