-
60. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
viragegroup Jul 29, 2013 1:26 PM (in response to viragegroup)Bad guess.
Our web app seems even slower with G1 gc.
Our web app seems ok 2 or 3 days and at a given moment all seems slower. We have a batch job that can run on demand. When all is ok the job runs in 2 minutes and when it's not ok, it's run in 1 hour.
We compared before / after with visualvm there is nothing obvious.
- cpu is 20% (one core i guess) during all the job execution. When it's ok and the job finished, the cpu goes down to 0% but when it's not ok, it's take 1 hour and after the job completion, the cpu keeps jumping around 10% altough there no user connected (perhaps because of the visualvm polling ?)
- ram is around 1Go (Xms = Xmx = 2Go). If we force GC with visualvm, the ram go down to 350Mo but the slowness remains
- During all the job execution, When all is ok, the GC (visible with visualgc plugin) does a lot of collections (1 /sec) to clean up the eden space but when it's not ok, the GC activity slow down too (1 / minute) although the job is running.
- Thread dumps are the same in both case, our job is stuck in AbstractFlushingEventListener.flushEntities() from hibernate (the job does mass updates) and especially
AbstractVisitor.includeProperty()
- There is no blocked threads (the job is always in "running" state)
- We also checked the allocated bytes/sec for the job's thread: When it's ok: 150Mo/s. If not ok, 5Mo/s
- When it's not ok, there is a class (yes a class not an instance) that appears to be one of the biggest object (by retained size) :
class org.apache.pdfbox.pdmodel.font.PDFont that comes with tika (and lucene). There is also a
org.jboss.modules.ModuleClassLoader instance that take up to 18Mo (because of the PDFont class surely) and 4300 classes. is this a problem ?
any clue ?
-
61. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Jul 31, 2013 6:43 AM (in response to viragegroup)Virage Group,
Could you please start a new thread and include the details that you have posted in this last post, into that new thread? I do have a reply but this thread has now gone into 5 pages with multiple users talking about various different things. In your case, it seems to be something very Hibernate/JPA specific. So I would like to reply to that in a separate thread.
-
62. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 31, 2013 6:54 AM (in response to jaikiran)@jaikiran: see my last comment. We have also seen problems with hibernate: "Last time our application was slow we see slowness in the flush event of hibernate in the isDirty-Method."
-
63. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 31, 2013 7:00 AM (in response to daniel.wehrle)@all: what we also have seen is a really huge amount of hashmaps in class org.jboss.as.ee.component.ViewService$DefaultViewInstanceFactory. We think that there are more then 14.000 Hashmaps. If there is an iteration over all entries the systems becomes slow. Perhaps there are also such Hashmap-Tree in other parts of JBoss.
-
64. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Jul 31, 2013 7:35 AM (in response to daniel.wehrle)Daniel Wehrle wrote:
@jaikiran: see my last comment. We have also seen problems with hibernate: "Last time our application was slow we see slowness in the flush event of hibernate in the isDirty-Method."
I think it might have to do with the flush mode set on the entitymanager. Read this question http://stackoverflow.com/questions/1791050/hibernate-queries-slow-down-drastically-after-an-entity-is-loaded-in-the-session to understand what I mean. Depending on your usecase/application, you can perhaps change the flushmode on the entitymanager to something more performant. Before doing that, do read the explanation about how flush mode plays a role in JPA here http://www.objectdb.com/java/jpa/query/setting#Flush_Mode_setFlushMode_
-
65. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Aug 1, 2013 11:29 AM (in response to daniel.wehrle)Information to my last post: this is a jrebel problem in dev environment. Nothing to do with our slowness on production server.
-
66. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Aug 1, 2013 12:04 PM (in response to jaikiran)@jaikiran: but why is the System not slow after restart? There can only be a problem with flush if there rests entities in the em after commit.
-
67. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Aug 2, 2013 2:06 AM (in response to daniel.wehrle)Daniel Wehrle wrote:
@jaikiran: but why is the System not slow after restart?
Because the entitymanager (session) hasn't yet accumulated many objects within itself? By the way, I've no clue what the problem is. I'm just throwing out some ideas.
-
68. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Aug 19, 2013 3:33 AM (in response to daniel.wehrle)Hi, any news from the thread participants? Perhaps any solution?
We monitor our application with yourkit. If it is very slow there are interrupts in the threads. The question is if this is a problem with yourkit or is this a problem with the application or java.
-
69. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
viragegroup Aug 20, 2013 4:03 AM (in response to daniel.wehrle)Hi all,
We have removed the following jvm option:
-XX:+TieredCompilation
and replaced it by -server
Our web app has been running for 15 days without slowness
-
70. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Aug 20, 2013 4:53 AM (in response to viragegroup)As I understand the JVM Faq correctly on 64bit-Systems there is only the server jvm. The default setting is "-server".
http://www.oracle.com/technetwork/java/hotspotfaq-138619.html#compiler_types
JVM FAQ: For 64-bit, only the server system is included. On Solaris, the 64-bit JRE is an overlay on top of the 32-bit distribution. However, on Linux and Windows, it's a completely separate distribution.
I will disable TieredCompilation.
-
71. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Aug 20, 2013 5:05 AM (in response to daniel.wehrle)@all: please see at https://issues.jboss.org/browse/AS7-4547. If this is realy a performance problem why it is enabled in Jboss by default?
-
72. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Aug 20, 2013 5:20 AM (in response to daniel.wehrle)Daniel Wehrle wrote:
@all: please see at https://issues.jboss.org/browse/AS7-4547. If this is realy a performance problem why it is enabled in Jboss by default?
It isn't enabled anymore after that bug was fixed.
-
73. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Aug 20, 2013 5:27 AM (in response to jaikiran)From standalone.bat of EAP 6.1
if not "%PRESERVE_JAVA_OPTS%" == "true" (
rem Add tiered compilation, if supported (64 bit VM), and not overriden
echo "%JAVA_OPTS%" | findstr /I "\-XX:\-TieredCompilation \-client" > nul
if errorlevel == 1 (
"%JAVA%" -XX:+TieredCompilation -version > nul 2>&1
if not errorlevel == 1 (
set "JAVA_OPTS=-XX:+TieredCompilation %JAVA_OPTS%"
)
)
)
-
74. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Aug 20, 2013 6:22 AM (in response to daniel.wehrle)Ah! Looks like the change only made it to the standalone.sh (and not the .bat) https://source.jboss.org/changelog/JBossAS?cs=c525e538fda12bde76b44a6db4f61d55105c1cea. Please file a JIRA here https://issues.jboss.org/browse/JBEAP