-
45. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
ctomc Jul 9, 2013 6:17 PM (in response to daniel.wehrle)Hey,
do you maybe have a example application that can reproduce this on your system?
if so can you share it and i will try to reproduce it on exactly same hw/sw configuration.
-
46. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 10, 2013 6:42 AM (in response to ctomc)Tomaz Cerar schrieb:
Hey,
do you maybe have a example application that can reproduce this on your system?
if so can you share it and i will try to reproduce it on exactly same hw/sw configuration.
Hi Tomaz,
we can only reproduce this problem with our big web application on the production systems hostet in different vmware enviroments. Sorry that we can't share this with you outside our office.
What support options are there from RedHat so solve such a problem? I think there is some case where JBoss gets slower without touching RAM or CPU. This happens in complex web applications using many of the possibilities of JBoss AS and other JBoss projects. In smaller applications there is perhaps no problem.
-
47. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Jul 10, 2013 6:47 AM (in response to daniel.wehrle)I thought this was reproducible even with a simple bubblesort program from a servlet?
-
48. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
turchinc Jul 10, 2013 7:00 AM (in response to jaikiran)Daniel implemented the bubble sort as a health monitor - e.g. when the sort gets slow then our app has slowed down as well. I would not go so far as to assert that the bubble sort running alone as a servlet on EAP is creating slowdowns (though I do not think we have eliminated this as a possibility either). The interesting thing is that the bubble sort only slows when running in the web thread pool. If our app starts the bubble sort via JMS then it is still fast, even when our app and the servlet bubble sort (which is running in a different application context) have slowed down to a crawl.
Summarized: something is slowing down the web threads but not impacting the JMS and also not impacting the JVM as a whole on the server (e.g. when I start the sort as a standalone it is also still fast).
This specific slowdown of the web threads is what we are trying to narrow in upon.
-
49. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Jul 10, 2013 7:12 AM (in response to turchinc)One thing you could perhaps try is configuring the web subsystem to use an (thread) executor of your own instead of the pre-configured thread pool that the web subsystem uses. The web subsystem xsd should have the relevant information on how to do it.
-
50. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 10, 2013 7:28 AM (in response to jaikiran)@Jaikiran: can you give us an example for a production system? I know how to do but i need a proposal for production configuration. There are many ways described in https://access.redhat.com/site/documentation/en-US/JBoss_Enterprise_Application_Platform/6.1/html/Administration_and_Configuration_Guide/sect-Connector_Configuration.html#Define_Thread_Pools_for_HTTP_Connector_in_JBoss_Enterprise_Application_Platform.
I.e. which executer type shoud be used?
-
51. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
jaikiran Jul 10, 2013 7:37 AM (in response to daniel.wehrle)I can't recommend anything for a production system since I haven't tried or looked at the performance aspects myself. IMO, the unbounded-queue-thread-pool should be fine, but that's just a suggestion.
-
52. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
viragegroup Jul 10, 2013 8:05 AM (in response to turchinc)Summarized: something is slowing down the web threads but not impacting the JMS and also not impacting the JVM as a whole on the server (e.g. when I start the sort as a standalone it is also still fast).
This specific slowdown of the web threads is what we are trying to narrow in upon.
We are using quartz scheduler that has it own thread pool. When our app slows down, we also see our quartz jobs slowing down.
-
53. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
ctomc Jul 16, 2013 8:04 AM (in response to daniel.wehrle)Daniel,
what are exact CPUs you use in VM hosts?
I am trying to reproduce your problems but for now without any success
--
tomaz
-
54. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 17, 2013 9:24 AM (in response to ctomc)Tomaz Cerar schrieb:
Daniel,
what are exact CPUs you use in VM hosts?
--
tomaz
Guest: vCPU = 4
Host: CPU = 2x intel XEON E5530
-
55. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
viragegroup Jul 17, 2013 10:20 AM (in response to viragegroup)... I experienced yet another slowdown in one of our demo servers. There was no active session (no one using de demo app) except mine.
The java process was around 10-20% cpu !
I generated a thread dump but there was no activities but
sun.nio.ch.EPollArrayWrapper.poll()
and
java.net.SocketInputStream.socketRead0() (obviously my own connection)
I googled "nio poll high cpu" and found a lot of "known bugs"
https://forums.oracle.com/thread/1146875
https://issues.apache.org/bugzilla/show_bug.cgi?id=52858
https://java.net/jira/browse/GLASSFISH-5148
etc..
any known bug related to nio with jboss ?
We are using the jdk 7u25 ? And also testing open jdk 7(no slow down at the moment).
-
56. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 17, 2013 10:28 AM (in response to jaikiran)jaikiran pai schrieb:
I can't recommend anything for a production system since I haven't tried or looked at the performance aspects myself. IMO, the unbounded-queue-thread-pool should be fine, but that's just a suggestion.
I have defined an executor with unbounded-queue-thread-pool an max thread count of 20 threads. Interesting effect: bubblesort is now fast. Also after 4 days it takes 20 sek. But: our application still slows down.
Hibernate question: is it possible that in hibernate there is a bug which holds objects in the hibernate session if it is closed?
-
57. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
viragegroup Jul 17, 2013 11:37 AM (in response to daniel.wehrle)Daniel Wehrle a écrit:
Hibernate question: is it possible that in hibernate there is a bug which holds objects in the hibernate session if it is closed?
you mean : in the hibernate session even if it is closed?
I'd say somehow if you're using hibernate level 2 cache for instance. Technically, the object reference are not in the session but are hold by hibernate.
-
58. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
daniel.wehrle Jul 18, 2013 4:22 PM (in response to viragegroup)viragegroup schrieb:
you mean : in the hibernate session even if it is closed?
I'd say somehow if you're using hibernate level 2 cache for instance. Technically, the object reference are not in the session but are hold by hibernate.
Last time our application was slow we see slowness in the flush event of hibernate in the isDirty-Method. As i know there are only the objects checked which are in the hibernate session / transaction. Slowness can here only appear if the amount of objects in the session increase. But we have check the transitions. And there are no sessions left open. BUT: we see this behaviour only once.
-
59. Re: Slow performance with JBoss 7.x/EAP 6.1 after 24 hours
viragegroup Jul 23, 2013 10:19 AM (in response to daniel.wehrle)We are seeing the same things: We have a quartz batch that runs during 1-2 minutes when the server is ok. When the server is slow, the batch can take more than 30 minutes. If we generate thread dumps, the only thing visible is hibernate trying to flush objects again and again. The amount of dirty objects is the same as the server is ok or not.
I wonder if it could be a memory fragmentation issue ?
see The Specter of Fragmentation in http://blog.mgm-tp.com/2013/03/garbage-collection-tuning/
or
http://javabook.compuware.com/content/memory/impact-of-garbage-collection-on-performance.aspx
We are testing the "new" G1 garbage collector in one of our test server. Wait and see.