-
1. Re: JBoss AS 7.1.0.Final - performance on RHEL6
jaikiran Apr 23, 2013 3:32 AM (in response to mausbull)What kind of performance problem? On a related note, please use a latest released version instead of 7.1.0.Final.
-
2. Re: JBoss AS 7.1.0.Final - performance on RHEL6
mausbull Apr 23, 2013 3:53 AM (in response to jaikiran)It takes 8 minutes from start to final deployment.
On an Ubuntu system, the same deployment takes 30 seconds.
I can't use the latest version as our application is not compatible with the latest JBoss version.
-
3. Re: JBoss AS 7.1.0.Final - performance on RHEL6
jaikiran Apr 23, 2013 4:04 AM (in response to mausbull)Do you have logs or thread dumps which suggest where it's taking time? Also, what's the output of:
java -version
on that system?
-
4. Re: JBoss AS 7.1.0.Final - performance on RHEL6
ctomc Apr 23, 2013 4:30 AM (in response to mausbull)Hi,
what is hardware like? Virutal? How much ram? what cpu, what kind of disks.
i asume 64bit, but can you be more precise.
what does uname -a say?
what is exact version of your jvm?
looking at your jvm settings. they seem bit strange, but without hardware info i cannot comment if they are wrong.
--
tomaz
-
5. Re: JBoss AS 7.1.0.Final - performance on RHEL6
mausbull Apr 23, 2013 4:46 AM (in response to ctomc)It takes 53804ms to start Thunder.
Then 5mins to do:
10:10:59,710 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-21) JBAS010400: Bound data source [java:jboss/DefaultDS]
10:15:27,355 WARN [org.jboss.as.ejb3.deployment.processors.SessionBeanComponentDescriptionFactory]
Another 6 mins:
10:15:33,785 INFO [javax.enterprise.resource.webcontainer.jsf.config] (MSC service thread 1-47) Initializing Mojarra 2.1.5 (SNAPSHOT 20111202) for context '/SeqBench'
10:21:52,712 INFO [org.primefaces.webapp.PostConstructApplicationEventListener] (MSC service thread 1-47) Running on PrimeFaces 3.4.1
Java: java version "1.7.0_21"
Uname: "Linux 2.6.32-220.17.1.el6.x86_64 #1 SMP Thu Apr 26 13:37:13 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux"
MemTotal: 132141840 kB
Proc: 24 processors a X7542 @ 2.67GHz
JVM Settings: I played around a bit with them to see if I can fix the problem. Didn't improved the performance compared to the standard/shipped settings.
Log is at https://dl.dropboxusercontent.com/u/14071835/jboss_log_2013_04_23.txt
Thanks Stephan
-
6. Re: JBoss AS 7.1.0.Final - performance on RHEL6
jaikiran Apr 23, 2013 4:50 AM (in response to mausbull)This doesn't match with the initial post in this thread which said:
Startup is extremely slow even when nothing is deployed.
It looks like the server starts in 53 odd seconds when nothing is deployed and it's the deploying of the deployment which is taking time.
Having said that, from what you say about the timings, I now have the same questions as Tomaz about the hardware.
-
7. Re: JBoss AS 7.1.0.Final - performance on RHEL6
mausbull Apr 23, 2013 5:03 AM (in response to jaikiran)"53 sec startup" vs "30 sec startup & deployment" -- in my view that's slow. What bugs me is that it's the exact same system that runs on a multitude of OS really well, just on this one it doesn't.
In addition to what I posted in the previous answer, what do you need to know about the hardware?
The server is a HP Proliant DL580 G7
- 4 CPUs with in total 24 cores (Intel X7542, 2.66 GHz)
- 128 GB RAM (5.3 GB/core)
- 1.2 TB local high performance storage (SAS disks, 10k rpm)
- 24 TB local storage (HP-StorageWorks P2000) with 6GBit/sec SATA
-
8. Re: JBoss AS 7.1.0.Final - performance on RHEL6
sfcoy Apr 23, 2013 8:56 AM (in response to mausbull)This sounds like an under resourced virtual machine, and that hardware smells a lot like a VM hosting box.
-
9. Re: JBoss AS 7.1.0.Final - performance on RHEL6
mausbull Apr 24, 2013 10:22 AM (in response to sfcoy)Well, it's neither a virtualbox where JBoss is running in, nor is the server a VM hosting box. It's a server where we're running successfully (other JEE) applications on.
Any updates about the performance issue?
-
10. Re: JBoss AS 7.1.0.Final - performance on RHEL6
ctomc Apr 24, 2013 10:31 AM (in response to mausbull)Hi,
that is bizzare to put it midly....
how long does boot take without deployments? Given that log it also takes about 10 sec+ which is huge.
The only issue about slow(er) boot in 7.1.0 i know about is that it could take up to few extra seconds if there are many network interfaces on system.
but that should not result in such difference.
Can you try two things.
running server with 32bit jvm (with smaller heap sizes)
and can you try with 7.2 http://www.jboss.org/jbossas/downloads/ (EAP6.1.*)
i am curius what is going on.
I am running many instances on RHEL6 on much much slower hardware and have boot times under 3sec... (without deployments)
-
11. Re: JBoss AS 7.1.0.Final - performance on RHEL6
ctomc Apr 24, 2013 10:40 AM (in response to ctomc)Also could you do thread dump when this slowness is happening.
It would be nice to get it in operations that take long time, like when web server is starting
inbetween this two log messages:
10:10:08,705 INFO [org.jboss.as.mail.extension] (MSC service thread 1-2) JBAS015400: Bound mail session [java:jboss/mail/Default]
10:10:58,151 INFO [org.apache.coyote.ajp.AjpProtocol] (MSC service thread 1-47) Starting Coyote AJP/1.3 on ajp--127.0.0.1-18009
you can get thread dump by calling jstack <java process id> (jstack is present in bin folder of jdk)
--
tomaz
-
12. Re: JBoss AS 7.1.0.Final - performance on RHEL6
sfcoy Apr 24, 2013 11:02 AM (in response to mausbull)I agree with Tomaz that it's quite bizzare.
It certainly seems environmental.
This 50 second gap for example:
{code}
10:10:08,705 INFO [org.jboss.as.mail.extension] (MSC service thread 1-2) JBAS015400: Bound mail session [java:jboss/mail/Default]
10:10:58,151 INFO [org.apache.coyote.ajp.AjpProtocol] (MSC service thread 1-47) Starting Coyote AJP/1.3 on ajp--127.0.0.1-18009
{code}
and the fact that it's up to MSC service thread 1-47 suggests maybe CPU starvation?
-
13. Re: JBoss AS 7.1.0.Final - performance on RHEL6
nickarls Apr 25, 2013 1:43 AM (in response to sfcoy)Does the server have free access to the outside internet? Surely the CPU can't show 100% when it's booting, right?
-
14. Re: JBoss AS 7.1.0.Final - performance on RHEL6
mausbull Apr 25, 2013 9:33 AM (in response to nickarls)32bit, 7.2, jstack, "access to outside" --- didn't resolve the issue.
We finally applied solution number 1 -> Reboot.
JBoss is now behaving normally, no performance issues so far.
We believe it could have been the "Leap second bug" [1] that caused it.
Thanks for your help!
Stephan
[1]
http://www.wired.com/wiredenterprise/2012/07/leap-second-bug-wreaks-havoc-with-java-linux/