JBoss 7.1.1.Final slow call local Ejb method on Windows Server 2008 R2 Standard 64.bit

[hurdes]

Hellow I ran into a problem that Windows Server 2008 R2 Standard 64 bit application does not manage to save my data. With tracing, I measure a very long time over the local call to the EJB method (over local interface). On windows 7 and same JBoss 7.1.1 and JDK me it takes 0-10ms on Windows Server that is between 600 to 2000 ms (This is not a one-time call, but multiple calls to the already established connection with EJB). You have met with similar behavior?

Runtime environment: Windows Server 2008 R2 Standard 64.bit + JBoss 7.1.1.Final + JDK 1.6.0_45 64bit.

Thanks for any answers or directions to solve the problem.

Jaroslav Hurdes

[milko.krastev]

I experience the same issue. Calls to EJBs take forever. And that happens only with Server 2008 R2. No problems with Windows 7 or Server 2007. Using JBoss 7.1.3. No idea how to solve it so far.

[ctomc]

jvm settings used?

[milko.krastev]

I've tried with all kinds of settings, these are the current ones:

-XX:+UseCompressedOops -Dprogram.name=standalone.bat -Xms1024M -Xmx1024M -XX:MaxPermSize=256M -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -Dsun.rmi.dgc.client.gcInterval=360000 -Dsun.rmi.dgc.server.gcInterval=360000 -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.server.default.config=standalone.xml -Dorg.jboss.boot.log.file=C:\Develop\Java\jboss-as-7.1.3.Final\standalone\log\boot.log -Dlogging.configuration=file:C:\Develop\Java\jboss-as-7.1.3.Final\standalone/configuration/logging.properties

[wdfink]

Forever means longer than what was said in the initial issue (2000ms).

Do you see any entry-exit points from your code or can add additional debug?

Did the server respond for other requests?

[milko.krastev]

Q: Forever means longer than what was said in the initial issue (2000ms).

A: "Forever" is of course strongly exaggerated. The system is supposed to process market quotes using EJBs in an ACID compliant manner as fast as possible < 10 ms average case scenario. A very slow VPS with one core and 2 GB of RAM achieves that. My laptop does it better. A dedicated server with 16GB of RAM, 4 cores, running Server 2008 R2 fails really badly. Average case varies wildly from 100 ms to over 4000 ms, never better than 90 ms. I introduced System.nanoTime() checkpoints to measure what takes what amount of time. The database query is fast, the JMS is relatively fast, the EJB invocations are terribly slow.

Q: Do you see any entry-exit points from your code or can add additional debug?

A: We have tested on Server 2003, 2007, Window 7. It runs fine. EJB calls are slow on Server 2008 R2 only. There is no debug info I can provide. The log is clean.

Q: Did the server respond for other requests?

A: Throughput is not an issue. It responds OK, but there is some nasty lag in these EJB invocations that increases, decrease, then increases again in an unpredictable manner (only on Server 2008 R2).

[ctomc]

JVM settings look ok, but given that you have more than enough ram you could omit compressedoops.

G1 GC is also not stable yet in JDK6, I would really recommend upgrading to JDK7, as there should be improvements in any case.

Also bit of site note, in jdk7u10 and up compressedoops are automatically turned on by jvm given system resources.

You are mentioning WildFly, so you are using AS7.1.1 or WildFly if so which version? Or are you just using mix of servers on different VM/hardware.

I would also check your firewall settings on Windows 2K8 server, and try turning off firewall completely and repeat the test, just to make sure FW is not the issue.

If noting of this helps, profiler is your only option, attach to app server JVM with profiler(yourkit, jprofiler) and monitor how long it takes to process the request and where.

You can post the CPU profile trace here, for us to analyze.

I still think problem is more environmental but hard to conclude that without hard evidence.

[milko.krastev]

I was using JBoss 7.1.3 final tag built from sources. When downgrading to 7.1.2 also built from sources the problem with the high latency disappeared. It must be a JBoss bug in the 7.1.3 branch.

[wdfink]

What if you use AS7.2 (EAP 6.1.Alpha) or later?