This just happened again after one hour. Any ideas?
This appears to be related to open connections.
mod_jk was holding connections open to the JBoss web server on 8009, and the number of ESTABLISHED connections kept growing until tomcat started rejecting new connections. Some of the connections would get reused, but not that many.
I set the connection_pool_timeout in the mod_jk config to 10 seconds. This keeps the number of established connections down around 50-60. However, the connections that have timed out of the pool, appear to persist forever.
The connection from the mod_jk TO JBossWeb sticks in FIN_WAIT1 or FIN_WAIT2:
10.10.77.194:57587 10.10.77.194:8009 FIN_WAIT1
The connection view from JBossWeb gets stuck in CLOSE_WAIT:
10.10.77.194:8009 10.10.77.194:57587 CLOSE_WAIT
Once the number of connections hits 1024 (or thereabouts) everything falls apart, of course, and I have to kill -9 jboss and start it up again.
This never happened with JBoss 4.0.5. Apache and mod_jk haven't changed at all, so this has to be a change between Tomcat 5.5 or JBossWeb. I can't be the only person to run into this.....
FYI: I've switched over to mod proxy_http connecting to the http connector in JBossWeb and that works fine, so the issue is specific to the AJP connector in the JBossWeb side. Either a code change or a config change from tomcat 5.5 (in Jboss 4.0.5) to JBossWeb.
If anyone has the same issue, I've opened a Jira ticket on it JBWEB-82.
did changing the connectionTimeout to 600000 for the ajp connector help as recommended in http://jira.jboss.com/jira/browse/JBWEB-82 ? I am experiencing the same issue, except I was NOT seeing connections stuck in a CLOSE_WAIT state until I made the recommended connectionTimeout change to 600000.
I haven't tried changing back to AJP yet. I hate impacting my users with errors when I don't absolutely need to.
Your statement makes me even more leery.
I've had the same problem since upgrading to JBoss 4.2.1 from 4.0.4 Adding the connection timeout in mod_jk helped it not happen as frequently, but it still happens. Changing the timeout for the ajp connection in the server.xml file for jboss web did nothing. I even made sure to install the newest version of the jk connector. I don't see any alternative but to go back to 4.0.x
Well, you can always use mod proxy_http, but yes, I still haven't been able to get AJP working properly under load.
I get the same problem. When we run our functional test suite it hangs after several tests have run.
I've updated the associated Jira bug.
I also get the same problem.
Having looked at http://jira.jboss.com/jira/browse/JBWEB-82 i'm wondering how could i practically use the workaround? i mean, what does it mean to use the jbossnative?
BTW, is it possible to run jboss-4.2.2 with normal tomcat instead of jboss-web?
I was earlier running with following versions:
Apache - 2.0.55
mod_jk - 1.2.20
JBoss - 4.0.3sp1
After thr upgrade of JBoss to 4.2.2GA, I have this same issue.
So, Whats teh work around or solution?
Reverting back to 4.0.3sp1 is a HUGE impact, especially when we have new applications expecting to use EJB3 and SEAM technologies.