Well I have been having a problem with our JBoss server stop responding every couple of days. I think it is that we are running out of threads, but I am not certain. I have done some thread dumping and noticed that we run about 100 or so threads during normal operations, but while it is hanging we have threads hanging all over the place. Most of these seem to be Tomcat just waiting for something or another, but not sure why. Anyway the symptoms I am seeing are these
14:47:53,658 WARN [TransactionImpl] Transaction TransactionImpl:XidImpl
[FormatId=257, GlobalId=triton//74009, BranchQual=] timed out. status=STATUS_ACTIVE
ERROR [STDERR] java.net.SocketException: Invalid argument
14:37:19,189 ERROR [STDERR] at java.net.PlainSocketImpl.socketSetOption(Native Method)
14:37:19,190 ERROR [STDERR] at java.net.PlainSocketImpl.setOption(PlainSocketImpl.java:240)
14:37:19,190 ERROR [STDERR] at java.net.Socket.setTcpNoDelay(Socket.java:771)
14:37:19,191 ERROR [STDERR] at org.apache.tomcat.util.net.PoolTcpEndpoint.setSocketOptions(PoolTcpEndpoint.java:468)
we are also seeing a fair amout of IO exceptions via "java.net.SocketException: Broken pipe" and "java.io.InterruptedIOException"
Anyway it seems like there is no longer any resources to try and get to the stateless session beans that we have (fails to get a thread). We do NOT run any database, instead all of our data comes from sockets that we open in our own pool. These pools do not seem to be full, so that doesn's seem to be the problem.
Other things of interest are, we started seeing this after we moved to a loadbalancer talking directly to Tomcat in JBoss instead of going through apache like we use to. We also increased the number of threads from 100 to 500 in the jboss-service.xml which seemed to help a little.
Thanks for your help
