JBOSS crashes, possibly database related; Exception: recv fa
Hello everybody,
we are running a commercial application on JBOSS 4.0.5GA (about 2000 users, 20 of them using the application at a time on average). OS is MS Web Server 2008, the persistency layer is provided by MySQL 5.1.35.
Up to now, everything has been running without any problems. Recently, however, there have been more and more frequent server crashes. It appears to happen when the server load increases beyond a certain point, but I can't definitely tell whether there is a real causation between the load situation and the crashes.
Having analyzed the server logs, it appears to me that the crashes are related to the database. However, I'm not a specialist on that area. Here is a log excerpt:
(...) com.mysql.jdbc.CommunicationsException MESSAGE: Communications link failure due to underlying exception: ** BEGIN NESTED EXCEPTION ** java.net.SocketException MESSAGE: Software caused connection abort: recv failed STACKTRACE: java.net.SocketException: Software caused connection abort: recv failed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:147) at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:113) at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:160) at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:188) at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1910) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2304) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2803) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:812) at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3269) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1182) at com.mysql.jdbc.Connection.createNewIO(Connection.java:2670) at com.mysql.jdbc.Connection.<init>(Connection.java:1531) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:171) at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:565) at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:250) at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:529) at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:341) at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:301) at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:396) at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:842) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:88) at org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand.execute(JDBCAbstractQueryCommand.java:225) at org.jboss.ejb.plugins.cmp.jdbc.JDBCAbstractQueryCommand.execute(JDBCAbstractQueryCommand.java:144) at org.jboss.ejb.plugins.cmp.jdbc.JDBCFindByPrimaryKeyQuery.execute(JDBCFindByPrimaryKeyQuery.java:155) at org.jboss.ejb.plugins.cmp.jdbc.JDBCFindEntityCommand.execute(JDBCFindEntityCommand.java:61) at org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.findEntity(JDBCStoreManager.java:604) at org.jboss.ejb.plugins.CMPPersistenceManager.findEntity(CMPPersistenceManager.java:315) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.findEntity(CachedConnectionInterceptor.java:236) at org.jboss.ejb.EntityContainer.findSingleObject(EntityContainer.java:1099) at org.jboss.ejb.EntityContainer.find(EntityContainer.java:728) at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.jboss.invocation.Invocation.performCall(Invocation.java:359) at org.jboss.ejb.EntityContainer$ContainerInterceptor.invokeHome(EntityContainer.java:1126) at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:105) at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invokeHome(EntitySynchronizationInterceptor.java:203) at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invokeHome(CachedConnectionInterceptor.java:189) at org.jboss.ejb.plugins.AbstractInterceptor.invokeHome(AbstractInterceptor.java:105) at org.jboss.ejb.plugins.EntityInstanceInterceptor.invokeHome(EntityInstanceInterceptor.java:134) at org.jboss.ejb.plugins.EntityLockInterceptor.invokeHome(EntityLockInterceptor.java:76) at org.jboss.ejb.plugins.EntityCreationInterceptor.invokeHome(EntityCreationInterceptor.java:43) at org.jboss.ejb.plugins.CallValidationInterceptor.invokeHome(CallValidationInterceptor.java:56) at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:125) at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:350) at org.jboss.ejb.plugins.TxInterceptorCMT.invokeHome(TxInterceptorCMT.java:161) at org.jboss.ejb.plugins.SecurityInterceptor.invokeHome(SecurityInterceptor.java:145) at org.jboss.ejb.plugins.LogInterceptor.invokeHome(LogInterceptor.java:132) at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invokeHome(ProxyFactoryFinderInterceptor.java:107) at org.jboss.ejb.EntityContainer.internalInvokeHome(EntityContainer.java:521) (..)
This is the second type of exception that shows up in the log during crashes:
(...) Last packet sent to the server was 1 ms ago. STACKTRACE: com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception: ** BEGIN NESTED EXCEPTION ** java.io.EOFException STACKTRACE: java.io.EOFException at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1913) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2304) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2803) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:812) at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:3269) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1182) at com.mysql.jdbc.Connection.createNewIO(Connection.java:2670) at com.mysql.jdbc.Connection.<init>(Connection.java:1531) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:266) at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:171) (...)
I have searched for the cause of these errors on the web myself, but the only related issue where they appear seems to be if the database connections are unused for a long time and have become stale. These circumstances certainly do not apply in this case. Interestingly, the MySQL error log does not contain anything that would point to a database problem, so I assume that JBOSS somehow can't connect to the DB anymore.
If anybody could give me a pointer to the real cause of the problem, I would be really happy.
Best regards,
Caspar
