JBossMQ Fails under load - Exiting on IOE
alchemista Mar 24, 2006 11:40 AMIn a production environment, I've had several cases where my JBossMQ has failed under load. The problem is described below. I was wondering if someone else has solved this problem before, or has any ideas?
The setup is that I have a JBoss application server (1) on Windows sending logs over log4j JMS appender to JBoss application server (2) on Linux.
The server (2) is configured to use the HSQLDB persistence manager, is set to use 1GB ram for the JVM and has the high/max memory set to 80% and 90% of the JVM ram.
The source of the logging data server (1) can send very large amounts of logs over JMS to the logging server (2). This load is fairly high during the day, and occasionally causes the JBossMQ on server (2) to completely fail.
The first errors are of the format below. In some cases reconnection occurs, but in other cases the logging server (2) will continuously refuse connections. When this failure happens, I have to stop the logging server (2), often delete the HSQLDB data directory and then restart logging server (2).
org.jboss.mq.SpyJMSException: Exiting on IOE; - nested throwable: (java.net.SocketException: Connection reset)
at org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:66)
at org.jboss.mq.Connection.asynchFailure(Connection.java:437)
at org.jboss.mq.il.uil2.UILClientILService.asynchFailure(UILClientILService.java:156)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleStop(SocketManager.java:413)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:345)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:235)
at org.jboss.util.stream.NotifyingBufferedInputStream.read(NotifyingBufferedInputStream.java:67)
at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2200)
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2380)
at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2447)
at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2519)
at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2668)
at java.io.ObjectInputStream.readByte(ObjectInputStream.java:864)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:290)
Hardcore failure can give continuous messages:
javax.naming.CommunicationException [Root exception is java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:
java.io.EOFException]
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:707)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:572)
at javax.naming.InitialContext.lookup(InitialContext.java:351)
at com.lmco.lits.james.core.cdf.ReconnectingJMSAppender.initializeConnection(ReconnectingJMSAppender.java:166)
at com.lmco.lits.james.core.cdf.ReconnectingJMSAppender.onException(ReconnectingJMSAppender.java:319)
at org.jboss.mq.Connection$ExceptionListenerRunnable.run(Connection.java:1366)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.rmi.UnmarshalException: Error unmarshaling return; nested exception is:
java.io.EOFException
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:217)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126)
at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)
at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:610)
... 6 more
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2502)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1267)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1628)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1293)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1912)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1836)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1713)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1299)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:215)
... 9 more