OILSeverILService Connection failure and log4j
neillamka Apr 28, 2006 4:15 PMI have been trying to track down a problem where for some reason we start getting SpyJMSExceptions that once they start continue until Jboss is restarted. In order to try and find where the problem is I turned up log4j logging levels.
What I found when I did that was that an immediate exception was thrown followed by a set of "receiver connections" being closed. After this which the SpyJMSExceptions started and will continue.
Now, of course this could just be a just a coincidence but I don't think so so I'm trying to find out a couple of things.
1. Why would changing logging levels cause a connection failure
and
2. Is there some way to detect/catch this event and try to restart connections so that we can stop the follow on SpyJMS Exceptions.
2006-04-28 08:48:58,833 INFO [TP-Processor5] [ChannelSocket] connection timeout reached
2006-04-28 08:49:04,874 INFO [TP-Processor7] [ChannelSocket] connection timeout reached
2006-04-28 08:51:09,164 INFO [Thread-1] [Log4jService$URLWatchTimerTask] Configuring from URL: resource:log4j.xml
2006-04-28 08:51:09,558 DEBUG [Thread-1] [Log4jService] Removed System.out adapter
2006-04-28 08:51:09,558 DEBUG [Thread-1] [Log4jService] Removed System.err adapter
2006-04-28 08:51:10,253 WARN [OIL Worker-9] [OILServerILService] Connection failure (1).
java.net.SocketTimeoutException: Read timed out
at java.net.AbstractSocketImpl$1.read(II)I(Optimized Method)
at jrockit.io.NativeIOInputStream.read(I[BI)I(Unknown Source)
at jrockit.io.NativeIOInputStream.read([BII)I(Unknown Source)
at java.io.BufferedInputStream.fill()V(Optimized Method)
at java.io.BufferedInputStream.read()I(Optimized Method)
at java.io.ObjectInputStream$PeekInputStream.peek()I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Z)I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.refill()V(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.read()I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
at java.io.ObjectInputStream.readByte()B(Unknown Source)
at org.jboss.mq.il.oil.OILServerILService$Client.run()V(OILServerILService.java:206)
at java.lang.Thread.run()V(Unknown Source)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Optimized Method)
2006-04-28 08:51:10,261 WARN [OIL Worker-4] [OILServerILService] Connection failure (1).
java.net.SocketTimeoutException: Read timed out
at java.net.AbstractSocketImpl$1.read(II)I(Optimized Method)
at jrockit.io.NativeIOInputStream.read(I[BI)I(Unknown Source)
at jrockit.io.NativeIOInputStream.read([BII)I(Unknown Source)
at java.io.BufferedInputStream.fill()V(Optimized Method)
at java.io.BufferedInputStream.read()I(Optimized Method)
at java.io.ObjectInputStream$PeekInputStream.peek()I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Z)I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.refill()V(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.read()I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
at java.io.ObjectInputStream.readByte()B(Unknown Source)
at org.jboss.mq.il.oil.OILServerILService$Client.run()V(OILServerILService.java:206)
at java.lang.Thread.run()V(Unknown Source)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
2006-04-28 08:51:10,261 WARN [OIL Worker-1] [OILServerILService] Connection failure (1).
java.net.SocketTimeoutException: Read timed out
at java.net.AbstractSocketImpl$1.read(II)I(Optimized Method)
at jrockit.io.NativeIOInputStream.read(I[BI)I(Unknown Source)
at jrockit.io.NativeIOInputStream.read([BII)I(Unknown Source)
at java.io.BufferedInputStream.fill()V(Optimized Method)
at java.io.BufferedInputStream.read()I(Optimized Method)
at java.io.ObjectInputStream$PeekInputStream.peek()I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Z)I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.refill()V(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.read()I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
at java.io.ObjectInputStream.readByte()B(Unknown Source)
at org.jboss.mq.il.oil.OILServerILService$Client.run()V(OILServerILService.java:206)
at java.lang.Thread.run()V(Unknown Source)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
2006-04-28 08:51:10,400 DEBUG [OILClientILService-9] [OILClientILService] Closing receiver connections on port: 55063
2006-04-28 08:51:10,502 DEBUG [OILClientILService-4] [OILClientILService] Closing receiver connections on port: 50047
2006-04-28 08:51:10,505 DEBUG [OILClientILService-1] [OILClientILService] Closing receiver connections on port: 49530
2006-04-28 08:51:10,547 WARN [Connection Monitor Thread] [Connection] Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.net.SocketException: Connection reset by peer)
at org.jboss.mq.Connection.asynchFailure(Ljava.lang.String;Ljava.lang.Exception;)V(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run()V(ClockDaemon.java:364)
at java.lang.Thread.run()V(Unknown Source)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
Caused by: java.net.SocketException: Connection reset by peer
at jrockit.net.SocketNativeIO.read(IIII)I(Unknown Source)
at jrockit.net.SocketNativeIO.read(Ljava.io.FileDescriptor;III)I(Unknown Source)
at java.net.AbstractSocketImpl$1.read(II)I(Optimized Method)
at jrockit.io.NativeIOInputStream.read(I[BI)I(Unknown Source)
at jrockit.io.NativeIOInputStream.read([BII)I(Unknown Source)
at java.io.BufferedInputStream.fill()V(Optimized Method)
at java.io.BufferedInputStream.read()I(Optimized Method)
at java.io.ObjectInputStream$PeekInputStream.peek()I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Z)I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.refill()V(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.read()I(Unknown Source)
at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
at java.io.ObjectInputStream.readByte()B(Unknown Source)
at org.jboss.mq.il.oil.OILServerIL.waitAnswer()Ljava.lang.Object;(OILServerIL.java:601)
at org.jboss.mq.il.oil.OILServerIL.ping(Lorg.jboss.mq.ConnectionToken;J)V(OILServerIL.java:424)
at org.jboss.mq.Connection.pingServer(J)V(Optimized Method)
at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1307)
... 3 more
2006-04-28 08:51:11,147 WARN [Connection Monitor Thread] [Connection] Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.EOFException)
at org.jboss.mq.Connection.asynchFailure(Ljava.lang.String;Ljava.lang.Exception;)V(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run()V(ClockDaemon.java:364)
at java.lang.Thread.run()V(Unknown Source)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
at java.io.ObjectInputStream.readByte()B(Unknown Source)
at org.jboss.mq.il.oil.OILServerIL.waitAnswer()Ljava.lang.Object;(OILServerIL.java:601)
at org.jboss.mq.il.oil.OILServerIL.ping(Lorg.jboss.mq.ConnectionToken;J)V(OILServerIL.java:424)
at org.jboss.mq.Connection.pingServer(J)V(Optimized Method)
at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1307)
... 3 more
2006-04-28 08:51:11,147 WARN [Connection Monitor Thread] [Connection] Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.EOFException)
at org.jboss.mq.Connection.asynchFailure(Ljava.lang.String;Ljava.lang.Exception;)V(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run()V(ClockDaemon.java:364)
at java.lang.Thread.run()V(Unknown Source)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
Caused by: java.io.EOFException
at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
at java.io.ObjectInputStream.readByte()B(Unknown Source)
at org.jboss.mq.il.oil.OILServerIL.waitAnswer()Ljava.lang.Object;(OILServerIL.java:601)
at org.jboss.mq.il.oil.OILServerIL.ping(Lorg.jboss.mq.ConnectionToken;J)V(OILServerIL.java:424)
at org.jboss.mq.Connection.pingServer(J)V(Optimized Method)
at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1307)
... 3 more
2006-04-28 08:51:11,154 DEBUG [Thread-1] [Log4jService] Installed System.out adapter
2006-04-28 08:51:11,157 DEBUG [Thread-1] [Log4jService] Installed System.err adapter
2006-04-28 08:51:11,315 INFO [TP-Processor5] [AuthTilesRequestProcessor] --------------------------- AUTHENTICATION CHECK --------------------------
... from that point on we get
2006-04-28 08:52:09,198 WARN [Connection Monitor Thread] [Connection] Connection failure:
org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
at org.jboss.mq.Connection.asynchFailure(Ljava.lang.String;Ljava.lang.Exception;)V(Connection.java:718)
at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1311)
at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run()V(ClockDaemon.java:364)
at java.lang.Thread.run()V(Unknown Source)
at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
Caused by: java.io.IOException: ping timeout.
at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1303)
... 3 more
I tried searching this forum for something like this but did not see anything that seemed to help but if there is already a post related to the issue a pointer would be useful.
Thanks