1 Reply Latest reply on Apr 28, 2006 4:31 PM by jaikiran

    OILSeverILService Connection failure and log4j

    neillamka

      I have been trying to track down a problem where for some reason we start getting SpyJMSExceptions that once they start continue until Jboss is restarted. In order to try and find where the problem is I turned up log4j logging levels.

      What I found when I did that was that an immediate exception was thrown followed by a set of "receiver connections" being closed. After this which the SpyJMSExceptions started and will continue.

      Now, of course this could just be a just a coincidence but I don't think so so I'm trying to find out a couple of things.

      1. Why would changing logging levels cause a connection failure

      and

      2. Is there some way to detect/catch this event and try to restart connections so that we can stop the follow on SpyJMS Exceptions.


      2006-04-28 08:48:58,833 INFO [TP-Processor5] [ChannelSocket] connection timeout reached
      2006-04-28 08:49:04,874 INFO [TP-Processor7] [ChannelSocket] connection timeout reached
      2006-04-28 08:51:09,164 INFO [Thread-1] [Log4jService$URLWatchTimerTask] Configuring from URL: resource:log4j.xml
      2006-04-28 08:51:09,558 DEBUG [Thread-1] [Log4jService] Removed System.out adapter
      2006-04-28 08:51:09,558 DEBUG [Thread-1] [Log4jService] Removed System.err adapter
      2006-04-28 08:51:10,253 WARN [OIL Worker-9] [OILServerILService] Connection failure (1).
      java.net.SocketTimeoutException: Read timed out
      at java.net.AbstractSocketImpl$1.read(II)I(Optimized Method)
      at jrockit.io.NativeIOInputStream.read(I[BI)I(Unknown Source)
      at jrockit.io.NativeIOInputStream.read([BII)I(Unknown Source)
      at java.io.BufferedInputStream.fill()V(Optimized Method)
      at java.io.BufferedInputStream.read()I(Optimized Method)
      at java.io.ObjectInputStream$PeekInputStream.peek()I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Z)I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.refill()V(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.read()I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
      at java.io.ObjectInputStream.readByte()B(Unknown Source)
      at org.jboss.mq.il.oil.OILServerILService$Client.run()V(OILServerILService.java:206)
      at java.lang.Thread.run()V(Unknown Source)
      at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Optimized Method)
      2006-04-28 08:51:10,261 WARN [OIL Worker-4] [OILServerILService] Connection failure (1).
      java.net.SocketTimeoutException: Read timed out
      at java.net.AbstractSocketImpl$1.read(II)I(Optimized Method)
      at jrockit.io.NativeIOInputStream.read(I[BI)I(Unknown Source)
      at jrockit.io.NativeIOInputStream.read([BII)I(Unknown Source)
      at java.io.BufferedInputStream.fill()V(Optimized Method)
      at java.io.BufferedInputStream.read()I(Optimized Method)
      at java.io.ObjectInputStream$PeekInputStream.peek()I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Z)I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.refill()V(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.read()I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
      at java.io.ObjectInputStream.readByte()B(Unknown Source)
      at org.jboss.mq.il.oil.OILServerILService$Client.run()V(OILServerILService.java:206)
      at java.lang.Thread.run()V(Unknown Source)
      at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
      2006-04-28 08:51:10,261 WARN [OIL Worker-1] [OILServerILService] Connection failure (1).
      java.net.SocketTimeoutException: Read timed out
      at java.net.AbstractSocketImpl$1.read(II)I(Optimized Method)
      at jrockit.io.NativeIOInputStream.read(I[BI)I(Unknown Source)
      at jrockit.io.NativeIOInputStream.read([BII)I(Unknown Source)
      at java.io.BufferedInputStream.fill()V(Optimized Method)
      at java.io.BufferedInputStream.read()I(Optimized Method)
      at java.io.ObjectInputStream$PeekInputStream.peek()I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Z)I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.refill()V(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.read()I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
      at java.io.ObjectInputStream.readByte()B(Unknown Source)
      at org.jboss.mq.il.oil.OILServerILService$Client.run()V(OILServerILService.java:206)
      at java.lang.Thread.run()V(Unknown Source)
      at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
      2006-04-28 08:51:10,400 DEBUG [OILClientILService-9] [OILClientILService] Closing receiver connections on port: 55063
      2006-04-28 08:51:10,502 DEBUG [OILClientILService-4] [OILClientILService] Closing receiver connections on port: 50047
      2006-04-28 08:51:10,505 DEBUG [OILClientILService-1] [OILClientILService] Closing receiver connections on port: 49530

      2006-04-28 08:51:10,547 WARN [Connection Monitor Thread] [Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.net.SocketException: Connection reset by peer)
      at org.jboss.mq.Connection.asynchFailure(Ljava.lang.String;Ljava.lang.Exception;)V(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run()V(ClockDaemon.java:364)
      at java.lang.Thread.run()V(Unknown Source)
      at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
      Caused by: java.net.SocketException: Connection reset by peer
      at jrockit.net.SocketNativeIO.read(IIII)I(Unknown Source)
      at jrockit.net.SocketNativeIO.read(Ljava.io.FileDescriptor;III)I(Unknown Source)
      at java.net.AbstractSocketImpl$1.read(II)I(Optimized Method)
      at jrockit.io.NativeIOInputStream.read(I[BI)I(Unknown Source)
      at jrockit.io.NativeIOInputStream.read([BII)I(Unknown Source)
      at java.io.BufferedInputStream.fill()V(Optimized Method)
      at java.io.BufferedInputStream.read()I(Optimized Method)
      at java.io.ObjectInputStream$PeekInputStream.peek()I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(Z)I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.refill()V(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.read()I(Unknown Source)
      at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
      at java.io.ObjectInputStream.readByte()B(Unknown Source)
      at org.jboss.mq.il.oil.OILServerIL.waitAnswer()Ljava.lang.Object;(OILServerIL.java:601)
      at org.jboss.mq.il.oil.OILServerIL.ping(Lorg.jboss.mq.ConnectionToken;J)V(OILServerIL.java:424)
      at org.jboss.mq.Connection.pingServer(J)V(Optimized Method)
      at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1307)
      ... 3 more
      2006-04-28 08:51:11,147 WARN [Connection Monitor Thread] [Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.EOFException)
      at org.jboss.mq.Connection.asynchFailure(Ljava.lang.String;Ljava.lang.Exception;)V(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run()V(ClockDaemon.java:364)
      at java.lang.Thread.run()V(Unknown Source)
      at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
      at java.io.ObjectInputStream.readByte()B(Unknown Source)
      at org.jboss.mq.il.oil.OILServerIL.waitAnswer()Ljava.lang.Object;(OILServerIL.java:601)
      at org.jboss.mq.il.oil.OILServerIL.ping(Lorg.jboss.mq.ConnectionToken;J)V(OILServerIL.java:424)
      at org.jboss.mq.Connection.pingServer(J)V(Optimized Method)
      at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1307)
      ... 3 more
      2006-04-28 08:51:11,147 WARN [Connection Monitor Thread] [Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.EOFException)
      at org.jboss.mq.Connection.asynchFailure(Ljava.lang.String;Ljava.lang.Exception;)V(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run()V(ClockDaemon.java:364)
      at java.lang.Thread.run()V(Unknown Source)
      at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.readByte()B(Unknown Source)
      at java.io.ObjectInputStream.readByte()B(Unknown Source)
      at org.jboss.mq.il.oil.OILServerIL.waitAnswer()Ljava.lang.Object;(OILServerIL.java:601)
      at org.jboss.mq.il.oil.OILServerIL.ping(Lorg.jboss.mq.ConnectionToken;J)V(OILServerIL.java:424)
      at org.jboss.mq.Connection.pingServer(J)V(Optimized Method)
      at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1307)
      ... 3 more
      2006-04-28 08:51:11,154 DEBUG [Thread-1] [Log4jService] Installed System.out adapter
      2006-04-28 08:51:11,157 DEBUG [Thread-1] [Log4jService] Installed System.err adapter
      2006-04-28 08:51:11,315 INFO [TP-Processor5] [AuthTilesRequestProcessor] --------------------------- AUTHENTICATION CHECK --------------------------


      ... from that point on we get

      2006-04-28 08:52:09,198 WARN [Connection Monitor Thread] [Connection] Connection failure:
      org.jboss.mq.SpyJMSException: Connection Failed; - nested throwable: (java.io.IOException: ping timeout.)
      at org.jboss.mq.Connection.asynchFailure(Ljava.lang.String;Ljava.lang.Exception;)V(Connection.java:718)
      at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1311)
      at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run()V(ClockDaemon.java:364)
      at java.lang.Thread.run()V(Unknown Source)
      at java.lang.Thread.startThreadFromVM(Ljava.lang.Thread;)V(Unknown Source)
      Caused by: java.io.IOException: ping timeout.
      at org.jboss.mq.Connection$PingTask.run()V(Connection.java:1303)
      ... 3 more

      I tried searching this forum for something like this but did not see anything that seemed to help but if there is already a post related to the issue a pointer would be useful.

      Thanks