EJB3/Socket invoker - connection timeouts| JBoss.org Content Archive (Read Only)

15. Re: EJB3/Socket invoker - connection timeouts

javajedi Jun 27, 2007 3:13 AM (in response to javajedi)

Thanks. Posted at http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4057982

16. Re: EJB3/Socket invoker - connection timeouts

gurgstar Jun 27, 2007 12:37 PM (in response to javajedi)

Hey Ron, How would you recommend javajedi work around this issue until he can upgrade to 4.2.0? Could he replace, say, the socket invoker with the rmi invoker?

17. Re: EJB3/Socket invoker - connection timeouts

javajedi Jun 27, 2007 5:14 PM (in response to javajedi)

I tried JBoss Remoting 2.2.0 and I was unable to get it to work with JBoss 4.0.4. I'm also not too enthusiastic about that approach because looking at the code, I don't think that this bug has been fixed in the latest version. I am now able to reproduce this problem easily by setting maxPoolSize to 10 (instead of 300) on the invoker; it fails almost right away. I turned on debug and was able to get some more information.

I start off getting a handful of these errors:

2007-06-27 11:32:13,108 82807 ERROR [org.jboss.remoting.transport.socket.SocketServerInvoker] (SocketServerInvoker#0-3873:) Failed to accept socket connection
java.lang.reflect.InvocationTargetException
 at sun.reflect.GeneratedConstructorAccessor54.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at org.jboss.remoting.transport.socket.ServerThread.createServerSocket(ServerThread.java:185)
 at org.jboss.remoting.transport.socket.ServerThread.wakeup(ServerThread.java:226)
 at org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:467)
 at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:391)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Mismatch version of JBossSerialization signature
 at org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:113)
 at org.jboss.serial.io.JBossObjectInputStream.<init>(JBossObjectInputStream.java:94)
 at org.jboss.serial.io.JBossObjectInputStream.<init>(JBossObjectInputStream.java:83) at org.jboss.remoting.serialization.impl.jboss.JBossSerializationManager.createInput(JBossSerializationManager.java:57)
 at org.jboss.remoting.transport.socket.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:56)
 at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:76)
 at org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(ClientSocketWrapper.java:54)
 at org.jboss.remoting.transport.socket.ServerSocketWrapper.<init>(ServerSocketWrapper.java:50)
 ... 8 more

Don't know what that means. Then, at the end, just before the deadlock occurs, I get this debug message:

007-06-27 11:32:13,487 83186 DEBUG [org.jboss.remoting.transport.socket.ServerThread] (SocketServerInvokerThread-68.142.219.226-13:) Exiting run on exception
java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:280)

This appears to be where the lost notification happens. This interrupt seems to be coming from the evict() method in ServerThread. I love this comment in ServerThread.evict():

 // This is a race and there is a chance
 // that a invocation is going on at the time
 // of the interrupt. But I see no way right
 // now to protect for this.
 // There may not be a problem because interrupt only effects
 // threads blocking on IO.

So at least whoever wrote this has admitted the race condition that seems to be crashing us. :) I'm trying to come up with a good solution to this, but so far, no luck. I'm hoping that someone more familiar with the code can give me an answer besides "try 2.2".

Thanks.

18. Re: EJB3/Socket invoker - connection timeouts

javajedi Jun 27, 2007 7:29 PM (in response to javajedi)

Apologies to Ron for double-posting both here and on the support ticket, but I want to give the other kind souls who have been helping me out an update.

Just tried with store-bought 1.4.6. Exact same problem. Lots of these:

2007-06-27 16:19:37,799 71584 ERROR [org.jboss.remoting.transport.socket.SocketServerInvoker] (SocketServerInvoker#0-3873:) Failed to accept socket connection
java.lang.reflect.InvocationTargetException
 at sun.reflect.GeneratedConstructorAccessor53.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at org.jboss.remoting.transport.socket.ServerThread.createServerSocket(ServerThread.java:175)
 at org.jboss.remoting.transport.socket.ServerThread.wakeup(ServerThread.java:216)
 at org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:488)
 at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:412)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Mismatch version of JBossSerialization signature
 at org.jboss.serial.io.JBossObjectInputStream.checkSignature(JBossObjectInputStream.java:113)
 at ...

Followed by one of these:

2007-06-27 16:19:37,800 71585 DEBUG [org.jboss.remoting.transport.socket.ServerThread] (SocketServerInvokerThread-68.142.219.95-2:) Exiting run on exception
java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)

(the missed notify)

and then deadlock. The deadlocked threads:

One of these:

Thread: SocketServerInvoker#0-3873 : priority:5, demon:false, threadId:23, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.LRUPool@4f2076

 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:485)
 org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:460)
 org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:412)
 java.lang.Thread.run(Thread.java:619)

and several of these:

Thread: SocketServerInvokerThread-68.142.219.226-0 : priority:5, demon:false, threadId:79, threadState:WAITING, threadLockName:org.jboss.remoting.transport.socket.ServerThread@134ab9b

 java.lang.Object.wait(Native Method)
 java.lang.Object.wait(Object.java:485)
 org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:282)

The call to ServerThread.wakeup is throwing an exception, caused by that "mismatch version of JBossSerialization signature" error. Assuming that we can track down the root cause of that error, it's still going to be possible for a socket timeout to occur here, so we have to be prepared for this failure. When wakeup throws the exception, it never makes it to the end of the method to invoke notify() to wake up the waiting ServerThread thread. So the ServerThreads just continue to disappear from the threadpool every time wakeup() throws an exception before it gets to notify(), until eventually, the threadpool is empty, and the SocketServerInvoker just sits and waits forever (deadlock).

19. Re: EJB3/Socket invoker - connection timeouts

javajedi Jun 27, 2007 7:52 PM (in response to javajedi)

Ok, I wrapped the code in wakeup() inside of a try/catch, with a catch block that looks like this:

 } catch (Exception e) {
 synchronized (clientpool) {
 synchronized (threadpool) {
 clientpool.remove(this);
 threadpool.add(this);
 Thread.interrupted();
 }
 }
 throw e;
 }

and so far no deadlocks, but there were still plenty "mismatch version of JBossSerialization signature" errors spewing. I tried switching from JBoss Serialization back to Java Serialization. Instead of the "mismatch version" errors, I got a bunch of these:

2007-06-27 16:37:22,519 156879 ERROR [org.jboss.remoting.transport.socket.SocketServerInvoker] (SocketServerInvoker#0-3873:) Failed to accept socket connection
java.lang.reflect.InvocationTargetException
 at sun.reflect.GeneratedConstructorAccessor62.newInstance(Unknown Source)
 at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
 at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
 at org.jboss.remoting.transport.socket.ServerThread.createServerSocket(ServerThread.java:188)
 at org.jboss.remoting.transport.socket.ServerThread.wakeup(ServerThread.java:233)
 at org.jboss.remoting.transport.socket.SocketServerInvoker.processInvocation(SocketServerInvoker.java:461)
 at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:392)
 at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.EOFException
 at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2279)
 at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2748)
 at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:780)
 at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
 at org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.<init>(ObjectInputStreamWithClassLoader.java:73)
 at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:52)
 at org.jboss.remoting.transport.socket.ServerSocketWrapper.createInputStream(ServerSocketWrapper.java:56)
 at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:76)
 at org.jboss.remoting.transport.socket.ClientSocketWrapper.<init>(ClientSocketWrapper.java:54)
 at org.jboss.remoting.transport.socket.ServerSocketWrapper.<init>(ServerSocketWrapper.java:50)
 ... 8 more

So I don't think JBoss Serialization is at fault here. It looks like my socket timeout of 6 seconds was just not long enough to read what it needed before the socket timed out. I switched the socket timeout back to the default (60 seconds?) This seems to have halted all of the errors coming from the wakeup() method.

So the key to preventing the deadlock seems to be catching the exception that is coming from wakeup() and resetting the state (i.e. adding the thread back to the threadpool) before continuing.

Does this seem reasonable?

20. Re: EJB3/Socket invoker - connection timeouts

gurgstar Jun 27, 2007 7:53 PM (in response to javajedi)

Two suggestions

(1) Stop writing code that relies on interrupting threads.

(2) To avoid lost wake up problems, replace foo.notify() with something that stays signalled if there are no threads calling foo.wait(). Consider using a Semaphore or a BlockingQueue from util.concurrent / java.util.concurrent.

21. Re: EJB3/Socket invoker - connection timeouts

javajedi Jun 28, 2007 5:57 PM (in response to javajedi)

Bug filed at http://jira.jboss.com/jira/browse/JBREM-762

22. Re: EJB3/Socket invoker - connection timeouts

javajedi Jul 9, 2007 4:52 PM (in response to javajedi)

My patch has been working fine in production for 11 days now; 4 of them at full capacity. The pool sizes look to be correct and under control. I think that my patch has resolved the issue.