JBoss 5.1 lockup of clustered application
I currently have an application which I am running in a JBoss 5.1.0 cluster, and after what appears to be a random period of time, processing appears to lock up.
One one of the nodes JBoss appears to have completely frozen, I cannot access the JMX console, so therefore I am unable to take a thread dump. I have enabled the Java options PrintGCDetails and PrintGCTimeStamps, and I can therefore see that the garbage collector is no longer processing either. However, I can see that the Java process is still running using the task manager.
The other node is not completely frozen, I am able to access the JMX console, and have taken a thread dump. From the thread dump it appears that all of my worker threads are hung up while attempting to perform a JNDI lookup of a remote session bean. I therefore assume that this lookup is attempting to communicate with the other node, and has become stuck. Also, when I kill the Java process on the node that is completely frozen, this node suddenly kicks back into life. The relevant part of the thread dump is shown below.
Thread: WorkManager(2)-6 : priority:5, demon:true, threadId:458, threadState:RUNNABLE
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x2f4d155a> (a java.io.BufferedInputStream)
java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2288)
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2301)
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2772)
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:778)
java.io.ObjectInputStream.(ObjectInputStream.java:278)
org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.(ObjectInputStreamWithClassLoader.java:100)
org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:54)
org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:75)
org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:58)
org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:179)
org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:162)
org.jboss.remoting.transport.socket.ClientSocketWrapper.(ClientSocketWrapper.java:66)
sun.reflect.GeneratedConstructorAccessor473.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
org.jboss.remoting.transport.socket.SocketClientInvoker.createClientSocket(SocketClientInvoker.java:171)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1116)
org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:762)
org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:165)
org.jboss.remoting.Client.invoke(Client.java:1724)
org.jboss.remoting.Client.invoke(Client.java:629)
org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:84)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.ejb3.proxy.impl.remoting.IsLocalProxyFactoryInterceptor.invoke(IsLocalProxyFactoryInterceptor.java:72)
org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
org.jboss.aspects.remoting.ClusteredPojiProxy.invoke(ClusteredPojiProxy.java:79)
com.sun.proxy.$Proxy576.createProxyBusiness(Unknown Source)
org.jboss.ejb3.proxy.impl.objectfactory.session.SessionProxyObjectFactory.createProxy(SessionProxyObjectFactory.java:129)
org.jboss.ejb3.proxy.clustered.objectfactory.session.SessionClusteredProxyObjectFactory.getProxy(SessionClusteredProxyObjectFactory.java:87)
org.jboss.ejb3.proxy.impl.objectfactory.ProxyObjectFactory.getObjectInstance(ProxyObjectFactory.java:158)
javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
org.jnp.interfaces.NamingContext.getObjectInstance(NamingContext.java:1479)
org.jnp.interfaces.NamingContext.getObjectInstanceWrapFailure(NamingContext.java:1496)
org.jnp.interfaces.NamingContext.lookup(NamingContext.java:822)
org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
javax.naming.InitialContext.lookup(InitialContext.java:392)
com.sw.ServiceLocator.getRemoteInterface(ServiceLocator.java:72)
Does anyone have any suggestions as to why the first node is locking up? Alternatively, is there some way that I can ensure that a lockup of one node does not cause the other node to lockup? Any help would be very much appreciated.
