2 Replies Latest reply on Apr 26, 2013 9:10 AM by Radoslav Husar

    JBoss 5.1 lockup of clustered application

    Stuart Williamson Novice

      I currently have an application which I am running in a JBoss 5.1.0 cluster, and after what appears to be a random period of time, processing appears to lock up.

       

      One one of the nodes JBoss appears to have completely frozen, I cannot access the JMX console, so therefore I am unable to take a thread dump. I have enabled the Java options PrintGCDetails and PrintGCTimeStamps, and I can therefore see that the garbage collector is no longer processing either. However, I can see that the Java process is still running using the task manager.

       

      The other node is not completely frozen, I am able to access the JMX console, and have taken a thread dump. From the thread dump it appears that all of my worker threads are hung up while attempting to perform a JNDI lookup of a remote session bean. I therefore assume that this lookup is attempting to communicate with the other node, and has become stuck. Also, when I kill the Java process on the node that is completely frozen, this node suddenly kicks back into life. The relevant part of the thread dump is shown below.

       

      Thread: WorkManager(2)-6 : priority:5, demon:true, threadId:458, threadState:RUNNABLE

      java.net.SocketInputStream.socketRead0(Native Method)
      java.net.SocketInputStream.read(SocketInputStream.java:129)
      java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
      java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
      java.io.BufferedInputStream.read(BufferedInputStream.java:317)
      - locked <0x2f4d155a> (a java.io.BufferedInputStream)
      java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2288)
      java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2301)
      java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2772)
      java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:778)
      java.io.ObjectInputStream.(ObjectInputStream.java:278)
      org.jboss.remoting.loading.ObjectInputStreamWithClassLoader.(ObjectInputStreamWithClassLoader.java:100)
      org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:54)
      org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:75)
      org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:58)
      org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:179)
      org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:162)
      org.jboss.remoting.transport.socket.ClientSocketWrapper.(ClientSocketWrapper.java:66)
      sun.reflect.GeneratedConstructorAccessor473.newInstance(Unknown Source)
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      org.jboss.remoting.transport.socket.SocketClientInvoker.createClientSocket(SocketClientInvoker.java:171)
      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1116)
      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:762)
      org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:165)
      org.jboss.remoting.Client.invoke(Client.java:1724)
      org.jboss.remoting.Client.invoke(Client.java:629)
      org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:60)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:84)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      org.jboss.ejb3.proxy.impl.remoting.IsLocalProxyFactoryInterceptor.invoke(IsLocalProxyFactoryInterceptor.java:72)
      org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
      org.jboss.aspects.remoting.ClusteredPojiProxy.invoke(ClusteredPojiProxy.java:79)
      com.sun.proxy.$Proxy576.createProxyBusiness(Unknown Source)
      org.jboss.ejb3.proxy.impl.objectfactory.session.SessionProxyObjectFactory.createProxy(SessionProxyObjectFactory.java:129)
      org.jboss.ejb3.proxy.clustered.objectfactory.session.SessionClusteredProxyObjectFactory.getProxy(SessionClusteredProxyObjectFactory.java:87)
      org.jboss.ejb3.proxy.impl.objectfactory.ProxyObjectFactory.getObjectInstance(ProxyObjectFactory.java:158)
      javax.naming.spi.NamingManager.getObjectInstance(NamingManager.java:304)
      org.jnp.interfaces.NamingContext.getObjectInstance(NamingContext.java:1479)
      org.jnp.interfaces.NamingContext.getObjectInstanceWrapFailure(NamingContext.java:1496)
      org.jnp.interfaces.NamingContext.lookup(NamingContext.java:822)
      org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
      javax.naming.InitialContext.lookup(InitialContext.java:392)
      com.sw.ServiceLocator.getRemoteInterface(ServiceLocator.java:72)


       

      Does anyone have any suggestions as to why the first node is locking up? Alternatively, is there some way that I can ensure that a lockup of one node does not cause the other node to lockup? Any help would be very much appreciated.