3 Replies Latest reply on Sep 30, 2009 6:55 PM by ron_sigal

    Infinite lock in MicroSocketClientInvoker

    mayankmit2002

      I am running 2 Jboss Nodes (4.2.3GA) in a cluster with Jboss Messaging (1.4.2 SP1) and Jboss remoting ( 2.4.0SP1).
      Now problem is that if I unplugs the network cable the clients get hanged. When I try to dig out the cause using JConsole. I found follwing thread dumps


      ame: pool-3-thread-1
      State: BLOCKED on java.util.LinkedList@2c1b1c owned by: Thread-29
      Total blocked: 6 Total waited: 71
      
      Stack trace:
      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1058)
      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:757)
      org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:423)
      org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:175)
      org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:495)
       - locked java.lang.Object@caafcc
      org.jboss.remoting.Client.setupClientLease(Client.java:1673)
      org.jboss.remoting.Client.connect(Client.java:1575)
      org.jboss.remoting.Client.connect(Client.java:487)
      org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:354)
      java.security.AccessController.doPrivileged(Native Method)
      org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:350)
      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:158)
      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
      org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:81)
      org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
      org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
      org.jboss.jms.client.container.ClusteringAspect.handleCreateConnectionDelegate(ClusteringAspect.java:134)
      sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
      sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
      java.lang.reflect.Method.invoke(Unknown Source)
      org.jboss.aop.advice.PerInstanceAdvice.invoke(PerInstanceAdvice.java:121)
      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeNext(ClientClusteredConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.createConnectionDelegate(ClientClusteredConnectionFactoryDelegate.java)
      org.jboss.jms.client.FailoverCommandCenter.failureDetected(FailoverCommandCenter.java:129)
      org.jboss.jms.client.container.FailoverValveInterceptor.invoke(FailoverValveInterceptor.java:124)
      org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
      org.jboss.jms.client.delegate.ClientConsumerDelegate$close_N4742752445160157748.invokeNext(ClientConsumerDelegate$close_N4742752445160157748.java)
      org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:170)
      org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
      org.jboss.jms.client.delegate.ClientConsumerDelegate$close_N4742752445160157748.invokeNext(ClientConsumerDelegate$close_N4742752445160157748.java)
      org.jboss.jms.client.delegate.ClientConsumerDelegate.close(ClientConsumerDelegate.java)
      org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:306)
      org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:165)
      org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
      org.jboss.jms.client.delegate.ClientSessionDelegate$closing_2473194355759371067.invokeNext(ClientSessionDelegate$closing_2473194355759371067.java)
      org.jboss.jms.client.delegate.ClientSessionDelegate.closing(ClientSessionDelegate.java)
      org.jboss.jms.client.container.ClosedInterceptor.maintainRelatives(ClosedInterceptor.java:305)
      org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:165)
      org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
      org.jboss.jms.client.delegate.ClientConnectionDelegate$closing_2473194355759371067.invokeNext(ClientConnectionDelegate$closing_2473194355759371067.java)
      org.jboss.jms.client.delegate.ClientConnectionDelegate.closing(ClientConnectionDelegate.java)
      org.jboss.jms.client.JBossConnection.close(JBossConnection.java:131)
      com.barco.cms.perspectivemgtproxy.eventsupport.PerspectiveMgtEventReader.stopJMSEvents(Unknown Source)
      com.barco.cms.perspectivemgtproxy.eventsupport.PerspectiveMgtEventReader.restartJMSEvents(Unknown Source)
      com.barco.cms.perspectivemgtproxy.PerspectiveMgt.refreshJMSReader(Unknown Source)
      com.barco.cms.perspectivemgtproxy.PerspectiveMgt.disconnectPerspectiveMgtProxy(Unknown Source)
      com.barco.cms.dependency.CmsPerspectiveMgt.disconnectPerspectiveMgtProxy(Unknown Source)
      com.barco.cms.perspectivemgtproxy.eventsupport.PerspectiveMgtEventReader$EventPoller.run(Unknown Source)
      java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
      java.util.concurrent.FutureTask.run(Unknown Source)
      java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
      java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      java.lang.Thread.run(Unknown Source)



      Name: Thread-29
      State: RUNNABLE
      Total blocked: 1 Total waited: 0
      
      Stack trace:
      java.net.SocketInputStream.socketRead0(Native Method)
      java.net.SocketInputStream.read(Unknown Source)
      java.io.BufferedInputStream.fill(Unknown Source)
      java.io.BufferedInputStream.read(Unknown Source)
       - locked java.io.BufferedInputStream@92446e
      java.io.DataInputStream.readByte(Unknown Source)
      org.jboss.jms.client.remoting.ClientSocketWrapper.checkConnection(ClientSocketWrapper.java:92)
      org.jboss.remoting.transport.socket.SocketClientInvoker.getPooledConnection(SocketClientInvoker.java:223)
      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1060)
       - locked java.util.LinkedList@2c1b1c
      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:757)
      org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:423)
      org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:175)
      org.jboss.remoting.transport.bisocket.BisocketClientInvoker.getSecondaryLocator(BisocketClientInvoker.java:567)
      org.jboss.remoting.transport.bisocket.BisocketServerInvoker.createControlConnection(BisocketServerInvoker.java:331)
      org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:412)
      org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:175)
      org.jboss.remoting.Client.invoke(Client.java:1683)
      org.jboss.remoting.Client.addCallbackListener(Client.java:1749)
      org.jboss.remoting.Client.addListener(Client.java:977)
      org.jboss.jms.client.remoting.JMSRemotingConnection.addInvokerCallbackHandler(JMSRemotingConnection.java:259)
      org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:375)
      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.establishCallback(ClientClusteredConnectionFactoryDelegate.java:99)
       - locked org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate@ed9f47
      org.jboss.jms.client.remoting.ConnectionFactoryCallbackHandler$CallbackConnectionListener.handleConnectionException(ConnectionFactoryCallbackHandler.java:105)
      org.jboss.remoting.ConnectionValidator$1.run(ConnectionValidator.java:608)
      


      Name: Thread-5
      State: BLOCKED on java.util.LinkedList@2c1b1c owned by: Thread-29
      Total blocked: 1 Total waited: 0
      
      Stack trace:
      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1058)
      org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:757)
      org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:423)
      org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:175)
      org.jboss.remoting.Client.invoke(Client.java:1683)
      org.jboss.remoting.Client.invoke(Client.java:607)
      org.jboss.remoting.Client.removeListener(Client.java:1129)
      org.jboss.jms.client.remoting.JMSRemotingConnection.stop(JMSRemotingConnection.java:389)
      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.closeCallback(ClientClusteredConnectionFactoryDelegate.java:178)
      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate.finalize(ClientClusteredConnectionFactoryDelegate.java:158)
      org.jboss.jms.client.delegate.ClientClusteredConnectionFactoryDelegate$FinalizerShutdownHook.run(ClientClusteredConnectionFactoryDelegate.java:418)
      


      Is it a bug in the JBoss Remoting, some where I am doing wrong I have already set jnp.sotimeout value to 60000 during connection creation at client end.


        • 1. Re: Infinite lock in MicroSocketClientInvoker
          ron_sigal

          Hi Mayank,

          I notice that Thread-29, which is holding the java.util.LinkedList@2c1b1c that the other two threads are blocked on, is in the middle of a ClientSocketWrapper.checkConnection(), which is called only if the parameter socket.check_connection is set to "true". I usually advise against turning on the check connection facility, since it involves an extra round trip i/o. Did you mean to turn that on? I think that if you turn it off, the problem might go away.

          "mayankmit2002" wrote:

          I have already set jnp.sotimeout value to


          I don't recognize "jnp.sotimeout", but the parameter that controls socket timeouts in Remoting is "timeout". Setting "timeout" might also solve your problem.

          -Ron

          • 2. Re: Infinite lock in MicroSocketClientInvoker
            mayankmit2002

            Hello Ron,
            Thanks for correcting me, in my configuration "socket.check_connection" was set to true, after setting it to false. one of my problem is fixed.
            But till now, I'am just unable to set the socket time out at the client end when looking for the remote SLSB, even after setting the timeout value to 10000 ms

            I'm creating my connection in the following way.

            private static Properties getContextProperties ()
             {
             final Properties props = new Properties();
             props.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
             props.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces");
             props.put("jnp.sotimeout", "10000");
            // props.put("jnp.timeout", "10000");
             props.put("timeout", "10000");
            
             if (mActiveNetworkAddress != null)
             {
             props.put(NamingContext.JNP_LOCAL_ADDRESS, mActiveNetworkAddress);
             }
            
             // for HA-JNDI lookup
            
             String partitionName = java.lang.System.getProperty("cms.partition.name", java.lang.System.getenv("COMPUTERNAME") + "_PARTITION");
             partitionName = partitionName.toUpperCase();
            
             // for auto discovery through partition name
             props.put(NamingContext.JNP_PARTITION_NAME, partitionName);
            
             props.put(NamingContext.JNP_DISABLE_DISCOVERY, "false");
            
             return props;
             }
            


            I am passing above properties in the inital context.
            is am missing some thing?

            • 3. Re: Infinite lock in MicroSocketClientInvoker
              ron_sigal

              Hi Mayank,

              "mayankmit2002" wrote:

              I'am just unable to set the socket time out at the client end


              The timeout isn't set in the initial context call. If you look in $JBOSS_HOME/server/$CONFIG/deploy/ejb3.deployer/META-INF/jboss-service.xml, you'll see the MBean definition

               <mbean code="org.jboss.remoting.transport.Connector"
               name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
               <depends>jboss.aop:service=AspectDeployer</depends>
               <attribute name="InvokerLocator">socket://${jboss.bind.address}:3873</attribute>
               <attribute name="Configuration">
               <handlers>
               <handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
               </handlers>
               </attribute>
               </mbean>
              


              If you add "timeout" to the InvokerLocator attribute, it will be used by the Remoting server defined by the MBean and also by all of its clients:

               <mbean code="org.jboss.remoting.transport.Connector"
               name="jboss.remoting:type=Connector,name=DefaultEjb3Connector,handler=ejb3">
               <depends>jboss.aop:service=AspectDeployer</depends>
               <attribute name="InvokerLocator">socket://${jboss.bind.address}:3873/?timeout=10000</attribute>
               <attribute name="Configuration">
               <handlers>
               <handler subsystem="AOP">org.jboss.aspects.remoting.AOPRemotingInvocationHandler</handler>
               </handlers>
               </attribute>
               </mbean>
              


              For more information about Remoting configuration, see Chapter 5 of the Remoting Guide: http://jboss.org/jbossremoting/docs/guide/2.2/html/index.html.

              -Ron