4 Replies Latest reply on Mar 11, 2010 9:45 AM by pbhanupratap

    TomcatClusteringCache problem in jboss cluster

    venkat.g99

      We have a jboss cluster in production, we have two servers each server has two jboss nodes, when i start the nodes first time after deployment it is working fine, but when i kill one of the node and try to restart again it is failing to get the TomcatClusteringCache. this is happening especially when one node went down with the following error(need help on the below error also when this kind of error will occur ).

       

      2010-02-21 22:37:57,762 INFO  [STDOUT] 924370863 [ConnectionTable.Connection.Receiver [IP1:7810 - IP2:60058]] ERROR org.jgroups.protocols.TCP  - failed handling data from IP2:7810
      QueueClosedException
      at org.jgroups.util.Queue.add(Queue.java:139)
      at org.jgroups.protocols.TP.receive(TP.java:778)
      at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:228)
      at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)
      at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)
      at java.lang.Thread.run(Thread.java:595)

       

      When the above error occurs we are killing the node and restarting the same that time it was not able to find the TomcatClusteringCache.

       

      Appriciate your quick response.

       

       

      -------------------------------------------------------
      GMS: address is IP1:7810
      -------------------------------------------------------
      2010-02-22 00:07:01,525 INFO  [org.jboss.cache.TreeCache] viewAccepted(): [IP1:7811|14] [IP1:7811, IP2:7810, IP2:7811, 172.16.64.21:7810]
      2010-02-22 00:07:01,600 INFO  [org.jboss.cache.TreeCache] TreeCache local address is IP1:7810
      2010-02-22 00:07:16,662 WARN  [org.jgroups.protocols.pbcast.STATE_TRANSFER] state received from IP1:7811 is null, will return null state to application
      2010-02-22 00:07:16,793 WARN  [org.jboss.system.ServiceController] Problem starting service jboss.cache:service=TomcatClusteringCache
      org.jboss.cache.CacheException: Initial state transfer failed: Channel.getState() returned false
              at org.jboss.cache.TreeCache.fetchStateOnStartup(TreeCache.java:3363)
              at org.jboss.cache.TreeCache.startService(TreeCache.java:1555)
              at org.jboss.cache.aop.PojoCache.startService(PojoCache.java:94)
              at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
              at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
              at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
              at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
              at $Proxy0.start(Unknown Source)
              at org.jboss.system.ServiceController.start(ServiceController.java:417)
              at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
              at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
              at $Proxy4.start(Unknown Source)
              at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
              at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
              at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
              at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
              at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
              at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
              at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
              at $Proxy9.deploy(Unknown Source)
              at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
              at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
              at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
              at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
              at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
              at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
              at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
              at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
              at $Proxy0.start(Unknown Source)
              at org.jboss.system.ServiceController.start(ServiceController.java:417)
              at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
              at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
              at $Proxy4.start(Unknown Source)
              at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302)
              at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
              at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
              at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
              at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:585)
              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
              at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
              at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
              at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
              at $Proxy5.deploy(Unknown Source)
              at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
              at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
              at org.jboss.Main.boot(Main.java:200)
              at org.jboss.Main$1.run(Main.java:508)
              at java.lang.Thread.run(Thread.java:595)
      2010-02-22 00:07:17,146 INFO  [STDOUT] no object for null
      2010-02-22 00:07:17,160 INFO  [STDOUT] no object for null
      2010-02-22 00:07:17,270 INFO  [STDOUT] no object for null
      2010-02-22 00:07:17,393 INFO  [STDOUT] no object for {urn:jboss:bean-deployer}supplyType
      2010-02-22 00:07:17,446 INFO  [STDOUT] no object for {urn:jboss:bean-deployer}dependsType
      2010-02-22 00:07:26,381 INFO  [org.jboss.wsf.stack.jbws.NativeServerConfig] JBoss Web Services - Native
      2010-02-22 00:07:26,382 INFO  [org.jboss.wsf.stack.jbws.NativeServerConfig] jbossws-native-2.0.1.SP2 (build=200710210837)
      2010-02-22 00:07:29,411 INFO  [org.jboss.jmx.adaptor.snmp.agent.SnmpAgentService] SNMP agent going active
      2010-02-22 00:07:30,554 INFO  [org.jboss.ha.framework.interfaces.HAPartition.AtheebWeb] Initializing

        • 1. Re: TomcatClusteringCache problem in jboss cluster

          I am going to guess your firewall might be blocking UDP packets and not allowing stat replication to go through.  Another one that I ran into was the cluster was to big and to much data flying around to replicate to all nodes.  At that point we turned buddy replication on to limit who replicated to which node.  Here are some useful links

           

          http://community.jboss.org/wiki/JGroupsLargetStateOrPacketTransferProblems

          http://community.jboss.org/wiki/JBossCacheBuddyReplicationDesign

          1 of 1 people found this helpful
          • 2. Re: TomcatClusteringCache problem in jboss cluster
            venkat.g99

            Dear Zachary,

             

            We have checked with network team on the firewall policy, they are not blocking UDP packets and both servers are in the same zone. I will try with increasing the StateTransferTimeout to 75000.

             

            Thank you very much for your reply and still i want to hear more from you.

            • 3. Re: TomcatClusteringCache problem in jboss cluster
              pbhanupratap

              Venkat,

               

              We are also experiencing a similar kind of problem in our JBOSS cluster..

              We have 8nodes in the cluster 4 on server1 and 4 on server2

              we are getting the below error messages on server2 nodes:

               

              13:15:17,922 ERROR [TCP] failed handling data from 10.4.21.31:10800QueueClosedException        at org.jgroups.util.Queue.add(Queue.java:139)        at org.jgroups.protocols.TP.receive(TP.java:777)        at org.jgroups.protocols.BasicTCP.receive(BasicTCP.java:226)        at org.jgroups.blocks.BasicConnectionTable.receive(BasicConnectionTable.java:167)        at org.jgroups.blocks.BasicConnectionTable$Connection.run(BasicConnectionTable.java:580)        at java.lang.Thread.run(Thread.java:595)

               

              and below messages on all the nodes:

               

              14:44:19,303 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:7800 - 10.4.21.31:45537],5,ConnectionTableGroup]14:44:19,305 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,305 ERROR [ConnectionTable] 2nd attempt to send data failed too14:44:19,306 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:44857 - 10.4.21.30:8800],5,ConnectionTableGroup]14:44:19,307 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,307 ERROR [ConnectionTable] 2nd attempt to send data failed too14:44:19,304 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:44858 - 10.4.21.30:9800],5,ConnectionTableGroup]14:44:19,309 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,306 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:44859 - 10.4.21.30:10800],5,ConnectionTableGroup]14:44:19,312 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,303 WARN  [ConnectionTable] sender thread was interrupted, but is still alive: Thread[ConnectionTable.Connection.Sender [10.4.21.31:7800 - 10.4.21.30:46137],5,ConnectionTableGroup]14:44:19,313 WARN  [ConnectionTable] peer closed connection, trying to re-send msg14:44:19,321 ERROR [ConnectionTable] 2nd attempt to send data failed too14:44:19,322 ERROR [ConnectionTable] 2nd attempt to send data failed too14:44:19,325 ERROR [ConnectionTable] 2nd attempt to send data failed too

               

              There is no StateTransferTimeout in the JbossCacheConfig.xml..

              Please find below the entries in our jbosscacheconfig.xml and please suggest a solutions::

               

              Also, did your problem got resolved? Please share the solutions::

               

                      <attribute name="ClusterConfig">
                           <config>

                 <TCP bind_addr="lnxtaxqpas02" start_port="7800" loopback="true"/>
                 <TCPPING initial_hosts="lnxtaxqpas02[7800],lnxtaxqpas02[8800],lnxtaxqpas02[9800],lnxtaxqpas02[10800],lnxtaxqpas01[7800],lnxtaxqpas01[8800],lnxtaxqpas01[9800],lnxtaxqpas01[10800]" port_range="1" timeout="3500" num_initial_members="3" up_thread="true" down_thread="true"/>
                 <MERGE2 min_interval="5000" max_interval="10000"/>
                 <FD shun="true" timeout="2500" max_tries="5" up_thread="true" down_thread="true"/>
                 <VERIFY_SUSPECT timeout="1500" down_thread="false" up_thread="false"/>
                 <pbcast.NAKACK down_thread="true" up_thread="true" gc_lag="100" retransmit_timeout="3000"/>
                 <pbcast.STABLE desired_avg_gossip="20000" down_thread="false" up_thread="false"/>
                 <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="false" print_local_addr="true" down_thread="true" up_thread="true"/>
                 <pbcast.STATE_TRANSFER up_thread="true" down_thread="true"/>

                           </config>
                      </attribute>

               

              Thanks,

              Bhanu.

              • 4. Re: TomcatClusteringCache problem in jboss cluster
                pbhanupratap

                This error looks like a Hardware configuration Issue of the server.

                When we increased the RAM from 8GB to 16GB and added 1 more CPU on each of the 2 servers, this error is gone.