7 Replies Latest reply on Sep 19, 2006 5:38 PM by brian.stansberry

    Jboss 4 startup error: jgroup or Tree cache?

    xh_wu

      I ran Jboss4.0.2 on linux and had start up problem for "-c all --host ". The "-c default" runs fine.
      The console error msgs and main thread dump msgs are attached below. It looks like the error is related to clustering or cache, but I am not sure. Any help is greatly appeciated!
      -Herbert

      The repeating console error msgs is:
      ******************************
      21:24:03,353 WARN [ClientGmsImpl] handleJoin(cdsapp1-node1:33684) failed, retrying
      21:24:07,668 ERROR [ClientGmsImpl] suspect() should not be invoked on an instance of org.jgroups.protocols.pbcast.ClientGmsImpl
      21:24:07,669 ERROR [ClientGmsImpl] suspect() should not be invoked on an instance of org.jgroups.protocols.pbcast.ClientGmsImpl
      21:24:07,669 ERROR [ClientGmsImpl] suspect() should not be invoked on an instance of org.jgroups.protocols.pbcast.ClientGmsImpl
      21:24:07,669 ERROR [ClientGmsImpl] suspect() should not be invoked on an instance of org.jgroups.protocols.pbcast.ClientGmsImpl
      21:24:07,669 ERROR [ClientGmsImpl] suspect() should not be invoked on an instance of org.jgroups.protocols.pbcast.ClientGmsImpl
      *****************************************

      I did a thread dump and the main thread msgs are:
      ******************************************
      "main" prio=1 tid=0x0000002aea026100 nid=0x5d6f in Object.wait() [0x0000000040fce000..0x0000000040fd1cb0]
      at java.lang.Object.wait(Native Method)
      - waiting on <0x0000002ae4ed99d8> (a java.lang.Object)
      at java.lang.Object.wait(Object.java:474)
      at org.jgroups.JChannel.connect(JChannel.java:378)
      - locked <0x0000002ae4ed99d8> (a java.lang.Object)
      - locked <0x0000002ae4ed4b20> (a org.jgroups.JChannel)
      at org.jboss.cache.TreeCache.startService(TreeCache.java:1112)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
      at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:897)
      at $Proxy0.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:418)
      - locked <0x0000002adf9f49c0> (a org.jboss.system.ServiceController)
      at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:273)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:964)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:775)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:738)
      at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:121)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
      at $Proxy8.deploy(Unknown Source)
      at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:325)
      at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:501)
      - locked <0x0000002adf851518> (a org.jboss.deployment.scanner.URLDeploymentScanner)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:204)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:277)
      - locked <0x0000002adf8513a0> (a org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:272)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:222)
      at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:897)
      at $Proxy0.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:418)
      - locked <0x0000002adf9f49c0> (a org.jboss.system.ServiceController)
      at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:273)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:964)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:775)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:738)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:722)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:585)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:121)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:249)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
      at $Proxy5.deploy(Unknown Source)
      at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:434)
      at org.jboss.system.server.ServerImpl.start(ServerImpl.java:315)
      at org.jboss.Main.boot(Main.java:195)
      at org.jboss.Main$1.run(Main.java:463)
      at java.lang.Thread.run(Thread.java:595)


        • 1. Re: Jboss 4 startup error: jgroup or Tree cache?

          You need to troubleshoot to make sure JGroups work first (or even mcast is working).

          -Ben

          • 2. Re: Jboss 4 startup error: jgroup or Tree cache?
            cglommen

            I get the same problem. I have six servers in my cluster. Whenever I see this message ([org.jgroups.protocols.pbcast.ClientGmsImpl] suspect() should not be invoked on an instance of org.jgroups.protocols.pbcast.ClientGmsImpl) JBoss will not start up.

            But it gets worse. When a server exeriences this problem, it seems to be due to one of the nodes in the cluster, because one by one, as they are all restarted, they will ALL have this and none can restart. As if one node "poisons" the entire cluster. The only solution I have found, which is not very practical, is to change the multicast port for the entire pool, and restart each node.

            You say to troubleshoot JGroups first, but in my case, I am using JBoss "out-of-the-box" using the clustering provided by the "all" configuration.

            Can you provide a clear solution on how to prevent this?

            • 3. Re: Jboss 4 startup error: jgroup or Tree cache?
              belaban

              Can you try this with JGroups 2.2.9RC1 and the VIEW_SYNC protocol below GMS, snippet:

              <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
              down_thread="false" up_thread="false"
              max_bytes="400000"/>
              <VIEW_SYNC avg_send_interval="60000" down_thread="false" up_thread="false" />
              <pbcast.GMS print_local_addr="true" join_timeout="3000"
              down_thread="false" up_thread="false"
              join_retry_timeout="2000" shun="true"/>

              • 4. Re: Jboss 4 startup error: jgroup or Tree cache?
                sventura

                 

                "ben.wang@jboss.com" wrote:
                You need to troubleshoot to make sure JGroups work first (or even mcast is working).

                -Ben


                How can I do that?

                I mean, I do not want clustering enabled but some dependecies ties my to 'all' config.

                I would like to just disable any clustering defualt setting.

                How do I do that?

                This is hitting us hard.

                • 5. Re: Jboss 4 startup error: jgroup or Tree cache?

                  You can remove tc5-cluster-service.xml and cluster-service.xml from deploy directory to disable clustering.

                  -Ben

                  • 6. Re: Jboss 4 startup error: jgroup or Tree cache?
                    dpabhay

                    We get the same error. And this started out of the blue. It has been no problem for 3 months.

                    The only solution is to either change the mcast IP or port. This is very severe - mostly because we don't know what is causing it. And not sure if the current production environment will be affected by it or not.

                    Also, there are two places for mcast IP and Port specification - tc5-cluster-service.xml and cluster-service.xml.
                    Does the IP and Port on these two should be ALWAYS same?
                    If they can be different, what is the impact to keep them separate?
                    Is it a good idea to provide different mcast IP AND Port for each environment (Production, UAT, DEV, developer specific). So as to avoid interference? How does one make sure that one environment does not affect the other in ANY way?

                    This is causing some serious concerns. Please resolve.

                    • 7. Re: Jboss 4 startup error: jgroup or Tree cache?
                      brian.stansberry

                      The port *must* be different between cluster-service.xml and tc5-cluster-service.xml.

                      See http://wiki.jboss.org/wiki/Wiki.jsp?page=TwoClustersSameNetwork for info on isolating clusters.