2 Replies Latest reply on Sep 18, 2013 9:36 AM by pallotjk

    Error Joining a node in a Jboss Clúster 6.1 - Receview new cluster view

    fivalo
    fivalo Apr 5, 2013 1:13 PM

      Hi all,

       

       

      Having:

      Jboss 6.1.0 Final

      jdk 1.6.0_33

      Suse Linux Enterprise Server

       

      I'm having a problem in a six-nodes cluster, everything had been working fine until the cluster didn´t allow to join any new node, when I see the logs files, It says:

       

      11:45:41,734 INFO  [JGroupsTransport] Received new cluster view: [172.20.1.86:1099|23][172.20.1.86:1099, 172.20.1.80:1099, ce7ee4e1-56a0-bfd5-50dc-53be35af167c, 172.20.1.85:1099,172.20.1.55:1099, 172.20.1.87:1099]

      11:45:41,761 INFO  [JGroupsTransport] Cache local address is 172.20.1.87:1099, physical addresses are[172.20.1.87:40868]

       

      There is something wrong with one IP of the nodes and this is "ce7ee4e1-56a0-bfd5-50dc-53be35af167c" this is not a valid IP, so when another node wants to join itself to the cluster, it produces this exception:

       

      20:42:53,912 INFO  [DefaultCacheContainerFactory] Started "distributed-state" cache from "ha-partition" container

      20:42:53,928 ERROR [NEWPartition] ViewAccepted failed: java.lang.IllegalStateException: Address ce7ee4e1-56a0-bfd5-50dc-53be35af167cnot registered in transport layer

              at org.jboss.ha.core.framework.server.CoreGroupCommunicationService$ClusterNodeFactoryImpl.getClusterNode(CoreGroupCommunicationService.java:2126) [:1.0.0.Final]

              at org.jboss.ha.core.framework.server.CoreGroupCommunicationService.translateAddresses(CoreGroupCommunicationService.java:1584) [:1.0.0.Final]

              at org.jboss.ha.core.framework.server.CoreGroupCommunicationService$GroupView.<init>(CoreGroupCommunicationService.java:1719) [:1.0.0.Final]

              at org.jboss.ha.core.framework.server.CoreGroupCommunicationService.processViewChange(CoreGroupCommunicationService.java:1443) [:1.0.0.Final]

              at org.jboss.ha.core.framework.server.CoreGroupCommunicationService$MembershipListenerImpl.viewAccepted(CoreGroupCommunicationService.java:2222) [:1.0.0.Final]

              at org.jboss.ha.core.framework.server.CoreGroupCommunicationService.startService(CoreGroupCommunicationService.java:1176) [:1.0.0.Final]

              at org.jboss.ha.core.framework.server.HAPartitionImpl.startService(HAPartitionImpl.java:215) [:1.0.0.Final]

              at org.jboss.ha.framework.server.ClusterPartition.startService(ClusterPartition.java:115) [:2.0.0.Final]

              at org.jboss.ha.core.framework.server.CoreGroupCommunicationService.start(CoreGroupCommunicationService.java:1040) [:1.0.0.Final]

              at org.jboss.ha.framework.server.ClusterPartition.pojoStart(ClusterPartition.java:522) [:2.0.0.Final]

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [:1.6.0_31]

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [:1.6.0_31]

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_31]

              at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_31]

              at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:60) [jboss-reflect.jar:2.2.1.SP1]

              at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:168) [jboss-reflect.jar:2.2.1.SP1]

              at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) [jboss-reflect.jar:2.2.1.SP1]

              at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:257) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:125) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:72) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:202) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.system.ServiceController.doChange(ServiceController.java:671) [:6.1.0.Final (Build SVNTag:JBoss_6.1.0.Final date: 20110816)]

              at org.jboss.system.ServiceController.start(ServiceController.java:443) [:6.1.0.Final (Build SVNTag:JBoss_6.1.0.Final date: 20110816)]

              at org.jboss.system.microcontainer.jmx.ServiceControllerStartStopLifecycleCallback.install(ServiceControllerStartStopLifecycleCallback.java:44) [:6.1.0.Final]

              at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source) [:1.6.0_31]

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [:1.6.0_31]

              at java.lang.reflect.Method.invoke(Method.java:597) [:1.6.0_31]

              at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:60) [jboss-reflect.jar:2.2.1.SP1]

              at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:168) [jboss-reflect.jar:2.2.1.SP1]

              at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66) [jboss-reflect.jar:2.2.1.SP1]

              at org.jboss.beans.info.plugins.AbstractBeanInfo.invoke(AbstractBeanInfo.java:300) [jboss-reflect.jar:2.2.1.SP1]

              at org.jboss.kernel.plugins.dependency.AbstractKernelControllerContext.invoke(AbstractKernelControllerContext.java:305) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractLifecycleCallbackItem.install(AbstractLifecycleCallbackItem.java:87) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.handleLifecycleCallbacks(AbstractController.java:2018) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.handleInstallLifecycleCallbacks(AbstractController.java:1983) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1091) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:894) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:641) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:182) [:2.2.2.GA]

              at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:58) [:2.2.2.GA]

              at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:63) [:2.2.2.GA]

              at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55) [:2.2.2.GA]

              at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179) [:2.2.2.GA]

              at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1832) [:2.2.2.GA]

              at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1550) [:2.2.2.GA]

              at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1571) [:2.2.2.GA]

              at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1491) [:2.2.2.GA]

              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.deployers.plugins.deployers.DeployersImpl.change(DeployersImpl.java:1983) [:2.2.2.GA]

              at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:1076) [:2.2.2.GA]

              at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:679) [:2.2.2.GA]

              at org.jboss.system.server.profileservice.deployers.MainDeployerPlugin.process(MainDeployerPlugin.java:106) [:6.1.0.Final]

              at org.jboss.profileservice.dependency.ProfileControllerContext$DelegateDeployer.process(ProfileControllerContext.java:143) [:0.2.2]

              at org.jboss.profileservice.dependency.ProfileDeployAction.deploy(ProfileDeployAction.java:151) [:0.2.2]

              at org.jboss.profileservice.dependency.ProfileDeployAction.installActionInternal(ProfileDeployAction.java:94) [:0.2.2]

              at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42) [jboss-kernel.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:379) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:2044) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:1083) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.executeOrIncrementStateDirectly(AbstractController.java:1322) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1246) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1139) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:939) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:654) [jboss-dependency.jar:2.2.0.SP2]

              at org.jboss.profileservice.dependency.ProfileActivationWrapper$BasicProfileActivation.start(ProfileActivationWrapper.java:190) [:0.2.2]

              at org.jboss.profileservice.dependency.ProfileActivationWrapper.start(ProfileActivationWrapper.java:87) [:0.2.2]

              at org.jboss.profileservice.dependency.ProfileActivationService.activateProfile(ProfileActivationService.java:215) [:0.2.2]

              at org.jboss.profileservice.dependency.ProfileActivationService.activate(ProfileActivationService.java:159) [:0.2.2]

              at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.activate(AbstractProfileServiceBootstrap.java:112) [:0.2.2]

              at org.jboss.profileservice.resolver.BasicResolverFactory$ProfileResolverFacade.deploy(BasicResolverFactory.java:87) [:0.2.2]

              at org.jboss.profileservice.bootstrap.AbstractProfileServiceBootstrap.start(AbstractProfileServiceBootstrap.java:91) [:0.2.2]

              at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:132) [:6.1.0.Final]

              at org.jboss.system.server.profileservice.bootstrap.BasicProfileServiceBootstrap.start(BasicProfileServiceBootstrap.java:56) [:6.1.0.Final]

              at org.jboss.bootstrap.impl.base.server.AbstractServer.startBootstraps(AbstractServer.java:827) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-6]

              at org.jboss.bootstrap.impl.base.server.AbstractServer$StartServerTask.run(AbstractServer.java:417) [jboss-bootstrap-impl-base.jar:2.1.0-alpha-6]

              at java.lang.Thread.run(Thread.java:662) [:1.6.0_31]


       

      NEWPartition is the -g parameter included in the Jboss-start script. The configuración for start the jboss is:

       

      run.sh -c all -b 172.20.1.87 -g NEWPartition -Djboss.default.jgroups.stack=tcp

       

      We suspected it can be something with IPv6 but in the Linux Suse Server doesnt have enabled this feature.

       

      We are using transport over TCP for jgropus and the discovery of the nodes is MPING (by the way the TCPPING with the static list of the nodes never has worked for me)

       

      Thanks if anyone could help me

       

      Best regards

        • 1. Re: Error Joining a node in a Jboss Clúster 6.1 - Receview new cluster view
          rhusar
          rhusar Apr 26, 2013 7:09 AM (in response to fivalo)

          Seems weird. Looks like you have misconfigured something that results in exceptions like above.

           

          What is the exception on the joining node?

            • Actions
          • 2. Re: Error Joining a node in a Jboss Clúster 6.1 - Receview new cluster view
            pallotjk
            pallotjk Sep 18, 2013 9:36 AM (in response to fivalo)

            I have a very similar/identical problem but with jboss-as-7.1.1final and jdk 1.7.0_11

             

            Just like Fidel I have a 6 node cluster – each node on a different server.

             

            When all 6 jboss services are started the cluster is created  successfully as per the following entry in the server.log file of node 6 (very similar entries would be seen in server.log files for nodes 1 to 5).

             

            11:47:23,640 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-11) JBAS015876: Starting deployment of "lbc.war"

            11:47:24,265 INFO  [stdout] (pool-14-thread-1)

            11:47:24,265 INFO  [stdout] (pool-14-thread-1) -------------------------------------------------------------------

            11:47:24,265 INFO  [stdout] (pool-14-thread-1) GMS: address=6/web, cluster=web, physical address=10.80.0.163:7600

            11:47:24,265 INFO  [stdout] (pool-14-thread-1) -------------------------------------------------------------------

             

            11:47:25,562 WARNING [org.jgroups.protocols.TCP] (FD_SOCK pinger,web,6/web) logical address cache didn't contain all physical address, sending up a discovery request

            11:47:25,562 SEVERE [org.jgroups.protocols.TCP] (FD_SOCK pinger,web,6/web) failed sending message to cluster (69 bytes): java.lang.NullPointerException, cause: null

            11:47:26,312 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-8) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

            11:47:26,312 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (MSC service thread 1-15) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be pasivated.

            11:47:26,390 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000078: Starting JGroups Channel

            11:47:26,390 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000094: Received new cluster view: [1/web|5] [1/web, 2/web, 3/web, 4/web, 5/web, 6/web]

            11:47:26,390 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (pool-15-thread-1) ISPN000079: Cache local address is 6/web, physical addresses are [10.80.0.163:7600]

             

            Now I stop jboss services hosting node 4, 2 and 3 successively. Entries in node 6 server .log are as expected.

             

            11:54:06,116 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-13,null) JBAS010248: New cluster view for partition web: 6 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@16d85ab delta: -1, merge: false)

            11:54:06,116 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-13,null) ISPN000094: Received new cluster view: [1/web|6] [1/web, 2/web, 3/web, 5/web, 6/web]

            11:54:48,649 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-17,null) JBAS010248: New cluster view for partition web: 7 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@ad73de delta: -1, merge: false)

            11:54:48,649 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-17,null) ISPN000094: Received new cluster view: [1/web|7] [1/web, 3/web, 5/web, 6/web]

            11:55:12,118 INFO  [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-1,null) JBAS010248: New cluster view for partition web: 8 (org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView@2bbc3a delta: -1, merge: false)

            11:55:12,118 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,null) ISPN000094: Received new cluster view: [1/web|8] [1/web, 5/web, 6/web]

             

            At this point clustering for nodes 1 5 and 6 is still working.

             

            Now I start the jboss service hosting node 4 (for example) and I get an error message similar to the one reported by Fidel.

            It doesn't look like node 4 is coming back in to the cluster!

             

            11:55:51,994 ERROR [org.jboss.as.clustering.impl.CoreGroupCommunicationService.web] (Incoming-18,null) JBAS010245: ViewAccepted failed: java.lang.IllegalStateException: JBAS010280: Address ee161f2c-5782-891b-a1cf-9aabb137e28e not registered in transport layer

            at org.jboss.as.clustering.impl.CoreGroupCommunicationService$ClusterNodeFactoryImpl.getClusterNode(CoreGroupCommunicationService.java:1648) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

            at org.jboss.as.clustering.impl.CoreGroupCommunicationService.translateAddresses(CoreGroupCommunicationService.java:1280) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

            at org.jboss.as.clustering.impl.CoreGroupCommunicationService$GroupView.<init>(CoreGroupCommunicationService.java:1371) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

            at org.jboss.as.clustering.impl.CoreGroupCommunicationService.processViewChange(CoreGroupCommunicationService.java:1180) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

            at org.jboss.as.clustering.impl.CoreGroupCommunicationService$MembershipListenerImpl.viewAccepted(CoreGroupCommunicationService.java:1727) [jboss-as-clustering-impl-7.1.1.Final.jar:7.1.1.Final]

            at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:506) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:545) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jboss.as.clustering.jgroups.ClassLoaderAwareUpHandler.up(ClassLoaderAwareUpHandler.java:56) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]

            at org.jgroups.blocks.mux.MuxUpHandler.passToAllHandlers(MuxUpHandler.java:156) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:123) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jboss.as.clustering.jgroups.MuxChannel$ClassLoaderAwareMuxUpHandler.up(MuxChannel.java:64) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]

            at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.FRAG2.up(FRAG2.java:181) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.FlowControl.up(FlowControl.java:418) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.pbcast.GMS.installView(GMS.java:625) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.pbcast.ParticipantGmsImpl.handleViewChange(ParticipantGmsImpl.java:152) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.pbcast.GMS.up(GMS.java:784) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:730) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:559) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.BARRIER.up(BARRIER.java:126) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:140) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.FD.up(FD.java:273) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:282) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.MERGE2.up(MERGE2.java:205) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.Discovery.up(Discovery.java:355) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.stack.Protocol.up(Protocol.java:358) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.TP.passMessageUp(TP.java:1174) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1722) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1704) [jgroups-3.0.6.Final.jar:3.0.6.Final]

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]

            at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]

             

            11:55:51,994 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-18,null) ISPN000094: Received new cluster view: [1/web|9] [1/web, 5/web, 6/web, ee161f2c-5782-891b-a1cf-9aabb137e28e]

            11:55:52,744 WARNING [org.jgroups.protocols.TCP] (JBossWeb-threads - 384) logical address cache didn't contain all physical address, sending up a discovery request

            11:55:52,744 SEVERE [org.jgroups.protocols.TCP] (JBossWeb-threads - 384) failed sending message to cluster (531 bytes): java.lang.NullPointerException, cause: null

             

            Does anyone know why I am getting “ee161f2c-5782-891b-a1cf-9aabb137e28e” rather than 4/web in the 11:55:51,994 line.

             

            This is the corresponding entry in the server.log of node 4 (the one I tried to bring back into the cluster):

            11:55:49,449 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-5) JBAS015876: Starting deployment of "lbc.war"

            11:55:50,105 INFO  [stdout] (pool-14-thread-1)

            11:55:50,105 INFO  [stdout] (pool-14-thread-1) -------------------------------------------------------------------

            11:55:50,105 INFO  [stdout] (pool-14-thread-1) GMS: address=4/web, cluster=web, physical address=10.80.0.161:7600

            11:55:50,105 INFO  [stdout] (pool-14-thread-1) -------------------------------------------------------------------

             

            11:55:51,996 WARNING [org.jgroups.protocols.TCP] (FD_SOCK pinger,web,4/web) logical address cache didn't contain all physical address, sending up a discovery request

            11:55:51,996 SEVERE [org.jgroups.protocols.TCP] (FD_SOCK pinger,web,4/web) failed sending message to cluster (69 bytes): java.lang.NullPointerException, cause: null

            11:55:52,527 ERROR [org.jboss.msc.service.fail] (pool-14-thread-1) MSC00001: Failed to start service jboss.jgroups.channel.web: org.jboss.msc.service.StartException in service jboss.jgroups.channel.web: java.lang.NullPointerException

            at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:62) [jboss-as-clustering-common-7.1.1.Final.jar:7.1.1.Final]

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_11]

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_11]

            at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_11]

            Caused by: java.lang.NullPointerException

            at org.jboss.as.clustering.jgroups.subsystem.ChannelService.start(ChannelService.java:51) [jboss-as-clustering-jgroups-7.1.1.Final.jar:7.1.1.Final]

            at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:59) [jboss-as-clustering-common-7.1.1.Final.jar:7.1.1.Final]

            ... 3 more

             

            11:55:52,730 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015870: Deploy of deployment "lbc.war" was rolled back with failure message JBAS014750: Operation handler failed to complete

            11:55:52,746 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) JBAS015877: Stopped deployment lbc.war in 14ms

            11:55:52,746 INFO  [org.jboss.as.controller] (DeploymentScanner-threads - 2) JBAS014774: Service status report

            JBAS014777: Services which failed to start:      service jboss.jgroups.channel.web: org.jboss.msc.service.StartException in service jboss.jgroups.channel.web: java.lang.NullPointerException

             

            11:55:52,746 ERROR [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) {"JBAS014653: Composite operation failed and was rolled back. Steps that failed:" => {"Operation step-2" => "JBAS014750: Operation handler failed to complete"}}

             

            We are using transport over TCP for jgropus and the discovery of the nodes is TCPPING.


            Apart from Fidel I see noone else with this problem  - any help/pointers  would be most gratefully recievied.


             

             

             

             

             

             

            .

              • Actions