Need help in resolving clustering issue in jboss 5.1.0 GA
Hello,
We are using jboss.5.1.0.GA in two servers which are in clustering. Recently we got an issue like if 2nd node is up then the application is not working. When we stop the 2nd node application is working fine the we came to know that there is an issue in node2. The below is the error we found in the log in node2.
---------------------------------------------------------
GMS: address is 10.31.12.103:7900 (cluster=HMM_PR_WEB_GROUP-DATA)
----------------------------------------------------------
2015-02-06 07:06:59,751 ERROR [ExceptionUtil]|||org.jboss.messaging.core.jmx.MessagingPostOfficeService@5eb9ec47 startService
java.lang.IllegalArgumentException: 10.31.12.3:7901 is not a member of the group [10.31.12.3:7900, 10.31.12.103:7900] (enabled_members=[])
at org.jgroups.protocols.UNICAST.down(UNICAST.java:332)
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:316)
at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:204)
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:905)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:475)
at org.jgroups.JChannel.down(JChannel.java:1517)
at org.jgroups.JChannel.send(JChannel.java:626)
at org.jboss.messaging.core.impl.postoffice.GroupMember.unicastData(GroupMember.java:320)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.requestDeliveries(MessagingPostOffice.java:1716)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.loadBindings(MessagingPostOffice.java:2583)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.start(MessagingPostOffice.java:394)
at org.jboss.messaging.core.jmx.MessagingPostOfficeService.startService(MessagingPostOfficeService.java:462)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)
at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:269)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)
at com.sun.proxy.$Proxy38.start(Unknown Source)
at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:286)
at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)
at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
at org.jboss.system.ServiceController.start(ServiceController.java:460)
at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)
at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)
at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
2015-02-06 07:06:59,753 ERROR [AbstractKernelController]|||Error installing to Start: name=jboss.messaging:service=PostOffice state=Create mode=Manual requiredState=Installed
java.lang.IllegalArgumentException: 10.31.12.3:7901 is not a member of the group [10.31.12.3:7900, 10.31.12.103:7900] (enabled_members=[])
at org.jgroups.protocols.UNICAST.down(UNICAST.java:332)
at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:316)
at org.jgroups.protocols.VIEW_SYNC.down(VIEW_SYNC.java:204)
at org.jgroups.protocols.pbcast.GMS.down(GMS.java:905)
at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:475)
at org.jgroups.JChannel.down(JChannel.java:1517)
at org.jgroups.JChannel.send(JChannel.java:626)
at org.jboss.messaging.core.impl.postoffice.GroupMember.unicastData(GroupMember.java:320)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.requestDeliveries(MessagingPostOffice.java:1716)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.loadBindings(MessagingPostOffice.java:2583)
at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.start(MessagingPostOffice.java:394)
at org.jboss.messaging.core.jmx.MessagingPostOfficeService.startService(MessagingPostOfficeService.java:462)
at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)
at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:269)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:206)
at com.sun.proxy.$Proxy38.start(Unknown Source)
at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:286)
at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1631)
at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)
at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
at org.jboss.system.ServiceController.start(ServiceController.java:460)
at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)
at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)
at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
DEPLOYMENTS IN ERROR:
Deployment "jboss.messaging:service=PostOffice" is in error due to the following reason(s): **ERROR**, java.lang.IllegalArgumentException: 10.31.12.3:7901 is not a member of the group [10.31.12.3:7900, 10.31.12.103:7900] (enabled_members=[])
Deployment "vfszip:/app/jboss/jboss-5.1.0.GA/server/airman/deploy/airman-web.ear/" is in error due to the following reason(s): org.jboss.deployers.spi.DeploymentException: URL file:/data/jboss/jboss-5.1.0.GA/airman/tmp/avc2v-c5rvp4-i5t8bwx3-1-i5t8ek26-ar/cache-web-3.0.PR3.war/ deployment failed
**************************************
And in node 1 if found only one error. that is
2015-02-06 07:08:44,751 ERROR [ConnectionTable]|||failed sending data to 10.31.12.103:7900: java.net.SocketException: Socket closed
2015-02-06 07:08:44,752 ERROR [ConnectionTable]|||failed sending data to 10.31.12.103:7900: java.net.SocketException: Socket closed
At present we stopped node2 and the application is running on only node1. Please help in resolving this issue as it is major now.
Thanks in advance