3 Replies Latest reply on Aug 3, 2009 11:19 AM by Alexander Hartner

    Error on clustered nodes on startup

    Alexander Hartner Expert

      My cluster consist of 8 JBoss instances. Using JBoss Messaging 1.4.2 GA SP1 most things work fine, however every so often an exception is thrown while trying to insert a duplicate row into the JBM_TX table


      16:16:08,155 WARN [JDBCSupport] SQLException caught, SQLState 23000 code:2627- assuming deadlock detected, try:1
      java.sql.SQLException: Violation of PRIMARY KEY constraint 'PK__JBM_TX__0C46B282'. Cannot insert duplicate key in object 'dbo.JBM_TX'.
      at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:365)
      at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2781)
      at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2224)
      at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:628)
      at net.sourceforge.jtds.jdbc.JtdsStatement.processResults(JtdsStatement.java:525)
      at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQL(JtdsStatement.java:487)
      at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeUpdate(JtdsPreparedStatement.java:421)
      at org.jboss.resource.adapter.jdbc.CachedPreparedStatement.executeUpdate(CachedPreparedStatement.java:95)
      at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:365)
      at org.jboss.messaging.core.impl.JDBCPersistenceManager.addTXRecord(JDBCPersistenceManager.java:2161)
      at org.jboss.messaging.core.impl.JDBCPersistenceManager$1HandleBeforePrepareRunner.doTransaction(JDBCPersistenceManager.java:1953)
      at org.jboss.messaging.core.impl.JDBCSupport$JDBCTxRunner2.execute(JDBCSupport.java:474)


      I haven't found this issue reported anywhere else, but found some fixes relating to clustering in the release notes for 1.4.4. After upgrading to 1.4.4 I am no longer able to start all my cluster nodes. The following exception is reported during startup of either the 3rd or 4th node in the cluster. At this point this error happens while trying to deploy the examples without my own application being deployed. Tests with my own application produces similar results.


      14:05:58,688 WARN [SecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has
      on how to do this.
      14:05:58,704 INFO [ServerPeer] JBoss Messaging 1.4.4.GA server [3] started
      14:05:58,750 INFO [STDOUT]
      -------------------------------------------------------
      GMS: address is 192.168.2.84:3820
      -------------------------------------------------------
      14:05:58,813 INFO [GroupMember] org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener@4b83a223 got new view [192.168.2.81:3911|3] [192.168.2.81:3911, 192.
      14:05:58,813 INFO [GroupMember] I am (192.168.2.84:3820)
      14:05:58,829 INFO [GroupMember] New Members : 4 ([192.168.2.81:3911, 192.168.2.82:3969, 192.168.2.83:3767, 192.168.2.84:3820])
      14:05:58,829 INFO [GroupMember] All Members : 4 ([192.168.2.81:3911, 192.168.2.82:3969, 192.168.2.83:3767, 192.168.2.84:3820])
      14:05:59,454 INFO [STDOUT]
      -------------------------------------------------------
      GMS: address is 192.168.2.84:7900
      -------------------------------------------------------
      14:05:59,876 INFO [ConnectionFactory] Connector bisocket://192.168.2.84:4457 has leasing enabled, lease period 10000 milliseconds
      14:05:59,876 INFO [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@7578f526 started
      14:06:09,048 INFO [ConnectionFactory] Connector bisocket://192.168.2.84:4457 has leasing enabled, lease period 10000 milliseconds
      14:06:09,048 INFO [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@5430b40a started
      14:06:10,063 WARN [SecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has
      on how to do this.
      14:06:10,313 WARN [SecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has
      on how to do this.
      14:06:10,313 WARN [SecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has
      on how to do this.
      14:06:40,158 ERROR [ExceptionUtil] org.jboss.jms.server.connectionfactory.ConnectionFactory@655fb6e0 startService
      java.lang.IllegalStateException: org.jboss.messaging.core.impl.postoffice.GroupMember@3832c852 response not received from 192.168.2.81:3911 - there may be others
      at org.jboss.messaging.core.impl.postoffice.GroupMember.multicastControl(GroupMember.java:268)
      at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.put(MessagingPostOffice.java:1450)
      at org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.registerConnectionFactory(ConnectionFactoryJNDIMapper.java:252)
      at org.jboss.jms.server.connectionfactory.ConnectionFactory.startService(ConnectionFactory.java:209)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
      at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
      at $Proxy0.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:417)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:304)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy51.start(Unknown Source)
      at org.jboss.deployment.XSLSubDeployer.start(XSLSubDeployer.java:197)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
      at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy9.deploy(Unknown Source)
      at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
      at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
      at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
      at $Proxy0.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:417)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:304)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy5.deploy(Unknown Source)
      at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
      at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
      at org.jboss.Main.boot(Main.java:200)
      at org.jboss.Main$1.run(Main.java:508)
      at java.lang.Thread.run(Thread.java:619)
      14:06:40,174 WARN [ServiceController] Problem starting service jboss.messaging.connectionfactory:service=ClusterPullConnectionFactory
      java.lang.IllegalStateException: org.jboss.messaging.core.impl.postoffice.GroupMember@3832c852 response not received from 192.168.2.81:3911 - there may be others
      at org.jboss.messaging.core.impl.postoffice.GroupMember.multicastControl(GroupMember.java:268)
      at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.put(MessagingPostOffice.java:1450)
      at org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.registerConnectionFactory(ConnectionFactoryJNDIMapper.java:252)
      at org.jboss.jms.server.connectionfactory.ConnectionFactory.startService(ConnectionFactory.java:209)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
      at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995)
      at $Proxy0.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:417)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at org.jboss.system.ServiceController.start(ServiceController.java:435)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:304)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy51.start(Unknown Source)
      at org.jboss.deployment.XSLSubDeployer.start(XSLSubDeployer.java:197)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
      at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy9.deploy(Unknown Source)
      at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421)
      at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
      at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289)
      at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245)
      at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
      at $Proxy0.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:417)
      at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:304)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
      at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
      at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
      at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
      at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
      at $Proxy5.deploy(Unknown Source)
      at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482)
      at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362)
      at org.jboss.Main.boot(Main.java:200)
      at org.jboss.Main$1.run(Main.java:508)
      at java.lang.Thread.run(Thread.java:619)
      14:06:40,190 INFO [QueueService] Queue[/queue/DLQ] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,190 INFO [QueueService] Queue[/queue/ExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,190 INFO [TopicService] Topic[/topic/testTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,190 INFO [TopicService] Topic[/topic/securedTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,205 INFO [TopicService] Topic[/topic/testDurableTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,205 INFO [QueueService] Queue[/queue/testQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,205 INFO [QueueService] Queue[/queue/A] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,205 INFO [QueueService] Queue[/queue/B] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,205 INFO [QueueService] Queue[/queue/C] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,205 INFO [QueueService] Queue[/queue/D] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,205 INFO [QueueService] Queue[/queue/ex] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,205 INFO [QueueService] Queue[/queue/PrivateDLQ] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,221 INFO [QueueService] Queue[/queue/PrivateExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,221 INFO [QueueService] Queue[/queue/QueueWithOwnDLQAndExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,221 INFO [TopicService] Topic[/topic/TopicWithOwnDLQAndExpiryQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,221 INFO [QueueService] Queue[/queue/QueueWithOwnRedeliveryDelay] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,221 INFO [TopicService] Topic[/topic/TopicWithOwnRedeliveryDelay] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,440 WARN [SecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has
      on how to do this.
      14:06:40,440 WARN [SecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has
      on how to do this.
      14:06:40,940 WARN [SecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has
      on how to do this.
      14:06:40,940 WARN [SecurityMetadataStore] WARNING! POTENTIAL SECURITY RISK. It has been detected that the MessageSucker component which sucks messages from one node to another has
      on how to do this.
      14:06:40,940 INFO [QueueService] Queue[/queue/testDistributedQueue] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:40,940 INFO [TopicService] Topic[/topic/testDistributedTopic] started, fullSize=200000, pageSize=2000, downCacheSize=2000
      14:06:41,049 INFO [WrapperDataSourceService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=jdbc/abcDS' to JNDI name 'jdbc/abcDS'
      14:06:41,065 INFO [WrapperDataSourceService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=jdbc/abcDSNoTX' to JNDI name 'jdbc/abcDSNoTX'
      14:06:41,080 INFO [WrapperDataSourceService] Bound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=jdbc/abcDSECF' to JNDI name 'jdbc/abcDSECF'
      14:06:41,096 INFO [TomcatDeployer] deploy, ctxPath=/jmx-console, warUrl=.../deploy/jmx-console.war/
      14:06:41,174 INFO [FarmMemberService] **** pullNewDeployments ****
      14:06:41,174 INFO [ClusterFileTransfer] Start pull of file cluster-examples-service.xml from cluster.
      14:06:41,190 INFO [ClusterFileTransfer] Finished cluster pull of file cluster-examples-service.xml to cluster-examples-service.xml
      14:06:41,205 ERROR [URLDeploymentScanner] Incomplete Deployment listing:

      --- MBeans waiting for other MBeans ---
      ObjectName: jboss.messaging.connectionfactory:service=ClusterPullConnectionFactory
      State: FAILED
      Reason: java.lang.IllegalStateException: org.jboss.messaging.core.impl.postoffice.GroupMember@3832c852 response not received from 192.168.2.81:3911 - there may be others
      I Depend On:
      jboss.messaging:service=ServerPeer
      jboss.messaging:service=Connector,transport=bisocket
      jboss.messaging:service=PostOffice

      --- MBEANS THAT ARE THE ROOT CAUSE OF THE PROBLEM ---
      ObjectName: jboss.messaging.connectionfactory:service=ClusterPullConnectionFactory
      State: FAILED
      Reason: java.lang.IllegalStateException: org.jboss.messaging.core.impl.postoffice.GroupMember@3832c852 response not received from 192.168.2.81:3911 - there may be others
      I Depend On:
      jboss.messaging:service=ServerPeer
      jboss.messaging:service=Connector,transport=bisocket
      jboss.messaging:service=PostOffice


      14:06:41,236 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-192.168.2.84-8080
      14:06:41,252 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-192.168.2.84-8009
      14:06:41,268 INFO [Server] JBoss (MX MicroKernel) [4.2.3.GA (build: SVNTag=JBoss_4_2_3_GA date=200807181439)] Started in 53s:205ms


      Any input on how to address this are appreciated.