10 Replies Latest reply on Dec 27, 2011 4:58 AM by chetanc

    JBm queue conflict in a cluster environment.

    mbaluch

      Hi,

       

      First I would like to say that I am aware that I posted a similar thread before and marked it as solved, but this time the situation is a bit different.

       

      I have set up a cluster with two instances (soa-p er7). First server starts no problem, but during the second server startup I see the following exception in log:

       

       

      2010-02-01 17:44:47,762 ERROR [org.jboss.messaging.util.ExceptionUtil] (main) Queue[null, name=CallbackQueue] startService
      java.lang.IllegalStateException: Channel id map for node 20 already contains binding for queue 10
           at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBindingInMemory(MessagingPostOffice.java:2543)
           at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalAddBinding(MessagingPostOffice.java:1980)
           at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBinding(MessagingPostOffice.java:470)
           at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:126)
           at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)
           at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:322)
           at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:597)
           at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
           at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
           at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
           at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
           at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
           at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
           at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:189)
           at $Proxy38.start(Unknown Source)
           at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
           at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
           at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
           at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
           at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
           at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
           at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)
           at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1633)
           at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
           at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
           at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
           at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:823)
           at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
           at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
           at org.jboss.system.ServiceController.start(ServiceController.java:460)
           at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)
           at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)
           at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
           at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
           at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
           at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
           at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1440)
           at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1158)
           at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1179)
           at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1099)
           at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
           at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1633)
           at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
           at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
           at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
           at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:823)
           at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
           at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:782)
           at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702)
           at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
           at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
           at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
           at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)
           at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
           at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1633)
           at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
           at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
           at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
           at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:775)
           at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)
           at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)
           at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)
           at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
           at org.jboss.Main.boot(Main.java:221)
           at org.jboss.Main$1.run(Main.java:556)
           at java.lang.Thread.run(Thread.java:619)
      


      I am using mysql51 database with two schemas. Each schema is used by one of the instances. Meaning I don't have a shared database. Also ServerPeerID's are different.

       

      The callback queue present in the log has it's Clustered attribute set to true.

       

      For completness I have attached also log, messaging/destinations-service.xml, messaging/db-persistence-service.xml and bpm.esb/jbm-queue-service.xml files from both servers.

       

      I have no idea if this is a bug or just a configuration problem therefor any help would be appreciated.

       

      Thanks a lot for your help.

       

      Marek

        • 1. Re: JBm queue conflict in a cluster environment.
          mbaluch

          Well sorry for the ampersands. Here's the error:

           

          2010-02-01 17:44:47,762 ERROR [org.jboss.messaging.util.ExceptionUtil] (main) Queue[null, name=CallbackQueue] startService
          java.lang.IllegalStateException: Channel id map for node 20 already contains binding for queue 10
              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBindingInMemory(MessagingPostOffice.java:2543)
              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.internalAddBinding(MessagingPostOffice.java:1980)
              at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.addBinding(MessagingPostOffice.java:470)
              at org.jboss.jms.server.destination.QueueService.startService(QueueService.java:126)
              at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:376)
              at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:322)
              at sun.reflect.GeneratedMethodAccessor80.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:157)
              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:96)
              at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:138)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
              at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:140)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:90)
              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:668)
              at org.jboss.system.microcontainer.ServiceProxy.invoke(ServiceProxy.java:189)
              at $Proxy38.start(Unknown Source)
              at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:42)
              at org.jboss.system.microcontainer.StartStopLifecycleAction.installAction(StartStopLifecycleAction.java:37)
              at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
              at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
              at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
              at org.jboss.system.microcontainer.ServiceControllerContext.install(ServiceControllerContext.java:297)
              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1633)
              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:823)
              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
              at org.jboss.system.ServiceController.doChange(ServiceController.java:688)
              at org.jboss.system.ServiceController.start(ServiceController.java:460)
              at org.jboss.system.deployers.ServiceDeployer.start(ServiceDeployer.java:163)
              at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:99)
              at org.jboss.system.deployers.ServiceDeployer.deploy(ServiceDeployer.java:46)
              at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
              at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
              at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
              at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1440)
              at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1158)
              at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1179)
              at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1099)
              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1633)
              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:823)
              at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
              at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:782)
              at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:702)
              at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)
              at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)
              at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)
              at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)
              at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1633)
              at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:935)
              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1083)
              at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:985)
              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:775)
              at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)
              at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)
              at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)
              at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)
              at org.jboss.Main.boot(Main.java:221)
              at org.jboss.Main$1.run(Main.java:556)
              at java.lang.Thread.run(Thread.java:619)

          • 2. Re: JBm queue conflict in a cluster environment.
            gaohoward

            I think you need to add this dependency to your queue mbean configuration.

             

            <depends>jboss.messaging:service=PostOffice</depends>

             

            Please try it and see if this solve the problem.

             

            Howard

            • 3. Re: JBm queue conflict in a cluster environment.
              mbaluch

              Hi Howard,

               

              Thanks for the quick post. I added the dependency to the jbpm queue configuration since that's the file which defines the queues in question but with no success. The second cluster instance still won't start .

               

              Any more thoughts?

               

              Marek

              • 4. Re: JBm queue conflict in a cluster environment.
                gaohoward

                Did you try cleaning up the database before starting up the cluster?

                 

                If you still get error, can you turn on the 'Trace' level log and attach them here?

                 

                Thanks

                Howard

                • 5. Re: JBm queue conflict in a cluster environment.
                  mbaluch
                  Sure I cleaned the database. I also deleted the working data from both instances prior startup.
                  • 6. Re: JBm queue conflict in a cluster environment.
                    mbaluch

                    Here are the logs. I enabled TRACE for org.jboss.messaging and org.jbpm.

                     

                    Thanks.

                    • 7. Re: JBm queue conflict in a cluster environment.
                      ticoli

                      I've got the same issue. Did you solve your problem?

                       

                      Thanks for your answer!

                      • 8. Re: JBm queue conflict in a cluster environment.
                        gaohoward

                        Jbm cluster needs a shared DB to work with. If you are using Mysql, you must use one instance of mysql for the cluster, i.e. all nodes in the cluster should point to the same DB for JBM.

                        • 9. Re: JBm queue conflict in a cluster environment.
                          ticoli

                          Thanks for your help.

                           

                          I already have a shared database (MySql).

                           

                          I've got 2 nodes. (Id 1 and Id 2). JBoss AS 5.1.0 GA

                           

                          I think that my Db is correctly configured: If i clean every thing, i start only node 1, tables prefixed with JBM are created on the shared database. Got the same result if i only start node 2.

                           

                          Questions:

                          • Should queue must be deployed in deployed-hasingleton or not?
                          • Clustered attrribut is enough?
                          • Xml must be deployed on each node?
                          • 10. Re: JBm queue conflict in a cluster environment.
                            chetanc

                            I am also facing the same issue with JBOSS 5.1.0 and JBOSS messaging 1.4.3 GA.

                             

                            I have 2 topics configured in my-jms-destinations.service.xml. If I only keep that topics there then both the nodes gets started successfully and everything works like waw. But If I am adding a queue to this file then I am observing java.lang.IllegalStateException: Channel id map for node 21 already contains binding for queue 2. If I remove the queue and the mdb for that queue then everything works fine.  I also tried updating the server PeedID to unique int value for both nodes but no luck.

                             

                            Could you please let me know if you have got any solution for this?