3 Replies Latest reply on Sep 14, 2011 2:52 AM by mickey8

    Core Bridge does not work on backup node with HA

    mickey8

      Hi

       

      I am testing each HornetQ feature to use in our production. We use HornetQ2.2.5-GA on JBoss EAP5.1.1.

      Now I am testing Core Bridge with HA. It works on the live node, but does not work on the backup node after the failover.

       

      Here are steps I have done:

      1) Starts the HornetQ of all nodes --> nodeA is live node, nodeB is backup node, nodeC is target node of Core Bridge, queueA is on nodeA and nodeB, queueB is on nodeC

      2) Send message to queueA --> message is stored in queueB

      3) Kill the nodeA --> failover occurs and nodeB becomes a live node

      4) Send message to queueA --> message is NOT stored in queueB, remained in queueA

       

      CoreBridge doesn't work on the nodeB though the sending and receiving message etc. work.

      Here is a part of server.log at 1) on the nodeB. It does not show on the nodeA.

      2011-09-06 18:26:20,968 WARN  [org.hornetq.core.server.cluster.impl.ClusterManagerImpl] (Thread-13) No queue found with name jms.queue.brdgQueue2 bridge will not be deployed.

       

      The bridge configuration of nodeA and nodeB is the same.

      Here is a part of my hornetq-configuration.xml

      <queues>

        <queue name="jms.queue.brdgQueue2">

           <address>jms.queue.brdgQueue2</address>

        </queue>

      </queues>

      <bridges>

        <bridge name="bridgeQueue2">

            <queue-name>jms.queue.brdgQueue2</queue-name>

            <forwarding-address>jms.queue.dstQueue2</forwarding-address>

            <reconnect-attempts>-1</reconnect-attempts>

            <static-connectors>

               <connector-ref>remote-connector</connector-ref>

            </static-connectors>

        </bridge>

      </bridges>

       

      Is there any other configuration necessary for the backup node to use a Core Bridge?

       

      Thanks,

      Mikitaro Hachiyanagi

        • 1. Re: Core Bridge does not work on backup node with HA
          ataylor

          at what point do you get the error message at start up or at failover

          • 2. Re: Core Bridge does not work on backup node with HA
            mickey8

            Hi Andy

             

            at what point do you get the error message at start up or at failover

            at start up.

             

            I tracked startup and failover process with a debugger on nodeB. It seems that process of HornetQServerImpl$SharedStoreBackupActivation#run has a cause.

            At startup:

            1) Deploy bridges at ClusterManagerImpl#start. (line 390)

            2) Wait until failover occur at NodeManager#awaitLiveNode. (line 396)

            After failover:

            3) Load journal and deploy queues at initialisePart2. (line 400)

            4) Start bridges at ClusterManagerImpl#activate. (line 402)

             

            - At 1), Although it is going to try to deploy bridges at ClusterManagerImpl#deployBridge(line 167), Bridges could not be deployed because it could not acquire the binding(line 627). And WARNING is outputted. The cause which could not acquire the binding is because bindings are created at 3).

            - At 3), Although it is going to try to deploy bridges again at ClusterManagerImpl#start(line 1519) after deploying queues, Nothing is done because ClusterManagerImpl is already started at 1).

            - At 4), Although it is going to try to start bridges at ClusterManagerImpl#activate(line 408), Nothing is done because bridges were not deployed at both of 1) and 3).

            • 3. Re: Core Bridge does not work on backup node with HA
              mickey8

              Hi

               

              I provide the example which a problem reproduces. The example is packaged as part of the base distribution. I send the message to queue which is bridged to destination queue at Step22. But I can not receive the message from destination queue at Step23.

               

              There is a more problem about bridge with HA. JMS Bridge does not work on the backup node, either. Here is a part of server.log.

               

              At start up: The following logs continue coming out until failover has occurred.

              2011-08-31 16:50:37,632 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (main) Failed to connect bridge

              javax.naming.NameNotFoundException: queue not bound

                      at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)

                      at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)

                      at org.jnp.server.NamingServer.getObject(NamingServer.java:785)

                      at org.jnp.server.NamingServer.lookup(NamingServer.java:396)

                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                      at java.lang.reflect.Method.invoke(Method.java:616)

                      at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)

                      at sun.rmi.transport.Transport$1.run(Transport.java:177)

                      at java.security.AccessController.doPrivileged(Native Method)

                      at sun.rmi.transport.Transport.serviceCall(Transport.java:173)

                      at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)

                      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)

                      at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

                      at java.lang.Thread.run(Thread.java:636)

                      at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273)

                      at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251)

                      at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160)

                      at org.jnp.server.NamingServer_Stub.lookup(Unknown Source)

                      at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:726)

                      at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)

                      at javax.naming.InitialContext.lookup(InitialContext.java:409)

                      at org.hornetq.jms.bridge.impl.JNDIFactorySupport.createObject(JNDIFactorySupport.java:58)

                      at org.hornetq.jms.bridge.impl.JNDIDestinationFactory.createDestination(JNDIDestinationFactory.java:40)

                      at org.hornetq.jms.bridge.impl.JMSBridgeImpl.setupJMSObjects(JMSBridgeImpl.java:1082)

                      at org.hornetq.jms.bridge.impl.JMSBridgeImpl.start(JMSBridgeImpl.java:345)

                      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

                      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                      at java.lang.reflect.Method.invoke(Method.java:616)

                      at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)

                      at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)

                      at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)

                      at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:243)

                      at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47)

                      at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:111)

                      at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:72)

                      at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221)

                      at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)

                      at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)

                      at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)

                      at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)

                              at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)

                      at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

                      at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

                      at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:125)

                      at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:52)

                      at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)

                      at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:55)

                      at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:179)

                      at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1454)

                      at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1172)

                      at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1193)

                      at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1113)

                      at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

                      at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

                      at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:826)

                      at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:556)

                      at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:789)

                      at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:699)

                      at org.jboss.system.server.profileservice.repository.MainDeployerAdapter.process(MainDeployerAdapter.java:117)

                      at org.jboss.system.server.profileservice.repository.ProfileDeployAction.install(ProfileDeployAction.java:70)

                      at org.jboss.system.server.profileservice.repository.AbstractProfileAction.install(AbstractProfileAction.java:53)

                      at org.jboss.system.server.profileservice.repository.AbstractProfileService.install(AbstractProfileService.java:403)

                      at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1652)

                      at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:938)

                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1082)

                      at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:988)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:778)

                      at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:543)

                      at org.jboss.system.server.profileservice.repository.AbstractProfileService.registerProfile(AbstractProfileService.java:308)

                      at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:256)

                      at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:461)

                      at org.jboss.Main.boot(Main.java:221)

                      at org.jboss.Main$1.run(Main.java:556)

                      at java.lang.Thread.run(Thread.java:636)

              2011-08-31 16:50:37,633 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (main) Failed to start bridge

              2011-08-31 16:50:37,652 WARNING [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-4-thread-1) Will retry after a pause of 5000 ms

               

               

              After failover:

              2011-08-31 16:54:14,293 INFO  [org.hornetq.jms.bridge.impl.JMSBridgeImpl] (pool-3-thread-1) Succeeded in connecting to servers

               

              JMBBridgeImpl failed to lookup the source queue from context at startup on backup node. A problem does not occur on live node. If I apply the patch of HORNETQ-496, then JMSBridge work on the backup node after failover. But in that case, it does no work if I configure the MaxRetries to jms-bridge-jboss-beans.xml and if the number of times of the MaxRetries is exceeded during the backup. A Configuration file is a same as both live node and backup node. I attach hornetq-jms.xml and jms-bridge-jboss-beans.xml which are used on my environment. Is there any problem on my configuration for the backup node to use a JMS Bridge?

               

              Thanks,

              Mikitaro Hachiyanagi