11 Replies Latest reply on Oct 2, 2012 1:36 AM by mko0815

    reconnect to remote hornetq cluster does not work

    mko0815

      Hi,

       

      we are currently using jboss 7.1.2

      our setup is as follows:

       

      1 cluster of 3 nodes, where our ear application is deployed

      1 cluster of 2 nodes where JMS is configured

       

      both clusterd are managed by 1 domain controller.

       

      now the app cluster should use the jms cluster for messaging.

       

      the app consitst of producers who will write to jms queue, this producers get the jms connection from jndi.

      and as second there are MDBs which are listen to queues on the remote cluster.

       

      if the all nodes are up, the messaging for producers works fine, thy can put messages to the cluster.

      but the MDBs do not listen to the cluster, they just listen to 1 node of JMS cluster, so if message send to frist node MDBs receive it, if message arrives on second node MDBs will not receive it.

       

      and as second problem

      when i shutdown the first node from jms cluster, the jms connection gets lost.

      for the producer i can hanle it, by just checking if the connection is alive, if doesn't i can create a new connection.

       

      but for MDBs they will fail, they completly lost the connection.

      from exception point it looks like the MDBs are not connected to the cluster, instead they are just connected to the first node, even if i restart the first node, the MDBs can't reconnect.

       

      here is how i configured the JMS Nodes:

       

       

      {code:xml}

      <subsystem xmlns="urn:jboss:domain:messaging:1.3">

                      <hornetq-server>

                          <clustered>true</clustered>

                          <persistence-enabled>true</persistence-enabled>

                          <security-enabled>false</security-enabled>

                          <cluster-user>1234</cluster-user>

                          <cluster-password>1234</cluster-password>

                          <journal-file-size>102400</journal-file-size>

                          <journal-min-files>2</journal-min-files>

       

                           <connectors>

                              <netty-connector name="netty" socket-binding="messaging"/>

                          </connectors>

       

                           <acceptors>

                              <netty-acceptor name="netty" socket-binding="messaging"/>

                          </acceptors>

       

                          <broadcast-groups>

                              <broadcast-group name="bg-group1">

                                  <socket-binding>messaging-group</socket-binding>

                                  <broadcast-period>5000</broadcast-period>

                                  <connector-ref>

                                      netty

                                  </connector-ref>

                              </broadcast-group>

                          </broadcast-groups>

       

                          <discovery-groups>

                              <discovery-group name="dg-group1">

                                  <socket-binding>messaging-group</socket-binding>

                                  <refresh-timeout>10000</refresh-timeout>

                              </discovery-group>

                          </discovery-groups>

       

                          <cluster-connections>

                              <cluster-connection name="my-cluster">

                                 <address>jms</address>

                                 <connector-ref>netty</connector-ref>

                                 <retry-interval>500</retry-interval>

                                 <use-duplicate-detection>true</use-duplicate-detection>

                                 <forward-when-no-consumers>true</forward-when-no-consumers>

                                 <max-hops>1</max-hops>

                                 <discovery-group-ref discovery-group-name="dg-group1"/>

                             </cluster-connection>

                          </cluster-connections>

       

                          <address-settings>

                              <address-setting match="jms.queue.MessageQueue">

                                  <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                                  <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                                  <max-size-bytes>419430400</max-size-bytes>

                                  <address-full-policy>PAGE</address-full-policy>

                                  <redistribution-delay>-1</redistribution-delay>

                              </address-setting>

                              <address-setting match="#">

                                  <dead-letter-address>jms.queue.DLQ</dead-letter-address>

                                  <expiry-address>jms.queue.ExpiryQueue</expiry-address>

                                  <max-size-bytes>209715200</max-size-bytes>

                                  <address-full-policy>PAGE</address-full-policy>

                                  <redistribution-delay>500</redistribution-delay>

                              </address-setting>

                          </address-settings>

       

       

                          <jms-destinations>

                               <jms-queue name="testQueue">

                                  <entry name="queue/test"/>

                                  <entry name="java:jboss/exported/jms/queue/test"/>

                              </jms-queue>

                              <jms-topic name="testTopic">

                                  <entry name="topic/test"/>

                                  <entry name="java:jboss/exported/jms/topic/test"/>

                              </jms-topic>

                          </jms-destinations>

                      </hornetq-server>

                  </subsystem>

       

      {code}

       

       

      and here is the config for the application nodes:

       

       

      {code:xml}

      <subsystem xmlns="urn:jboss:domain:messaging:1.3">

                      <hornetq-server>

       

                          <discovery-groups>

                              <discovery-group name="dg-group-test1">

                                  <socket-binding>remote-messaging-group</socket-binding>

                                  <refresh-timeout>10000</refresh-timeout>

                              </discovery-group>

                          </discovery-groups>

       

                          <jms-connection-factories>

                              <connection-factory name="hornetq-ra-cluster">

                                  <discovery-group-ref discovery-group-name="dg-group-test1"/>

                                  <entries>

                                      <entry name="java:/ConnectionFactory"/>

                                      <entry name="java:jboss/exported/jms/ConnectionFactory"/>

                                  </entries>

                              </connection-factory>

                              <pooled-connection-factory name="hornetq-ra">

                                  <user>1234</user>

                                  <password>1234</password>

                                  <transaction mode="xa"/>

                                  <discovery-group-ref discovery-group-name="dg-group-test1"/>

                                  <entries>

                                      <entry name="java:/JmsXA"/>

                                  </entries>

                              </pooled-connection-factory>

                          </jms-connection-factories>

       

                          <jms-destinations>

       

                              <jms-queue name="testQueue">

                                  <entry name="queue/test"/>

                                  <entry name="java:jboss/exported/jms/queue/test"/>

                              </jms-queue>

                              <jms-topic name="testTopic">

                                  <entry name="topic/test"/>

                                  <entry name="java:jboss/exported/jms/topic/test"/>

                              </jms-topic>

                          </jms-destinations>

                      </hornetq-server>

                  </subsystem>

      {code}

       

       

      the exception we got after we shutdown one node is the following;

       

       

      {code}

      [Server:integration-server-one] 11:02:25,040 WARN  [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Periodic Recovery) Can't connect to XARecoveryConfig [transportConfiguration = [org-hornetq-core-remoting-impl-netty-

      ettyConnectorFactory?port=6445&host=www-localhost], discoveryConfiguration = null, username=1234, password=1234] on auto-generated resource recovery: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried

      with all available servers.]

      [Server:integration-server-one]         at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:774) [hornetq-core-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

      [Server:integration-server-one]         at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.connect(HornetQXAResourceWrapper.java:347) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

      [Server:integration-server-one]         at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.getDelegate(HornetQXAResourceWrapper.java:262) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

      [Server:integration-server-one]         at org.hornetq.jms.server.recovery.HornetQXAResourceWrapper.isSameRM(HornetQXAResourceWrapper.java:147) [hornetq-jms-2.2.16.Final.jar:2.2.16.Final (HQ_2_2_16_FINAL, 122)]

      [Server:integration-server-one]         at com.arjuna.ats.internal.jta.recovery.arjunacore.RecoveryXids.isSameRM(RecoveryXids.java:131) [jbossjts-4.16.4.Final.jar:]

      [Server:integration-server-one]         at com.arjuna.ats.internal.jta.recovery.arjunacore.RecoveryXids.updateIfEquivalentRM(RecoveryXids.java:185) [jbossjts-4.16.4.Final.jar:]

      [Server:integration-server-one]         at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.refreshXidScansForEquivalentXAResourceImpl(XARecoveryModule.java:752) [jbossjts-4.16.4.Final.jar:]

      [Server:integration-server-one]         at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:532) [jbossjts-4.16.4.Final.jar:]

      [Server:integration-server-one]         at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:471) [jbossjts-4.16.4.Final.jar:]

      [Server:integration-server-one]         at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:385) [jbossjts-4.16.4.Final.jar:]

      [Server:integration-server-one]         at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:166) [jbossjts-4.16.4.Final.jar:]

      [Server:integration-server-one]         at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [jbossjts-4.16.4.Final.jar:]

      [Server:integration-server-one]         at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [jbossjts-4.16.4.Final.jar:]

       

      {code}

       

      from this exception it look like that the MDBs dont realy use the cluster configurtion :--> discoveryConfiguration = null

       

      did someone have any idea what we make wrong ?

       

      thanks for help

       

       

      kr Matthias

        • 1. Re: reconnect to remote hornetq cluster does not work
          jbertram

          What are your MDB's activation configuration properties?

          • 2. Re: reconnect to remote hornetq cluster does not work
            mko0815

            the MDB configuration ist this one:

             

             

            {code}

            @MessageDriven(activationConfig = {

                                @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),

                                @ActivationConfigProperty(propertyName = "destination", propertyValue = "/queue/test"),

                                @ActivationConfigProperty(propertyName = "user", propertyValue = "1234"),

                                @ActivationConfigProperty(propertyName = "password", propertyValue = "1234"),

                                @ActivationConfigProperty(propertyName = "maxSession", propertyValue = "50"),

                                @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),

             

            {code}

             

            and the conffig in the ejb subsystem is the following:

             

            {code:xml}

               <mdb>

                                <resource-adapter-ref resource-adapter-name="hornetq-ra"/>

                                <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>

                            </mdb>

             

            {code}

             

            so it should point the to pooled connection factory which should be connected to the jms cluster

            • 3. Re: reconnect to remote hornetq cluster does not work
              jbertram

              I recommend you add this to your <pooled-connection-factory>

               

              <ha>true</ha>
              

              After you make that change retry your test and let me know how it goes.

              1 of 1 people found this helpful
              • 4. Re: reconnect to remote hornetq cluster does not work
                mko0815

                Hi,

                 

                i added this flag, and there is a little difference in the logs:

                 

                 

                {code}

                [Server:integration-server-one] 10:02:05,826 INFO  [org.hornetq.jms.server.recovery.RecoveryDiscovery] (HornetQ Recovery Discovery Reinitialization) Starting RecoveryDiscovery on XARecoveryConfig [transportConfiguration = null, d

                iscoveryConfiguration = DiscoveryGroupConfiguration [discoveryInitialWaitTimeout=10000, groupAddress=231.7.7.9, groupPort=9876, localBindAddress=null, name=d7938353-02f8-11e2-8d05-1d0f6fe57c34, refreshTimeout=10000], username=1234, password=1234]

                 

                [Server:integration-server-one] 10:02:06,840 WARN  [org.hornetq.jms.server.recovery.HornetQXAResourceWrapper] (Periodic Recovery) Can't connect to XARecoveryConfig [transportConfiguration = [org-hornetq-core-remoting-impl-netty-N

                ettyConnectorFactory?port=6445&host=www-localhost], discoveryConfiguration = null, username=1234, password=1234] on auto-generated resource recovery: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried

                with all available servers.]

                 

                {code}

                 

                so it looks for first reconnect he knows about the cluster, but then its somehome lost.

                also the behaviour didnt change. (MDBs still connected to just a single node )

                 

                any other idea? or any other informations which could be helpfull ?

                 

                thanks

                • 5. Re: reconnect to remote hornetq cluster does not work
                  mko0815

                  and how does the JCA configuration have influence on it ?

                   

                  as i can see from this picture:

                   

                  http://docs.jboss.org/hornetq/2.3.0.Alpha/docs/user-manual/html_single/images/architecture2.jpg

                   

                  it have a lot, but how it should look like ?

                   

                  thanks

                  • 6. Re: reconnect to remote hornetq cluster does not work
                    mko0815

                    noone have a idea what else could it be ?

                     

                    thanks

                    • 7. Re: reconnect to remote hornetq cluster does not work
                      jbertram

                      I recommend you add this to your <pooled-connection-factory>

                       

                      <reconnect-attempts>-1</reconnect-attempts>
                      

                       

                      After you make that change retry your test and let me know how it goes.

                      • 8. Re: reconnect to remote hornetq cluster does not work
                        mko0815

                        hi,

                         

                        so finally we got it running,

                        but it was not with the reconnect-attempts.

                         

                        i just used the jboss trunk version from 2012-09-28

                        and with the HA flag and removing the max sessions flag it works as it should

                         

                        with max sessions, all consumers where just connection to one node, without this flag, there are some consumers (MDBs) connecting to one, and some to the other node.

                        for 7.1.2 Version this also does not work

                         

                        so it looks like there is a bug in the 7.1.2 Version which seems to be fixed in current trunk. (that consumers only connect to one node in cluster group)

                         

                        thanks

                        kr

                        1 of 1 people found this helpful
                        • 9. Re: reconnect to remote hornetq cluster does not work
                          jbertram

                          If you used an AS7 nightly build from September 28 then it would have had the fix for AS7-5601.  I believe manually specifying reconnect-attempts would have resolved the issue in earlier versions.

                           

                          I don't see how maxSession could have any impact on this.  It simply controls how many sessions are created for the MDB.  It doesn't control how they are distributed between the nodes of the cluster.

                          • 10. Re: reconnect to remote hornetq cluster does not work
                            mko0815

                            Hi,

                             

                            i guess this bug was not the problem, because the reconnect itself worked, what is not working is the connection to ALL Nodes in the Cluster.

                            so MDBs are connected just to one node, for this node the reconnection works.

                             

                            what happens with maxsessions is also not clear to mear, its just what i observed.

                            with maxsessions = 50, node 1 had 50 consumers, where node 2 have 0.

                            also a stop of node 1 didnt change something.

                             

                            when i dont set this parameter, then

                            node 1 have 8 consumers

                            node 2 have 7 consumers.

                             

                             

                            but, if i will add node 3 now,

                            then node 3 will never get consumers.

                             

                            so there is still something wrong

                            • 11. Re: reconnect to remote hornetq cluster does not work
                              mko0815

                              also there is no discoveryGroup config on reconnect attemp.

                              he only knows the connection to node 1 and node 3, directly