9 Replies Latest reply on Oct 21, 2010 12:01 AM by grantlittle

    HornetQ Failover with JCA in JBoss

    grantlittle

      Hi all,

       

      I am having problems getting failover to work with a JCA connector (for a live/backup pair) with JBoss 4.0.2 as the application server.

       

      Environment:

      * Java 1.6.0_21

      * JBoss 4.0.2

      * HornetQ 2.1.2

       

      1. I deployed the hornetq-ra.rar file into the jboss server/default/deploy folder and have changed the relevant files in the META-INF folder (ra.xml etc)
      2. I have deployed a jms-remote-ds.xml file for a XA transaction connection factory.
      3. I have a remote primary and backup servers correctly configured (and working when not using JCA ie failing over correctly).
      4. I can connect and send messages to my test queue (and read them off using a consumer).

       

       

      My issue starts when I do a kill -9 on the primary hornetq server (after I have sent a number of messages to the queue initally to make sure all is working).

       

      Expected behaviour (from my understanding):

      1. The client (ie the JCA client) determines that the connection has been lost.
      2. The FailoverManager fails over the existing connections in the connection pool to the backup server
      3. A new request to send a message to the queue should result in an existing connection (from the JBoss pooled connections) being used (which is now configured to the backup server).

       

      Actual behaviour I'm seeing

      1. The client (ie the JCA client) determines that the connection has been lost.
      2. The FailoverManager fails over the existing connections in the connection pool to the backup server
      3. After failing over to the backup server the FailoverManager notifies the FailoverListeners
      4. The call to the FailoverListeners propagates up to the JBoss TxConnectionManager$TxConnectionEventListener as a connection error event.
      5. The TxConnectionManager unregisters the connections (which have been failed over to the backup server) and removes them from the JBoss pool.
      6. A call to send a message to the queue now fails as the JCA connector tries to create a new connection (to the primary server) and fails. It does not failover to the backup as I don't have FailoverOnInitialConnection set to true.

       

      Here is a stack trace showing the failover path

       

      12:37:07,995 ERROR [STDERR] Oct 19, 2010 12:37:07 PM org.hornetq.core.logging.impl.JULLogDelegate warn
      WARNING: Handling JMS exception failure: org.hornetq.ra.HornetQRAManagedConnection@aea944
      javax.jms.JMSException: HornetQException[errorCode=2 message=Channel disconnected]
          at org.hornetq.jms.client.HornetQConnection$JMSFailureListener.connectionFailed(HornetQConnection.java:603)
          at org.hornetq.core.client.impl.FailoverManagerImpl.callFailureListeners(FailoverManagerImpl.java:768)
          at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrReconnect(FailoverManagerImpl.java:730)
          at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnectionFailure(FailoverManagerImpl.java:580)
          at org.hornetq.core.client.impl.FailoverManagerImpl.connectionDestroyed(FailoverManagerImpl.java:291)
          at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:679)
          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: HornetQException[errorCode=2 message=Channel disconnected]
          ... 6 more
      12:37:07,997 WARN  [TxConnectionManager] Connection error occured: org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener@1afcfd4[state=NORMAL mc=org.hornetq.ra.HornetQRAManagedConnection@aea944 handles=0 lastUse=1287462968271 permit=false trackByTx=false mcp=org.jboss.resource.connectionmanager.JBossManagedConnectionPool$OnePool@eae613 context=org.jboss.resource.connectionmanager.InternalManagedConnectionPool@1a7234f xaResource=org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper@1ff3f5d txSync=null]
      javax.jms.JMSException: HornetQException[errorCode=2 message=Channel disconnected]
          at org.hornetq.jms.client.HornetQConnection$JMSFailureListener.connectionFailed(HornetQConnection.java:603)
          at org.hornetq.core.client.impl.FailoverManagerImpl.callFailureListeners(FailoverManagerImpl.java:768)
          at org.hornetq.core.client.impl.FailoverManagerImpl.failoverOrReconnect(FailoverManagerImpl.java:730)
          at org.hornetq.core.client.impl.FailoverManagerImpl.handleConnectionFailure(FailoverManagerImpl.java:580)
          at org.hornetq.core.client.impl.FailoverManagerImpl.connectionDestroyed(FailoverManagerImpl.java:291)
          at org.hornetq.core.remoting.impl.netty.NettyConnector$Listener$1.run(NettyConnector.java:679)
          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
          at java.lang.Thread.run(Thread.java:619)
      Caused by: HornetQException[errorCode=2 message=Channel disconnected]
          ... 6 more

       

      Section 32.5 in the documentation does provide detail on XA recovery. However it seems to apply to JBoss 4.2.x onwards. Is there a similar machanism for JBoss 4.0.x? However I'm still not sure if this would fix my issue as my failover isn't actually happening during the scope of an XA transaction.

       

      Is anybody else using the JCA connector in this way with JBoss 4.0.2? I'm guessing somebody is so it may be simply a configuration error.

       

      Can anybody point me in the right direction? I don't want to put all of the configuration files up here initially. But if there is something specific that needs to be shown then please let me know and I will supply it.

       

      Really stuck on this one so any help is greatly appreciated.

       

      Grant

        • 1. Re: HornetQ Failover with JCA in JBoss
          ataylor

          its hard to give you any help without seeing your config, is reconnectattempts set to > 0, is your backup configured correctly, etc etc, i know people have had this working.

          • 2. Re: HornetQ Failover with JCA in JBoss
            grantlittle

            Hi Andy,

             

            Yeah I'm pretty sure its going to be configuration.

             

            Firstly to answer your questions:

             

            1. I have reconnectAttempts set to 5.
            2. I have a seperate non JCA client that listens to the same Hornet live/backup servers. They failover with no probs. So I'm taking that as the backup pair is working. Maybe a bad assumption!

             

            So I guess the best thing to do is provide configuration!

             

            jms-remote-ds.xml

             

            <?xml version="1.0" encoding="UTF-8"?>

             


            <!-- This file will be deployed by the target deploy on build.xml -->
            <connection-factories>

             


               <!--
                JMS XA Resource adapter, use this for outbound JMS connections.
                Inbound connections are defined at the @MDB activaction or at the resource-adapter properties.
               -->
               <tx-connection-factory>
                  <jndi-name>jms/XAConnectionFactory</jndi-name>
                  <xa-transaction/>
                  <rar-name>hornetq-ra.rar</rar-name>
                  <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>
                  <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>
                  <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>
                  <config-property name="ConnectionParameters" type="java.lang.String">port=6665</config-property>
                  <config-property name="BackupConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>
                  <config-property name="BackupConnectionParameters" type="java.lang.String">port=6675</config-property>
                  <config-property name="FailoverOnInitialConnection" type="java.lang.Boolean">false</config-property>
                  <max-pool-size>1</max-pool-size>
               </tx-connection-factory>

            </connection-factories>

             

            Primary server hornetq-configuration.xml

             

            <configuration xmlns="urn:hornetq"
                           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                           xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

            <connection-ttl-override>-1</connection-ttl-override>

             

                <jmx-management-enabled>true</jmx-management-enabled>

                <message-counter-enabled>true</message-counter-enabled>

                <message-counter-sample-period>2000</message-counter-sample-period>

                <message-counter-max-day-history>1</message-counter-max-day-history>

             

                <!--http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/duplicate-detection.html -->

                <!-- Default is 2000 -->

                <id-cache-size>10000</id-cache-size>

                <!-- Default is true -->

                <persist-id-cache>true</persist-id-cache>

             

                <clustered>false</clustered>

             

                <!-- Talk to the backup server -->

                <backup-connector-ref connector-name="backup-connector"/>

             

                <connectors>

                    <connector name="netty">

                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                        <param key="host" value="${hornetq.remoting.netty.host:hornetq-live}"/>

                        <param key="port" value="${hornetq.remoting.netty.port:6665}"/>

             

                    </connector>

             

                    <connector name="backup-connector">

                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                        <param key="host" value="${hornetq.remoting.netty.host:hornetq-backup}"/>

                        <param key="port" value="${hornetq.remoting.netty.port:6675}"/>

                  </connector>

             

                </connectors>

             

                <acceptors>

                    <acceptor name="netty">

                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                        <param key="host" value="${hornetq.remoting.netty.host:hornetq-live}"/>

                        <param key="port" value="${hornetq.remoting.netty.port:6665}"/>

                    </acceptor>

                </acceptors>

             

                <security-settings>

                    <security-setting match="#">

                        <permission type="createTempQueue" roles="guest"/>

                        <permission type="deleteTempQueue" roles="guest"/>

                        <permission type="consume" roles="guest"/>

                        <permission type="send" roles="guest"/>

                    </security-setting>

            </security-settings>

             

                <address-settings>

            <address-setting match="#">
                        <!-- Even though we have configured a dead letter destination -->
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <!-- Don't automaticcally put failed messages in the Dead Letter Queue -->
                        <max-delivery-attempts>-1</max-delivery-attempts>
                        <!-- How long does hornetq wait before attempting to rediliver a message -->
                        <!-- Default for shy queues is a retry every 5 seconds on failure.  This will
                             stop the logs from getting filled up. -->
                        <redelivery-delay>5000</redelivery-delay>
                        <!-- Specifies the queue to which expired messages are delivered. -->
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <!-- by default our queues are not last value queues. -->
                        <last-value-queue>false</last-value-queue>
                        <!-- Max queue size in memory is 200MiB (Mebibytes) -->
                        <!--<max-size-bytes>67108864</max-size-bytes>-->
                        <max-size-bytes>209715200</max-size-bytes>
                        <!-- Max page file size on disk is 100Mib (Mebibytes)-->
                        <!--<page-size-bytes>1073741824</page-size-bytes>-->
                        <page-size-bytes>104857600</page-size-bytes>
                        <!-- don't drop messages when full. We prefer catastrophic failure to deliberate message loss. -->
                        <!--<drop-messages-when-full>false</drop-messages-when-full>-->
                        <!-- How long to wait when the last consumer is closed on a queue before redistributing the messages. -->
                        <redistribution-delay>0</redistribution-delay>
                        <!-- The following settings control the collection of metrics -->
                        <message-counter-history-day-limit>1</message-counter-history-day-limit>
                        <!-- Behaviour to follow is a message does not end up being sent to a queue -->
                        <send-to-dla-on-no-route>false</send-to-dla-on-no-route>
                        <!-- Determines the behaviour that will happen when the queue fills up. -->
                        <address-full-policy>PAGE</address-full-policy>
                    </address-setting>

            </address-settings>

             

                <!-- Specifies where the various data files are stored. -->   
                <paging-directory>/tmp/hornetq-live/data/paging</paging-directory>
                <bindings-directory>/tmp/components/hornetq-live/data/bindings</bindings-directory>
                <journal-directory>/tmp/components/hornetq-live/data/journal</journal-directory>
                <large-messages-directory>/tmp/components/hornetq-live/data/large-messages</large-messages-directory>

             

            </configuration>

             

            Primary hornetq-beans.xml

             

            <?xml version="1.0" encoding="UTF-8"?>

             

            <deployment xmlns="urn:jboss:bean-deployer:2.0">

             

               <bean name="Naming" class="org.jnp.server.NamingBeanImpl"/>

             

               <!-- JNDI server. Disable this if you don't want JNDI -->
               <bean name="JNDIServer" class="org.jnp.server.Main">
                  <property name="namingInfo">
                     <inject bean="Naming"/>
                  </property>
                  <property name="port">6666</property>
                  <!-- doesnt seem to honour the naming, but rather needs client to attach via ip, so better to remove the restriction -->
                  <!--<property name="bindAddress">hornetq-live</property>-->
                  <property name="rmiPort">6668</property>
                  <!--<property name="rmiBindAddress">hornetq-live</property>-->
               </bean>
              
               <!-- MBean server -->
               <bean name="MBeanServer" class="javax.management.MBeanServer">
                  <constructor factoryClass="java.lang.management.ManagementFactory"
                               factoryMethod="getPlatformMBeanServer"/>

               </bean>

             

               <!-- The core configuration -->
               <bean name="Configuration" class="org.hornetq.core.config.impl.FileConfiguration">
               </bean>

             

                    <!-- The security manager -->
               <bean name="HornetQSecurityManager" class="org.hornetq.spi.core.security.HornetQSecurityManagerImpl">
                  <start ignored="true"/>
                  <stop ignored="true"/>
               </bean>

             

                    <!-- The core server -->
               <bean name="HornetQServer" class="org.hornetq.core.server.impl.HornetQServerImpl">
                  <constructor>
                     <parameter>
                        <inject bean="Configuration"/>
                     </parameter>
                     <parameter>
                        <inject bean="MBeanServer"/>
                     </parameter>
                     <parameter>
                        <inject bean="HornetQSecurityManager"/>
                     </parameter>       
                  </constructor>
                  <start ignored="true"/>
                  <stop ignored="true"/>
               </bean>
              
               <!-- The JMS server -->
               <bean name="JMSServerManager" class="org.hornetq.jms.server.impl.JMSServerManagerImpl">
                  <constructor>        
                     <parameter>
                        <inject bean="HornetQServer"/>
                     </parameter>        
                  </constructor>
               </bean>

             

            </deployment>

             

            Primary hornetq-jms.xml

             

            <configuration xmlns="urn:hornetq"
                           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                           xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">

             

                <!-- Please refer to
            http://www.jboss.org/hornetq/docs.html
            for details on the configuration parameters. -->

             

                <!-- Use this connection factory for queues/topics where you want unlimited consumers -->

                <connection-factory name="ConnectionFactoryUnlimitedConsumers">
                    <connectors>
                        <connector-ref connector-name="netty" backup-connector-name="backup-connector"/>           
                    </connectors>
                    <entries>
                        <entry name="jms/ConnectionFactory"/>
                    </entries>
                    <!-- When the server is cleanly shutdown we don't want to give up, we want to continue to try to reconnect -->
                     <!-- Multiply subsequent reconnect pauses by this multiplier. This can be used to
                      implement an exponential back-off. For our purposes we just set to 1.0 so each reconnect
                      pause is the same length -->
                      <retry-interval-multiplier>1.0</retry-interval-multiplier>

             

                      <!-- Try reconnecting an unlimited number of times (-1 means "unlimited") -->
                      <reconnect-attempts>5</reconnect-attempts>

             

                      <!-- When the server is cleanly shutdown we don't want to give up, we want to continue to try to reconnect -->
                      <failover-on-server-shutdown>true</failover-on-server-shutdown>

             

                      <!-- We need to specify a confirmation-window-size to enable re-attachment, default is -1 which means no re-attachment -->
                      <!--<confirmation-window-size>1048576</confirmation-window-size>-->

             

                      <client-failure-check-period>5000</client-failure-check-period>
                </connection-factory>

             

                <queue name="DLQ">
                    <entry name="/queue/DLQ"/>
                </queue>

             

                <queue name="MyQueue">
                    <entry name="/jms/MyQueue"/>

                </queue>

            </configuration>

             

            Backup hornetq-configuration.xml

             

            <configuration xmlns="urn:hornetq"

                           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                           xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

             

             

                 <backup>true</backup>

             

                 <shared-store>false</shared-store>

             

                <!-- Please refer to

                       http://www.jboss.org/hornetq/docs.html

                     for details on the configuration parameters. -->

             

                <!-- The connection-ttl-override setting controls how long a session remains active after the client disconnects

                     and before it is removed from the set of active sessions. -->

             

                <!-- For Test case scenarios may have to reduce this value so that test case

                     may be re-run, but for live scenarios needs to be bigger -->

             

                <!--

                    If value small, then when using listeners to a queue, appears that the server can kill the connection off early and

                    client will need to reconnect.  This puts lots of WARN level messages into the queue.

                    WARN - [org.springframework.jms.listener.DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'HornetQQueue[XDM_BT_SEND]' - trying to recover. Cause: Consumer is closed

                  -->

             

                <!-- See http://hornetq.sourceforge.net/docs/hornetq-2.0.0.BETA5/user-manual/en/html/connection-ttl.html -->

                <!-- The default value for connection ttl is 300000ms, i.e. 5 minutes. A value of -1 for ConnectionTTL means the server will never time out the connection on the server side. -->

                <connection-ttl-override>30000</connection-ttl-override>

             

                <jmx-management-enabled>true</jmx-management-enabled>

                <message-counter-enabled>true</message-counter-enabled>

                <message-counter-sample-period>2000</message-counter-sample-period>

                <message-counter-max-day-history>1</message-counter-max-day-history>

             

                <clustered>false</clustered>

             

                <connectors>

                    <connector name="netty">

                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

                        <param key="host" value="${hornetq.remoting.netty.host:hornetq-backup}"/>

                        <param key="port" value="${hornetq.remoting.netty.port:6675}"/>

                    </connector>

                </connectors>

             

                <acceptors>

                    <acceptor name="netty">

                        <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

                        <param key="host" value="${hornetq.remoting.netty.host:hornetq-backup}"/>

                        <param key="port" value="${hornetq.remoting.netty.port:6675}"/>

                    </acceptor>

                </acceptors>

             

                <security-settings>

                    <security-setting match="#">

                        <permission type="createTempQueue" roles="guest"/>

                        <permission type="deleteTempQueue" roles="guest"/>

                        <permission type="consume" roles="guest"/>

                        <permission type="send" roles="guest"/>

                    </security-setting>

                </security-settings>

             

                <address-settings>

                    <!-- default for catch all (leave as the last entry in the adress settings section)-->

                    <address-setting match="#">
                        <!-- Even though we have configured a dead letter destination -->
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <!-- Don't automaticcally put failed messages in the Dead Letter Queue -->
                        <max-delivery-attempts>-1</max-delivery-attempts>
                        <!-- How long does hornetq wait before attempting to rediliver a message -->
                        <!-- Default for shy queues is a retry every 5 seconds on failure.  This will
                             stop the logs from getting filled up. -->
                        <redelivery-delay>5000</redelivery-delay>
                        <!-- Specifies the queue to which expired messages are delivered. -->
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <!-- by default our queues are not last value queues. -->
                        <last-value-queue>false</last-value-queue>
                        <!-- Max queue size in memory is 200MiB (Mebibytes) -->
                        <!--<max-size-bytes>67108864</max-size-bytes>-->
                        <max-size-bytes>209715200</max-size-bytes>
                        <!-- Max page file size on disk is 100Mib (Mebibytes)-->
                        <!--<page-size-bytes>1073741824</page-size-bytes>-->
                        <page-size-bytes>104857600</page-size-bytes>
                        <!-- don't drop messages when full. We prefer catastrophic failure to deliberate message loss. -->
                        <!--<drop-messages-when-full>false</drop-messages-when-full>-->
                        <!-- How long to wait when the last consumer is closed on a queue before redistributing the messages. -->
                        <redistribution-delay>0</redistribution-delay>
                        <!-- The following settings control the collection of metrics -->
                        <message-counter-history-day-limit>1</message-counter-history-day-limit>
                        <!-- Behaviour to follow is a message does not end up being sent to a queue -->
                        <send-to-dla-on-no-route>false</send-to-dla-on-no-route>

                        <!-- Determines the behaviour that will happen when the queue fills up. -->
                        <address-full-policy>PAGE</address-full-policy>
                    </address-setting>
                </address-settings>

             

                <!-- Specifies where the various data files are stored. -->
                <paging-directory>/tmp/components/hornetq-backup/data/paging</paging-directory>
                <bindings-directory>/tmp/components/hornetq-backup/data/bindings</bindings-directory>
                <journal-directory>/tmp/components/hornetq-backup/data/journal</journal-directory>
                <large-messages-directory>/tmp/components/hornetq-backup/data/large-messages</large-messages-directory>

             

            </configuration>

             

            Backup hornetq-beans.xml

             

            <?xml version="1.0" encoding="UTF-8"?>

             

            <deployment xmlns="urn:jboss:bean-deployer:2.0">

             

               <bean name="Naming" class="org.jnp.server.NamingBeanImpl"/>

             

               <!-- JNDI server. Disable this if you don't want JNDI -->
               <bean name="JNDIServer" class="org.jnp.server.Main">
                  <property name="namingInfo">
                     <inject bean="Naming"/>
                  </property>
                  <property name="port">6676</property>
                  <!-- doesnt seem to honour the naming, but rather needs client to attach via ip, so better to remove the restriction -->
                  <!--<property name="bindAddress">hornetq-backup</property>-->
                  <property name="rmiPort">6678</property>
                  <!--<property name="rmiBindAddress">hornetq-backup</property>-->
               </bean>
              
               <!-- MBean server -->
               <bean name="MBeanServer" class="javax.management.MBeanServer">
                  <constructor factoryClass="java.lang.management.ManagementFactory"
                               factoryMethod="getPlatformMBeanServer"/>
               </bean>

             

               <!-- The core configuration -->
               <bean name="Configuration" class="org.hornetq.core.config.impl.FileConfiguration">
               </bean>

             

                    <!-- The security manager -->
               <bean name="HornetQSecurityManager" class="org.hornetq.spi.core.security.HornetQSecurityManagerImpl">
                  <start ignored="true"/>
                  <stop ignored="true"/>
               </bean>

             

                    <!-- The core server -->
               <bean name="HornetQServer" class="org.hornetq.core.server.impl.HornetQServerImpl">
                  <constructor>
                     <parameter>
                        <inject bean="Configuration"/>
                     </parameter>
                     <parameter>
                        <inject bean="MBeanServer"/>
                     </parameter>
                     <parameter>
                        <inject bean="HornetQSecurityManager"/>
                     </parameter>       
                  </constructor>
                  <start ignored="true"/>
                  <stop ignored="true"/>
               </bean>
              
               <!-- The JMS server -->
               <bean name="JMSServerManager" class="org.hornetq.jms.server.impl.JMSServerManagerImpl">
                  <constructor>        
                     <parameter>
                        <inject bean="HornetQServer"/>
                     </parameter>        
                  </constructor>
               </bean>

             

            </deployment>

             

            Backup hornetq-jms.xml

             

            <configuration xmlns="urn:hornetq"
                           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                           xsi:schemaLocation="urn:hornetq /schema/hornetq-jms.xsd">

            <!-- Use this connection factory for queues/topics where you want unlimited consumers -->
                <connection-factory name="ConnectionFactoryUnlimitedConsumers">
                    <connectors>
                        <connector-ref connector-name="netty" />
                    </connectors>
                    <entries>
                        <entry name="jms/ConnectionFactory"/>
                    </entries>
                    <retry-interval-multiplier>1.0</retry-interval-multiplier>
                    <failover-on-server-shutdown>true</failover-on-server-shutdown>
                    <client-failure-check-period>5000</client-failure-check-period>

             

                </connection-factory>

                <queue name="DLQ">
                    <entry name="/queue/DLQ"/>
                </queue>

             

                <queue name="MyQueue">
                    <entry name="/jms/MyQueue"/>

                </queue>

             

            </configuration>

             

            ra.xml inside hornetq-ra.rar

             

            <?xml version="1.0" encoding="UTF-8"?>

             

            <!-- This file will be installed by the example mdb-remote/build.xml, deploy target.
                 This is an example of how you could change the default configuration of a resource adapter

            -->

             

            <connector xmlns="http://java.sun.com/xml/ns/j2ee"
                       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                       xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
                       http://java.sun.com/xml/ns/j2ee/connector_1_5.xsd"
                       version="1.5">

             

               <description>HornetQ 2.0 Resource Adapter Alternate Configuration</description>
               <display-name>HornetQ 2.0 Resource Adapter Alternate Configuration</display-name>

             

               <vendor-name>Red Hat Middleware LLC</vendor-name>
               <eis-type>JMS 1.1 Server</eis-type>
               <resourceadapter-version>1.0</resourceadapter-version>

             

               <license>
                  <description>
            Copyright 2009 Red Hat, Inc.
            Red Hat licenses this file to you under the Apache License, version
            2.0 (the "License"); you may not use this file except in compliance
            with the License.  You may obtain a copy of the License at
               http://www.apache.org/licenses/LICENSE-2.0
            Unless required by applicable law or agreed to in writing, software
            distributed under the License is distributed on an "AS IS" BASIS,
            WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
            implied.  See the License for the specific language governing
            permissions and limitations under the License. 
                  </description>
                  <license-required>true</license-required>
               </license>

             

               <resourceadapter>
                  <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>
                  <config-property>
                     <description>The transport type</description>
                     <config-property-name>ConnectorClassName</config-property-name>
                     <config-property-type>java.lang.String</config-property-type>
                     <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>
                  </config-property>
                  <config-property>
                     <description>The transport configuration. These values must be in the form of key=val;key=val;</description>
                     <config-property-name>ConnectionParameters</config-property-name>
                     <config-property-type>java.lang.String</config-property-type>
                     <config-property-value>host=127.0.0.1;port=6665</config-property-value>
                  </config-property>
                  <config-property>
                     <description>The transport type</description>
                     <config-property-name>BackupConnectorClassName</config-property-name>
                     <config-property-type>java.lang.String</config-property-type>
                     <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>
                  </config-property>
                  <config-property>
                     <description>The transport configuration. These values must be in the form of key=val;key=val;</description>
                     <config-property-name>BackupTransportConfiguration</config-property-name>
                     <config-property-type>java.lang.String</config-property-type>
                     <config-property-value>host=127.0.0.1;port=6675</config-property-value>
                  </config-property>
                  <config-property>
                     <description>The transport configuration. These values must be in the form of key=val;key=val;</description>
                     <config-property-name>ReconnectAttempts</config-property-name>

            <config-property-type>java.lang.Integer</config-property-type>
                     <config-property-value>5</config-property-value>
                  </config-property>
                 
                  <outbound-resourceadapter>
                     <connection-definition>
                        <managedconnectionfactory-class>org.hornetq.ra.HornetQRAManagedConnectionFactory</managedconnectionfactory-class>

             

                        <config-property>
                           <description>The default session type</description>
                           <config-property-name>SessionDefaultType</config-property-name>
                           <config-property-type>java.lang.String</config-property-type>
                           <config-property-value>javax.jms.Queue</config-property-value>
                        </config-property>
                        <config-property>
                           <description>Try to obtain a lock within specified number of seconds; less than or equal to 0 disable this functionality</description>
                           <config-property-name>UseTryLock</config-property-name>
                           <config-property-type>java.lang.Integer</config-property-type>
                           <config-property-value>0</config-property-value>
                        </config-property>

             

                        <connectionfactory-interface>org.hornetq.ra.HornetQRAConnectionFactory</connectionfactory-interface>
                        <connectionfactory-impl-class>org.hornetq.ra.HornetQRAConnectionFactoryImpl</connectionfactory-impl-class>
                        <connection-interface>javax.jms.Session</connection-interface>
                        <connection-impl-class>org.hornetq.ra.HornetQRASession</connection-impl-class>
                     </connection-definition>
                     <transaction-support>XATransaction</transaction-support>
                     <authentication-mechanism>
                        <authentication-mechanism-type>BasicPassword</authentication-mechanism-type>
                        <credential-interface>javax.resource.spi.security.PasswordCredential</credential-interface>
                     </authentication-mechanism>
                     <reauthentication-support>false</reauthentication-support>
                  </outbound-resourceadapter>

             

                  <inbound-resourceadapter>
                     <messageadapter>
                        <messagelistener>
                           <messagelistener-type>javax.jms.MessageListener</messagelistener-type>
                           <activationspec>
                              <activationspec-class>org.hornetq.ra.inflow.HornetQActivationSpec</activationspec-class>
                              <required-config-property>
                                  <config-property-name>destination</config-property-name>
                              </required-config-property>
                           </activationspec>
                        </messagelistener>
                     </messageadapter>
                  </inbound-resourceadapter>

             

               </resourceadapter>
            </connector>

            • 3. Re: HornetQ Failover with JCA in JBoss
              ataylor

              are both servers running on localhost, i don't see this set in the *-ds.xml file

              • 4. Re: HornetQ Failover with JCA in JBoss
                grantlittle

                Thanks for the help Andy.

                 

                I don't have localhost configured however I do have two entries in my hosts file - hornetq-live & hornetq-backup, both of which resolve to 127.0.0.1.

                 

                Hopefully my configuration should be correctly configured with these entries.

                • 5. Re: HornetQ Failover with JCA in JBoss
                  ataylor

                  thats my point, in the jms-ds.xml config you havent got this configured, "
                        <config-property name="ConnectionParameters" type="java.lang.String">port=6665</config-property>", so will default to localhost, maybe this is the problem

                  • 6. Re: HornetQ Failover with JCA in JBoss
                    grantlittle

                    I changed the configuration to include the host names e.g

                     

                    <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>
                          <config-property name="ConnectionParameters" type="java.lang.String">host=hornetq-live;port=6665</config-property>
                          <config-property name="BackupConnectorClassName" type="java.lang.String">org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property>
                          <config-property name="BackupConnectionParameters" type="java.lang.String">host=hornetq-backup;port=6675</config-property>

                     

                    This didn't make any difference as at the end of the day in this situation they were resolving to the same server anyway (localhost). Obviously it wouldn't have worked if they were actually on different servers (as is our plan).

                     

                    I have debugged through the code a bit more and can see that once I kill the primary hornetq server the FailoverManagerImpl successully fails over to the backup (e.g connections and clients) all failover without issue. This makes me believe that this is not a connection issue.

                     

                    I believe the issue is around what happens after the failover has happened.

                     

                    The code references are from HornetQ 2.1.2.FINAL and jboss-4.0.2

                     

                    FailoverManagerImpl.java  

                     

                    730.     callFailureListeners(me, true);

                     

                    private void callFailureListeners(final HornetQException me, final boolean afterReconnect)

                       {

                          final List<SessionFailureListener> listenersClone = new ArrayList<SessionFailureListener>(listeners);

                     

                          for (final SessionFailureListener listener : listenersClone)

                          {

                             try

                             {

                                if (afterReconnect)

                                {

                                   listener.connectionFailed(me);

                                }

                                else

                                {

                                   listener.beforeReconnect(me);

                                }

                             }

                             catch (final Throwable t)

                             {

                                // Failure of one listener to execute shouldn't prevent others

                                // from

                                // executing

                                FailoverManagerImpl.log.error("Failed to execute failure listener", t);

                             }

                          }

                       }

                     

                     

                    HornetQConnection.java

                     

                    public synchronized void connectionFailed(final HornetQException me)
                          {
                             if (me == null)
                             {
                                return;
                             }

                     

                             HornetQConnection conn = connectionRef.get();

                     

                             if (conn != null)
                             {
                                try
                                {
                                   final ExceptionListener exceptionListener = conn.getExceptionListener();

                     

                                   if (exceptionListener != null)
                                   {
                                      final JMSException je = new JMSException(me.toString());

                     

                                      je.initCause(me);

                     

                                      new Thread(new Runnable()
                                      {
                                         public void run()
                                         {
                                                exceptionListener.onException(je);
                                         }
                                      }).start();
                                   }
                                }
                                catch (JMSException e)
                                {
                                   if (!conn.closed)
                                   {
                                      HornetQConnection.log.error("Failed to get exception listener", e);
                                   }
                                }
                             }
                          }

                     

                    HornetQRAManagedConnection.java

                     

                    /**
                        * Notifies user of a JMS exception.
                        * @param exception The JMS exception
                        */
                       public void onException(final JMSException exception)
                       {
                          if (HornetQRAManagedConnection.trace)
                          {
                             HornetQRAManagedConnection.log.trace("onException(" + exception + ")");
                          }

                     

                          if (isDestroyed.get())
                          {
                             if (HornetQRAManagedConnection.trace)
                             {
                                HornetQRAManagedConnection.log.trace("Ignoring error on already destroyed connection " + this, exception);
                             }
                             return;
                          }

                     

                          HornetQRAManagedConnection.log.warn("Handling JMS exception failure: " + this, exception);

                     

                          try
                          {
                             connection.setExceptionListener(null);
                          }
                          catch (JMSException e)
                          {
                             HornetQRAManagedConnection.log.debug("Unable to unset exception listener", e);
                          }

                     

                          ConnectionEvent event = new ConnectionEvent(this, ConnectionEvent.CONNECTION_ERROR_OCCURRED, exception);
                          sendEvent(event);

                       }

                     

                    /**
                        * Send an event.
                        * @param event The event to send.
                        */
                       protected void sendEvent(final ConnectionEvent event)
                       {
                          if (HornetQRAManagedConnection.trace)
                          {
                             HornetQRAManagedConnection.log.trace("sendEvent(" + event + ")");

                          }

                     

                          int type = event.getId();

                     

                          // convert to an array to avoid concurrent modification exceptions
                          ConnectionEventListener[] list = eventListeners.toArray(new ConnectionEventListener[eventListeners.size()]);

                     

                          for (ConnectionEventListener l : list)
                          {
                             switch (type)
                             {
                                case ConnectionEvent.CONNECTION_CLOSED:
                                   l.connectionClosed(event);
                                   break;

                     

                                case ConnectionEvent.LOCAL_TRANSACTION_STARTED:
                                   l.localTransactionStarted(event);
                                   break;

                     

                                case ConnectionEvent.LOCAL_TRANSACTION_COMMITTED:
                                   l.localTransactionCommitted(event);
                                   break;

                     

                                case ConnectionEvent.LOCAL_TRANSACTION_ROLLEDBACK:
                                   l.localTransactionRolledback(event);
                                   break;

                     

                                case ConnectionEvent.CONNECTION_ERROR_OCCURRED:
                                   l.connectionErrorOccurred(event);
                                   break;

                     

                                default:
                                   throw new IllegalArgumentException("Illegal eventType: " + type);
                             }
                          }
                       }

                     

                    TxConnectionManager$TxConnectionEventListener.java (from jboss 4.0.2)

                     

                    public void connectionErrorOccurred(ConnectionEvent ce)
                          {
                             currentTx = null;
                             super.connectionErrorOccurred(ce);
                          }

                     

                    BaseConnectionManager2.java (from jboss 4.0.2: abstract class - the concrete implementation is TxConnectionManager$TxConnectionEventListener)

                     

                    public void connectionErrorOccurred(ConnectionEvent ce)
                          {
                             if (ce != null)
                                log.warn("Connection error occured: " + this, ce.getException());
                             else
                                log.warn("Unknown Connection error occured: " + this);
                             try
                             {
                                unregisterConnections();
                             }
                             catch (Exception e)
                             {
                                //ignore, it wasn't checked out.
                             }
                             if (ce.getSource() != getManagedConnection())
                                log.warn("Notified of error on a different managed connection?");
                             returnManagedConnection(this, true);
                          }

                     

                    From my reading of the code, this means that JBoss will unregister all of the failed over connections and this is why I'm seeing the stacktrace that I supplied earlier.

                     

                    On next attempt to send a message to a queue, a new connection has to be established. However, as this is a new connection it attempts to connect to the primary hornetq server but obviously fails as it has been killed.

                    • 7. Re: HornetQ Failover with JCA in JBoss
                      grantlittle

                      To further test this I replaced the hornetq-core-client-java5.jar with a version I hacked.

                       

                      In the replacement jar I removed the call to

                       

                      callFailureListeners(me, true);

                       

                      I then re-ran my test case and it worked as expected. On sending my next message after failing over the sender worked as expected.

                       

                      This at least proves that its not a connection issue.

                       

                      But I'm now not sure if this is a bug! If others have indeed had this working then there must be something else I'm missing.

                      • 8. Re: HornetQ Failover with JCA in JBoss
                        ataylor

                        actually this may be a bug, i dont think the  txconnectionmanager should be unregistering the connection, could you raise a jira?

                        • 9. Re: HornetQ Failover with JCA in JBoss
                          grantlittle