1 2 Previous Next 16 Replies Latest reply on Feb 3, 2010 4:40 AM by ataylor

    Problem sending messages from MDB on failover

    radhikasivaraj

      Hi, I have an issue sending messages from MDB on failover.

       

      Version: HornetQ GA, JBoss 5.1.0

       

      I have two queues (requestQ, responseQ) in queue node1 and node2 is configured as backup for node1. MDB is deployed in node 3 and is not part of the cluster.

       

      When node1 is active, I'm able to send messages to requestQ of node1 and mdb picks the message from requestQ, process and sends a reply message to responseQ of node1, all went fine.

       

      When node1 crashes, backup node (node2) got activated and sender application was able to send messages to requestQ (of node1 and in turn to backup node2) and mdb also picked the message from node2, but then the MDB is unable to send the reply message to responseQ of node2. Pls see below for the exception that i get on failover.

       

       

       

      16:20:04,082 WARN [ChannelImpl] Can't find packet to clear: last received command id 9 first stored command id 0 16:20:54,591 WARN [RemotingConnectionImpl] Connection failure has been detected: Did not receive ping from /172.18.10.7:1446. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. The connection will now be closed. [code=3] 16:20:54,606 WARN [ServerSessionImpl] Client connection failed, clearing up resources for session 6e41324b-0bfa-11df-9152-001c23457f19 16:20:54,622 ERROR [ServerSessionImpl] Failed to close connection org.hornetq.core.server.impl.ServerSessionImpl@1f06283

       

      I could see from log that the mdb picks the request message but unable to send the reply message. I dont see this line of code getting executed on / after failover.

      conn = connectionFactory.createConnection();

       

      16:21:38,737 INFO [STDOUT] m1-1 sent to q1 consumed by mdb : 1

      16:21:38,737 INFO [STDOUT] About to send the response message

       

      16:26:38,883 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX -53edf57b:82e:4b616b18:1b in state RUN

      16:26:38,883 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -53edf57b:82e:4b616b18:1b invoked while multiple threads active within it.

      16:26:38,883 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -53edf57b:82e:4b616b18:1b aborting with 1 threads active!

      16:26:39,211 INFO [STDOUT] m1-1 sent to q1 consumed by mdb : 1

      16:26:39,211 INFO [STDOUT] About to send the response message

      16:26:39,383 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX -53edf57b:82e:4b616b18:1b in state CANCEL

      16:26:39,884 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_18] - TransactionReaper::check timeout for TX -53edf57b:82e:4b616b18:1b in state CANCEL_INTERRUPTED

      16:26:39,884 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_6] - TransactionReaper::check worker Thread[Thread-15,5,jboss] not responding to interrupt when cancelling TX -53edf57b:82e:4b616b18:1b -- worker marked as zombie and TX scheduled for mark-as-rollback

      16:26:39,884 WARN [arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.TransactionReaper_11] - TransactionReaper::check failed to mark TX -53edf57b:82e:4b616b18:1b as rollback only

      16:26:48,888 WARN [ClientConsumerImpl] Timed out waiting for handler to complete processing

      16:26:58,893 WARN [ClientConsumerImpl] Timed out waiting for handler to complete processing

       

      I took the sample code from jca-config example of Hornetq distribution for sending messages from MDB but then the same code does not work in case of failover even after adding relevant params in ra.xml/ remote-jms-ds.xml. Am I missing anything?

       

      Attached is the MDB code, hornetq-configuration.xml, ra.xml and remote-jms-ds.xml of node3. Pls help

       

       

        • 1. Re: Problem sending messages from MDB on failover
          ataylor
          Looking at the ra.xml it is only configured for one noe. Take a look at the User manual section on JCA for how to configure it correctly.
          • 2. Re: Problem sending messages from MDB on failover
            radhikasivaraj

            Andy, I've configured both the active node ip and the back up node ip in ra.xml as below

             

            This is for node1

             

            <config-property>

            <description>The transport configuration. These values must be in the form of key=val;key=val;</description>

            <config-property-name>ConnectionParameters</config-property-name>

            <config-property-type>java.lang.String</config-property-type>

            <config-property-value>host=172.18.10.7;port=5445</config-property-value>

            </config-property>

             

            This is for node2

             

            <config-property>

            <description>The Backup transport configuration. These values must be in the form of key=val;key=val;</description>

            <config-property-name>BackupTransportConfiguration</config-property-name>

            <config-property-type>java.lang.String</config-property-type>

            <config-property-value>host=172.18.10.137;port=5445</config-property-value>

            </config-property>

             

            Is this not enough? Can you pls let me know what is that I'm missing? Thanks!

            • 3. Re: Problem sending messages from MDB on failover
              radhikasivaraj

              Also the issue is only with outbound connections from mdb that too happens only on/after failover. Also I've given both the live and backup node details in jms-remote-ds.xml as well.

               

              <config-property name="ConnectorClassName" type="java.lang.String">org.hornetq.integration.transports.netty.NettyConnectorFactory</config-property>

              <config-property name="ConnectionParameters" type="java.lang.String">host=172.18.10.7;port=5445</config-property>

              <config-property name="BackupConnectorClassName" type="java.lang.String">org.hornetq.integration.transports.netty.NettyConnectorFactory</config-property>

              <config-property name="BackupTransportConfiguration" type="java.lang.String">host=172.18.10.137;port=5445</config-property>

               

              • 4. Re: Problem sending messages from MDB on failover
                ataylor
                actually your config looks correct, my apolgies. I will take a look at this today and try to recreate the issue. leave it with me
                • 5. Re: Problem sending messages from MDB on failover
                  radhikasivaraj

                  Thanks a lot...your fix is "most wanted" now

                  • 6. Re: Problem sending messages from MDB on failover
                    ataylor

                    Ok, you will need to add the following config to the ra.xml file.

                     

                    <config-property>
                            <description>Should clean server shutdown trigger failover?</description>
                            <config-property-name>FailoverOnServerShutdown</config-property-name>
                            <config-property-type>java.lang.Boolean</config-property-type>
                            <config-property-value>true</config-property-value>
                    </config-property>

                     

                     

                     

                    • 7. Re: Problem sending messages from MDB on failover
                      timfox

                      This is actually discussed in a "yellow note" box in the user manual:

                       

                      http://hornetq.sourceforge.net/docs/hornetq-2.0.0.GA/user-manual/en/html/ha.html#ha.automatic.failover

                       

                      "By default, cleanly shutting down the server will                             not trigger failover on the client.

                      Using CTRL-C on a HornetQ server or JBoss AS instance causes the server to                             cleanly shut down, so will not trigger                         failover on the client.

                      If you want the client to failover when its server is cleanly shutdown                         then you must set the property FailoverOnServerShutdown to true"

                      • 8. Re: Problem sending messages from MDB on failover
                        radhikasivaraj

                        Hi, I'm not using ctrl-c to stop the live node, i kill using ctrl-alt-del /end process java.exe of live node.

                         

                        Also i could see that the failover happens and the backup server gets activated after killing the live node. MDB is also able to consume messages from the backup node (I could see that from JMX console of backup node), it is only that the MDB is unable to send the response message to the backup node.

                         

                        I also added the below code in queue live node (node1) and used both crtl-c/ ctrl-alt-del approach and in both the cases the MDB is unable to send a response message after failover.

                         

                        <config-property>
                        <description>Should clean server shutdown trigger failover?</description>
                        <config-property-name>FailoverOnServerShutdown</config-property-name>
                        <config-property-type>java.lang.Boolean</config-property-type>
                        <config-property-value>true</config-property-value>
                        </config-property>

                         

                        Pls let me know if you would like to see the queue nodes configuration files.

                        • 9. Re: Problem sending messages from MDB on failover
                          ataylor
                          I tested this and it failed over for me, however i didnt test receiving any messages. let me try.
                          • 10. Re: Problem sending messages from MDB on failover
                            radhikasivaraj

                            BTW, I've attached log files (before-failover.log and after-failover.log) after enabling trace for org.hornetq folder.

                             

                            I see that the below message is printed continously in the "after-failover.log" when it tries to get a connection to RemoteJmsXA and nothing happens after that.

                             

                            2010-02-01 14:43:39,351 DEBUG [org.hornetq.integration.transports.netty.NettyConnector] (Thread-8 (group:HornetQ-client-global-threads-5974682)) Started Netty Connector version 3.1.5.GA-r1772

                            From the logs i could trace the code and this is the line where it fails after failover

                             

                            HornetQRAManagedConnection.setup() {

                            ................

                             

                             

                            xaConnection = mcf.getHornetQConnectionFactory().createXAQueueConnection();

                            xaConnection = mcf.getHornetQConnectionFactory().createXAQueueConnection();

                             

                             

                             

                            which is not happening in the before-failover scenario, (pls see before-failover.log)

                            • 11. Re: Problem sending messages from MDB on failover
                              radhikasivaraj

                              Hi, to be specific, i see the issue is only with conn.createSession method while sending a reply message from mdb. Below is the code snippet from mdb.

                               

                              @Resource

                               

                              (mappedName = "java:RemoteJmsXA")

                               

                              private ConnectionFactory connectionFactory;

                               

                              conn = getConnectionFactory().createConnection();

                               

                              System.

                              out.println("1. created connection : " + conn);

                               

                              Session sess = conn.createSession(false, Session.AUTO_ACKNOWLEDGE); //This is where it goes into an infinite loop

                               

                               

                               

                              Queue destQueueA = sess.createQueue("responseQ");

                               

                              MessageProducer prodA = sess.createProducer(destQueueA);

                               

                              prodA.send(sess.createTextMessage(message));

                               

                              • 12. Re: Problem sending messages from MDB on failover
                                radhikasivaraj

                                ps:

                                 

                                actually i meant

                                 

                                 

                                 

                                conn = connectionFactory.createConnection(); //this works fine and gets a connection to backup server

                                 

                                • 13. Re: Problem sending messages from MDB on failover
                                  ataylor
                                  • 14. Re: Problem sending messages from MDB on failover
                                    radhikasivaraj

                                    Hi, Thanks a lot for adding this in JIRA as we badly need this feature.

                                     

                                    May I know when 2.1.0.beta1 will be available for download and also can we use this version in our production environment?

                                     

                                    For now, is there any workaround available for this issue?

                                     

                                    Thanks

                                    Radhika

                                    1 2 Previous Next