6 Replies Latest reply on Feb 13, 2012 10:05 AM by Celia Wang

    HornetQ live-Backup failover on JBoss AS 7

    Celia Wang Newbie

      Hello everyone, I am not a native English speaker and I'm new to here. I really need some help, please help me~thanks.

      I had spent a lot of time on the HornetQ User Manual but it seems like that there are some problems happened.

      What I want to do is that live server can failover to backup server when client is sending messages.

       

      My problem is: when I killed the live server, my backup server could not take over and become live.

       

      I have 4 servers -

      A: HornetQ Live Server on JBoss AS7 (jboss-as-7.0.1.Final)

           deploy mdb-ear-1.0.0.ear (andytaylor-HornetQ-AS7-examples-1a5935b  , it's a mdb example)

      B: HornetQ Backup Server on JBoss AS7

           deploy mdb-ear-1.0.0.ear , but backup server is lock, this deployment will not be active until backup server become live

      C: a client running MDB example, send 2000 of messages to live server  (andytaylor-HornetQ-AS7-examples-1a5935b/mdb/mdb-client)

      D: shared store - NFS, to store A&B's message journal

       

      The method of executing server:

      A: ./standalone.sh --server-config=standalone-preview-ha-live.xml

      B: ./standalone.sh --server-config=standalone-preview-ha-back.xml

      C: mvn -Pclient test

       

      Here is the configurations changed on A & B

      standalone-preview-ha-live.xml & standalone-preview-ha-back.xml are modified from standalone-preview-ha.xml

      where the font color is red is my modified for configurations, others are the same as standalone-preview-ha.xml

      A: live server -

      <subsystem xmlns="urn:jboss:domain:messaging:1.0">

                  <!-- Default journal file size is 10Mb, reduced here to 100k for faster first boot -->

                  <journal-file-size>102400</journal-file-size>

                  <journal-min-files>2</journal-min-files>

                  <journal-type>NIO</journal-type>

                  <!-- disable messaging persistence -->

                  <persistence-enabled>false</persistence-enabled>

      <!--add hornetq-configuration set live here-->

                <clustered>true</clustered>

                <failover-on-shutdown>false</failover-on-shutdown>

                <shared-store>true</shared-store>

      <!--end here-->

      ---

      skip

      ---

               <jms-destinations>

                      <jms-queue name="testQueue">

                         <entry name="queue/test"/>

                      </jms-queue>

                                       <jms-queue name="mdbQueue">

                          <entry name="queue/mdbQueue"/>

                      </jms-queue>

                      <jms-queue name="mdbReplyQueue">

                          <entry name="queue/mdbReplyQueue"/>

                      </jms-queue>

                      <jms-topic name="testTopic">

                         <entry name="topic/test"/>

                      </jms-topic>

                  </jms-destinations>

      ---

      skip

      ---

           <interfaces>

              <interface name="management">

                  <inet-address value="140.XXX.XX.231"/>

              </interface>

              <interface name="public">

                 <inet-address value="140.XXX.XX.231"/>

              </interface>

          </interfaces>

      B: backup server -

              <subsystem xmlns="urn:jboss:domain:messaging:1.0">

                  <!-- Default journal file size is 10Mb, reduced here to 100k for faster first boot -->

                  <journal-file-size>102400</journal-file-size>

                  <journal-min-files>2</journal-min-files>

                  <journal-type>NIO</journal-type>

                  <!-- disable messaging persistence -->

                  <persistence-enabled>false</persistence-enabled>

      <!--add hornetq-configuration set backup here -->

          <clustered>true</clustered>

                    <failover-on-shutdown>false</failover-on-shutdown>

                   <backup>true</backup>

          <shared-store>true</shared-store>

      <!--end here-->

                  <connectors>

                     <netty-connector name="netty" socket-binding="messaging" />

                     <netty-connector name="netty-throughput" socket-binding="messaging-throughput">

                        <param key="batch-delay" value="50"/>

                     </netty-connector>

                     <in-vm-connector name="in-vm" server-id="0" />

                  </connectors>

      ---

      skip

      ---

               <jms-destinations>

                      <jms-queue name="testQueue">

                         <entry name="queue/test"/>

                      </jms-queue>

                      <jms-queue name="mdbQueue">

                          <entry name="queue/mdbQueue"/>

                      </jms-queue>

                      <jms-queue name="mdbReplyQueue">

                          <entry name="queue/mdbReplyQueue"/>

                      </jms-queue>

                      <jms-topic name="testTopic">

                         <entry name="topic/test"/>

                      </jms-topic>

                  </jms-destinations>

      ---

      skip

      ---

           <interfaces>

              <interface name="management">

                  <inet-address value="140.XXX.XX.232"/>

              </interface>

              <interface name="public">

                 <inet-address value="140.XXX.XX.232"/>

              </interface>

          </interfaces>

      C: I modified "MDBClient.java", in order to send messages to the live server.

       

                final int numMessages = 2000;

                HashMap map = new HashMap();

                map.put("host", "140.XXX.XX.231");

                map.put("port", 5445);

                TransportConfiguration transportConfiguration = new TransportConfiguration(NettyConnectorFactory.class.getName(),map);

      ---skip---    

           TextMessage message=null;

            //Step 7. Create a Text Message

           for (int i = 0; i < numMessages; i++)

           {

                         message = session.createTextMessage("This is a text message"+ i);

                //Step 8. Send the Message

                         producer.send(message);

                         System.out.println("Sent message: " + message.getText());

           }

       

      D: shared store - NFS

      The NFS directory is /home/user/jboss-as-7.0.1.Final/standalone/data

      ==================================

      Now, after live & backup server is started, I executed "MDBClient.java", sending 2000 messages to the live server.

      Then, I went to the live server and killed its process id.

      jps

      kill XXXXX

      My backup server did not become live, but after some minutes, there are a lot of WARN on live server's boot.log & backup server's server.log

       

      WARN  [com.arjuna.ats.jta] ARJUNA16038: No XAResource to recover < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ebf, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ec0, eis_name=unknown eis name >

       

      Here is the server.log on live server.

      14:32:55,668 INFO  [org.apache.catalina.core.AprLifecycleListener] (MSC service thread 1-4) The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/jdk1.6.0_24/jre/lib/amd64/server:/opt/jdk1.6.0_24/jre/lib/amd64:/opt/jdk1.6.0_24/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

      14:32:55,701 INFO  [org.jboss.as.jmx.JMXConnectorService] (MSC service thread 1-1) Starting remote JMX connector

      14:32:55,707 INFO  [org.jboss.as.remoting] (MSC service thread 1-7) Listening on /140.XXX.XX.231:9999

      14:32:55,738 INFO  [org.jboss.as.ee] (Controller Boot Thread) Activating EE subsystem

      14:32:55,742 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-5) Starting Coyote HTTP/1.1 on http-WMC-Cloud-A-140.XXX.XX.231-8080

      14:32:55,751 INFO  [org.jboss.ws.common.management.AbstractServerConfig] (MSC service thread 1-6) JBoss Web Services - Stack CXF Server 4.0.0.Beta3

      14:32:55,889 INFO  [org.jboss.as.modcluster] (MSC service thread 1-4) Mod_cluster uses default load balancer provider

      14:32:55,963 INFO  [org.jboss.modcluster.ModClusterService] (MSC service thread 1-4) Initializing mod_cluster 1.1.3.Final

      14:32:55,982 INFO  [org.jboss.as.connector] (MSC service thread 1-2) Starting JCA Subsystem (JBoss IronJacamar 1.0.3.Final)

      14:32:55,986 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (MSC service thread 1-4) Listening to proxy advertisements on 224.0.1.105:23364

      14:32:56,045 INFO  [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-5) Bound data source [java:jboss/datasources/ExampleDS]

      14:32:56,056 INFO  [org.jboss.as.jacorb] (MSC service thread 1-3) CORBA ORB Service Started

      14:32:56,126 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) live server is starting with configuration HornetQ Configuration (clustered=true,backup=false,sharedStore=true,journalDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingjournal,bindingsDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingbindings,largeMessagesDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messaginglargemessages,pagingDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingpaging)

      14:32:56,161 \u8b66\u544a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

      14:32:56,172 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-1) Registered connection factory java:/JmsXA

      14:32:56,193 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-1) HornetQ resource adaptor started

      14:32:56,193 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-1) IJ020002: Deployed: file://RaActivatorhornetq-ra

      14:32:56,209 INFO  [org.jboss.as.jacorb] (MSC service thread 1-3) CORBA Naming Service Started

      14:32:56,304 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-5) Started Netty Acceptor version 3.2.3.Final-r${buildNumber} WMC-Cloud-A:5455 for CORE protocol

      14:32:56,305 INFO  [org.hornetq.core.remoting.impl.netty.NettyAcceptor] (MSC service thread 1-5) Started Netty Acceptor version 3.2.3.Final-r${buildNumber} WMC-Cloud-A:5445 for CORE protocol

      14:32:56,306 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) Server is now live

      14:32:56,306 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [b4c2c38f-3c18-11e1-8dd2-e41f13246ff6] started

      14:32:56,308 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-8) trying to deploy queue jms.topic.testTopic

      14:32:56,384 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-8) Bound messaging object to jndi name java:/topic/test

      14:32:56,395 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-2) Bound messaging object to jndi name java:/RemoteConnectionFactory

      14:32:56,397 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-1) Bound messaging object to jndi name java:/ConnectionFactory

      14:32:56,397 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) trying to deploy queue jms.queue.mdbQueue

      14:32:56,421 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-5) Bound messaging object to jndi name java:/queue/mdbQueue

      14:32:56,422 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-7) trying to deploy queue jms.queue.testQueue

      14:32:56,435 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-7) Bound messaging object to jndi name java:/queue/test

      14:32:56,435 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-4) trying to deploy queue jms.queue.mdbReplyQueue

      14:32:56,446 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-4) Bound messaging object to jndi name java:/queue/mdbReplyQueue

      14:32:56,623 INFO  [org.jboss.as.deployment] (MSC service thread 1-4) Started FileSystemDeploymentService for directory /home/wmc/jboss-as-7.0.1.Final/standalone/deployments

      14:32:56,641 INFO  [org.jboss.as] (Controller Boot Thread) JBoss AS 7.0.1.Final "Zap" started in 2665ms - Started 128 of 208 services (80 services are passive or on-demand)

      14:32:56,652 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) Starting deployment of "my-webapp.war"

      14:32:56,652 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-8) Starting deployment of "mdb-ear-1.0.0.ear"

      14:32:56,692 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-3) Starting deployment of "mdb-jar-1.0.0.jar"

      14:32:56,717 INFO  [org.jboss.as.jpa] (MSC service thread 1-3) added javax.persistence.api dependency to my-webapp.war

      14:32:56,720 INFO  [org.jboss.as.jpa] (MSC service thread 1-5) added javax.persistence.api dependency to mdb-ear-1.0.0.ear

      14:32:56,723 INFO  [org.jboss.as.jpa] (MSC service thread 1-5) added javax.persistence.api dependency to mdb-jar-1.0.0.jar

      14:32:56,875 INFO  [org.jboss.web] (MSC service thread 1-4) registering web context: /my-webapp

      14:32:56,923 INFO  [org.jboss.as.server.controller] (DeploymentScanner-threads - 2) Deployed "mdb-ear-1.0.0.ear"

      14:32:56,923 INFO  [org.jboss.as.server.controller] (DeploymentScanner-threads - 2) Deployed "my-webapp.war"

      14:33:05,921 INFO  [org.jboss.modcluster.ModClusterService] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Engine [jboss.web] will use jvmRoute: d883de1f-fe17-3214-8442-4e4880a8781a

      ---after client send messages---

      14:34:10,802 INFO  [stdout] (Thread-3 (group:HornetQ-client-global-threads-2000502626)) message This is a text message2

      14:34:10,805 INFO  [stdout] (Thread-3 (group:HornetQ-client-global-threads-2000502626)) connection created

      14:34:10,815 INFO  [stdout] (Thread-9 (group:HornetQ-client-global-threads-2000502626)) message This is a text message8

      14:34:10,815 INFO  [stdout] (Thread-9 (group:HornetQ-client-global-threads-2000502626)) connection created

      14:34:10,815 INFO  [stdout] (Thread-2 (group:HornetQ-client-global-threads-2000502626)) message This is a text message1

      14:34:10,816 INFO  [stdout] (Thread-2 (group:HornetQ-client-global-threads-2000502626)) connection created

      ---skip---kill the live server

      14:36:32,377 INFO  [org.apache.coyote.http11.Http11Protocol] (MSC service thread 1-5) Stopping Coyote HTTP/1.1 on http-WMC-Cloud-A-140.116.82.231-8080

      14:36:32,378 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-7) Unbound messaging object from jndi name java:/RemoteConnectionFactory

      14:36:32,380 INFO  [org.jboss.as.messaging.jms.AS7BindingRegistry] (MSC service thread 1-6) Unbound messaging object from jndi name java:/topic/test

       

      boot.log on live server (because live server is shutdown)

      14:32:55,322 INFO  [org.jboss.as.logging] Removing bootstrap log handlers

      The followings are the WARNs and ERRORs after the live server killed.

      14:36:32,398 INFO  [org.jboss.as.logging] Restored bootstrap log handlers

      14:36:32,415 INFO  [org.jboss.modcluster.ModClusterService] All active sessions drained from context [/my-webapp] in 0 seconds

      14:36:32,421 INFO  [org.jboss.as.jmx.JMXConnectorService] JMX remote connector stopped

      14:36:32,423 INFO  [stdout] message This is a text message415

      14:36:32,423 INFO  [stdout] connection created

      14:36:32,426 INFO  [org.jboss.modcluster.ModClusterService] All active sessions drained from context [/my-webapp] in 0 seconds

      14:36:32,429 INFO  [stdout] session created

      14:36:32,430 ERROR [stderr] javax.jms.InvalidDestinationException: Destination mdbReplyQueue does not exist

      14:36:32,430 ERROR [stderr]           at org.hornetq.jms.client.HornetQSession.createProducer(HornetQSession.java:337)

      14:36:32,430 ERROR [stderr]           at org.hornetq.ra.HornetQRASession.createProducer(HornetQRASession.java:1163)

      14:36:32,430 ERROR [stderr]           at org.hornetq.example.MDBSendExample.onMessage(MDBSendExample.java:77)

      --skip--

      14:36:32,608 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] HornetQ Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [b4c2c38f-3c18-11e1-8dd2-e41f13246ff6] stopped

      14:37:02,575 WARN  [com.arjuna.ats.jta] ARJUNA16086: TransactionImple.enlistResource setTransactionTimeout on XAResource < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ee7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ee8, eis_name=unknown eis name > threw: XAException.XAER_RMERR: javax.transaction.xa.XAException

      --skip

      14:37:02,599 ERROR [org.hornetq.core.client.impl.ClientSessionImpl] Caught jmsexecptione : HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 52]

      --skip

      14:37:02,605 WARN  [com.arjuna.ats.jta] ARJUNA16041: prepare on < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:edf, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ee0, eis_name=unknown eis name > (org.hornetq.core.client.impl.DelegatingSession@4aa594e1) failed with exception XAException.XAER_RMERR: javax.transaction.xa.XAException

      --skip

      14:37:02,610 WARN  [com.arjuna.ats.arjuna] ARJUNA12073: BasicAction.End() - prepare phase of action-id 0:ffff8c7452e7:7386542f:4f0d2d17:edf failed.

      14:37:02,610 WARN  [com.arjuna.ats.arjuna] ARJUNA12075: Action Aborting

      14:37:02,633 WARN  [org.hornetq.core.client.impl.ClientSessionImpl] failover occured during commit throwing XAException.XA_RETRY

      14:37:02,634 WARN  [com.arjuna.ats.jta] ARJUNA16036: commit on < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:eb7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:eb8, eis_name=unknown eis name > (org.hornetq.core.client.impl.DelegatingSession@58fe210a) failed with exception $XAException.XA_RETRY: javax.transaction.xa.XAException

       

      Here is the server.log on backup server.

      14:35:36,363 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (MSC service thread 1-5) backup server is starting with configuration HornetQ Configuration (clustered=true,backup=true,sharedStore=true,journalDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingjournal,bindingsDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingbindings,largeMessagesDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messaginglargemessages,pagingDirectory=/home/wmc/jboss-as-7.0.1.Final/standalone/data/messagingpaging)

      14:35:36,365 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-5) Caching command for createQueue for mdbReplyQueue since the JMS Server is not active yet

      14:35:36,367 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-5) Caching command for createConnectionFactory for RemoteConnectionFactory since the JMS Server is not active yet

      14:35:36,367 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-5) Caching command for createQueue for testQueue since the JMS Server is not active yet

      14:35:36,367 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-5) Caching command for createQueue for mdbQueue since the JMS Server is not active yet

      14:35:36,368 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-6) Caching command for createConnectionFactory for InVmConnectionFactory since the JMS Server is not active yet

      14:35:36,368 INFO  [org.hornetq.core.server.impl.FileLockNodeManager] (Thread-18) Waiting to become backup node

      14:35:36,368 \u8cc7\u8a0a    [org.hornetq.jms.server.impl.JMSServerManagerImpl] (MSC service thread 1-2) Caching command for createTopic for testTopic since the JMS Server is not active yet

      14:35:36,396 INFO  [org.jboss.as.jacorb] (MSC service thread 1-1) CORBA Naming Service Started

      14:35:36,401 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-3) Registered connection factory java:/JmsXA

      14:35:36,407 INFO  [org.hornetq.ra.HornetQResourceAdapter] (MSC service thread 1-3) HornetQ resource adaptor started

      14:35:36,408 INFO  [org.jboss.as.connector.services.ResourceAdapterActivatorService$ResourceAdapterActivator] (MSC service thread 1-3) IJ020002: Deployed: file://RaActivatorhornetq-ra

      14:35:36,427 INFO  [org.hornetq.core.server.impl.FileLockNodeManager] (Thread-18) ** got backup lock

      14:35:36,443 \u8b66\u544a    [org.hornetq.core.server.impl.HornetQServerImpl] (Thread-18) Security risk! It has been detected that the cluster admin user and password have not been changed from the installation default. Please see the HornetQ user guide, cluster chapter, for instructions on how to do this.

      14:35:36,487 \u8cc7\u8a0a    [org.hornetq.core.server.impl.HornetQServerImpl] (Thread-18) HornetQ Backup Server version 2.2.7.Final (HQ_2_2_7_FINAL_AS7, 121) [b4c2c38f-3c18-11e1-8dd2-e41f13246ff6] started, waiting live to fail before it gets active

      14:35:36,843 INFO  [org.jboss.as.deployment] (MSC service thread 1-5) Started FileSystemDeploymentService for directory /home/wmc/jboss-as-7.0.1.Final/standalone/deployments

      14:35:36,860 INFO  [org.jboss.as] (Controller Boot Thread) JBoss AS 7.0.1.Final "Zap" started in 2572ms - Started 128 of 208 services (80 services are passive or on-demand)

      14:35:46,162 INFO  [org.jboss.modcluster.ModClusterService] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Engine [jboss.web] will use jvmRoute: a8c846d5-8c74-327b-ba50-723177f332e2

      --the following are the WARNs after live server killed,

      14:37:56,286 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA16037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed8, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@78221c75 >

      14:37:56,287 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA16037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:edc, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@5271b383 >

      14:37:56,288 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA16038: No XAResource to recover < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed8, eis_name=unknown eis name >

      14:37:56,288 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA16038: No XAResource to recover < formatId=131076, gtrid_length=29, bqual_length=28, tx_uid=0:ffff8c7452e7:7386542f:4f0d2d17:ed7, node_name=1, branch_uid=0:ffff8c7452e7:7386542f:4f0d2d17:edc, eis_name=unknown eis name >

      --it will receive the same WARN repeately---

      =================================

      The live & backup server will always receive the same WARNs repeately, it seems like a deadlock.

      However, the backup server did not become live.

      Do I miss any other configurations on live & backup server?

      Does anyone knows how to successfully set live & backup server failover?

      I'm sorry about that I post a long description of my problem, if there is any question about my problem, just ask.

      Thanks for your help!

        • 1. Re: HornetQ live-Backup failover on JBoss AS 7
          rodakr Novice

          You don't need Live Backup for high availability .. :-)

          Active Active  Queue/Topic on both Instances ,  produced Messages are round robine distributed over both nodes.

          If one Node crash, producer can still produce, consumer can still consume...pretty much without interruption ( except some timeout for node failure detection maby ).

          So Service is still available...

          Once you start crashed node, persistent messages in store will be consumed.... if you don't have persistent messages, they are lost with node crash...

           

          live-backup failover and back might have some impact at performance....

           

          look hornetQ docs for HA and xml Element cluster-connections...

          1 of 1 people found this helpful
          • 2. Re: HornetQ live-Backup failover on JBoss AS 7
            Celia Wang Newbie

            Hello, Radek Rodak

            I was too busy to give you a reply~thanks for your advice. 

            I know there are other method to achieve HA , but... someone just want to try the live-backup failover.

            Now, I had already make the backup server become live.

            I just add some backup server's configurations(with HashMap and TransportConfiguration) in MDBClient.java.

            Then, after failover occured,  backup server will become live, but the backup server will not redeploy the MDBcase application.

            That means the status of deployments is still failed, so I need to redeploy it by myself.

            Do you know how to make the backup server redeploy its applications automatically when it become live?

            Thanks for your help!

            • 3. Re: HornetQ live-Backup failover on JBoss AS 7
              rodakr Novice

              Hi Wang

               

              I'm not so familiar with live-backup solution.  On my active active cluster deployments are working fine on both cluster nodes on 2 different linux servers.

              I started from standalone-ha.xml from standalone and this works perfekt.

              Check with yout live-back solution if both nodes have access to JMS stores, which should be on clustered file system like GFS2.

              You can configure in standalone-ha.xml the location.... if you don't do this, I think default is local on each nodes which can't work for live-backup failover, because both nodes need to be able to access to file based jmsstores for live-backup failover.

              • 4. Re: HornetQ live-Backup failover on JBoss AS 7
                Aparna Chaudhary Newbie

                I'm also having a similar problem. I'm trying to configure live-backup failover on jboss-as-7.1.0.Final-SNAPSHOT. On failure of live server, the backup server comes live, creates JMS queue; but it doesn't redeploy the .war artifact automatically. Failover works fine if I redeploy the war manually. Is this feature not supported yet, or 'm I missing something?

                 

                Regards,

                Aparna

                • 5. Re: HornetQ live-Backup failover on JBoss AS 7
                  Celia Wang Newbie

                  Thanks for your reply.

                  I didn't use GFS2, but I use NFS to store live-backup's messaging journals.

                  and I mount NFS on the directory - /home/user/jboss-as-7.0.1.Final/standalone/data

                  I'm sure that both live and backup servers can access this nfs directory.

                  I know using SAN-like shared file system is better, but I don't care the problem of efficiency now.

                  Does the difference of NFS and GFS2 may cause this kind of problem?

                  • 6. Re: HornetQ live-Backup failover on JBoss AS 7
                    Celia Wang Newbie

                    Hello, Aparna

                    I'm glad that someone has a similar problem with me.

                    I cannot find any solution to solve this problem in JBoss yet.

                    When the backup server becomes live, it will not redeploy its applications.

                    This means the status of the deployment is still failed.

                    Now, I write a shell script on the backup server.

                    It will check whether the live server is alive every one minute,if the live server is dead, kill the status file(rm /~/*.failed), then it will redeploy...

                    (I know it's not a good method.)

                    but I still want to find how to solve this problem by JBoss, or maybe it really did not support this feature.