7 Replies Latest reply on Aug 10, 2010 12:10 PM by clebert.suconic

Problem with Live-backup

joydeep.sarkar Aug 5, 2010 6:03 AM

Hello,

I am facing a problem with live-backup pair where the back up is not being recognized when the live is going down.

I have three servers,

10.240.132.213 (Acting as client and looking up using JmsNonXA)

10.240.132.215 (Live Server)

10.240.132.216 (Backup server).

Following are the configuration files,

Files of 10.240.132.215 (Live server):

hornetq-configuration.xml

<configuration xmlns="urn:hornetq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
   <clustered>true</clustered>
   <log-delegate-factory-class-name>org.hornetq.integration.logging.Log4jLogDelegateFactory</log-delegate-factory-class-name>
   <bindings-directory>${jboss.server.data.dir}/hornetq/bindings</bindings-directory>
   <journal-directory>${jboss.server.data.dir}/hornetq/journal</journal-directory>
   <large-messages-directory>${jboss.server.data.dir}/hornetq/largemessages</large-messages-directory>
   <paging-directory>${jboss.server.data.dir}/hornetq/paging</paging-directory>
   <backup-connector-ref connector-name="backup-connector"/>
   <shared-store>true</shared-store>
  <!-- <FailoverOnInitialConnection>true</FailoverOnInitialConnection>-->
   <connectors>
      <connector name="netty">
         <factory-class>org.hornetq.integration.transports.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:10.240.132.215}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </connector>
      <connector name="in-vm">
         <factory-class>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</factory-class>
      </connector>
      <connector name="backup-connector">
        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
        <param key="host" value="10.240.132.216"/>
        <param key="port" value="5445"/>
      </connector>
   </connectors>
    <acceptors>
      <acceptor name="netty">
         <factory-class>org.hornetq.integration.transports.netty.NettyAcceptorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:10.240.132.215}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </acceptor>
      <acceptor name="in-vm">
        <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>
        <param key="server-id" value="0"/>
      </acceptor>
   </acceptors>
   <broadcast-groups>
      <broadcast-group name="bg-group1">
        <local-bind-address>10.240.132.215</local-bind-address>
         <group-address>224.0.0.1</group-address>
         <group-port>9876</group-port>
         <broadcast-period>5000</broadcast-period>
         <connector-ref connector-name="netty"
          backup-connector-name="backup-connector"/>
      </broadcast-group>
   </broadcast-groups>
   <discovery-groups>
      <discovery-group name="dg-group1">
         <group-address>224.0.0.1</group-address>
         <group-port>9876</group-port>
         <refresh-timeout>10000</refresh-timeout>
      </discovery-group>
   </discovery-groups>
   <cluster-connections>
      <cluster-connection name="my-cluster">
         <address>jms</address>
              <discovery-group-ref discovery-group-name="dg-group1"/>
      </cluster-connection>
   </cluster-connections>

   <security-settings>
      <security-setting match="#">
         <permission type="createTempQueue" roles="guest"/>
         <permission type="deleteTempQueue" roles="guest"/>
         <permission type="consume" roles="guest"/>
         <permission type="send" roles="guest"/>
      </security-setting>
   </security-settings>
   <address-settings>
      <!--default for catch all-->
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>-1</max-size-bytes>
         <page-size-bytes>10485760</page-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
      </address-setting>
   </address-settings>
</configuration>

ra.xml

<?xml version="1.0" encoding="UTF-8"?>
<!-- $Id: ra.xml 76819 2008-08-08 11:04:20Z jesper.pedersen $ -->
<connector xmlns="http://java.sun.com/xml/ns/j2ee"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee
           http://java.sun.com/xml/ns/j2ee/connector_1_5.xsd"
           version="1.5">
   <description>HornetQ 2.0 Resource Adapter</description>
   <display-name>HornetQ 2.0 Resource Adapter</display-name>
   <vendor-name>Red Hat Middleware LLC</vendor-name>
   <eis-type>JMS 1.1 Server</eis-type>
   <resourceadapter-version>1.0</resourceadapter-version>
   <license>
      <description>
Copyright 2009 Red Hat, Inc.
 Red Hat licenses this file to you under the Apache License, version
 2.0 (the "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at
   http://www.apache.org/licenses/LICENSE-2.0
 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
 implied.  See the License for the specific language governing
 permissions and limitations under the License.  
      </description>
      <license-required>true</license-required>
   </license>
   <resourceadapter>
      <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>
      <config-property>
         <description>The transport type</description>
         <config-property-name>ConnectorClassName</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</config-property-value>
      </config-property>
      <config-property>
         <description>The transport configuration. These values must be in the form of key=val;key=val;</description>
         <config-property-name>ConnectionParameters</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value>server-id=0</config-property-value>
      </config-property>
      <config-property>
        <description>Use XA?</description>
        <config-property-name>UseXA</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value>true</config-property-value>
      </config-property>
      <!--
      <config-property>
        <description>The method to use for locatingthe transactionmanager</description>
        <config-property-name>TransactionManagerLocatorMethod</config-property-name>
        <config-property-type>java.lang.String</config-property-type>
        <config-property-value>getTm</config-property-value>
      </config-property>
      <config-property>
        <description>Use A local Transaction instead of XA?</description>
        <config-property-name>UseLocalTx</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value>false</config-property-value>
      </config-property>
      <config-property>
        <description>The user name used to login to the JMS server</description>
        <config-property-name>UserName</config-property-name>
        <config-property-type>java.lang.String</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The password used to login to the JMS server</description>
        <config-property-name>Password</config-property-name>
        <config-property-type>java.lang.String</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
         <description>The Backup transport type</description>
         <config-property-name>BackUpTransportType</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value></config-property-value>
      </config-property>
      <config-property>
         <description>The Backup transport configuration. These values must be in the form of key=val;key=val;</description>
         <config-property-name>TransportConfiguration</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The discovery group address</description>
        <config-property-name>DiscoveryGroupAddress</config-property-name>
        <config-property-type>java.lang.String</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The discovery group port</description>
        <config-property-name>DiscoveryGroupPort</config-property-name>
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The discovery refresh timeout</description>
        <config-property-name>DiscoveryRefreshTimeout</config-property-name>
        <config-property-type>java.lang.Long</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The discovery initial wait timeout</description>
        <config-property-name>DiscoveryInitialWaitTimeout</config-property-name>
        <config-property-type>java.lang.Long</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The load balancing policy class name</description>
        <config-property-name>LoadBalancingPolicyClassName</config-property-name>
        <config-property-type>java.lang.String</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The client failure check period</description>
        <config-property-name>ClientFailureCheckPeriod</config-property-name>
        <config-property-type>java.lang.Long</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The connection TTL</description>
        <config-property-name>ConnectionTTL</config-property-name>
        <config-property-type>java.lang.Long</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The call timeout</description>
        <config-property-name>CallTimeout</config-property-name>
        <config-property-type>java.lang.Long</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The dups ok batch size</description>
        <config-property-name>DupsOKBatchSize</config-property-name>
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The transaction batch size</description>
        <config-property-name>TransactionBatchSize</config-property-name>
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The consumer window size</description>
        <config-property-name>ConsumerWindowSize</config-property-name>
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The consumer max rate</description>
        <config-property-name>ConsumerMaxRate</config-property-name>
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The confirmation window size</description>
        <config-property-name>ConfirmationWindowSize</config-property-name>
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The producer max rate</description>
        <config-property-name>ProducerMaxRate</config-property-name>
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The min large message size</description>
        <config-property-name>MinLargeMessageSize</config-property-name>
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The block on acknowledge</description>
        <config-property-name>BlockOnAcknowledge</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The block on non durable send</description>
        <config-property-name>BlockOnNonDurableSend</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The block on durable send</description>
        <config-property-name>BlockOnDurableSend</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The auto group</description>
        <config-property-name>AutoGroup</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The max connections</description>        
        <config-property-type>java.lang.Integer</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The pre acknowledge</description>
        <config-property-name>PreAcknowledge</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The retry interval</description>
        <config-property-name>RetryInterval</config-property-name>
        <config-property-type>java.lang.Long</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>The retry interval multiplier</description>
        <config-property-name>RetryIntervalMultiplier</config-property-name>
        <config-property-type>java.lang.Double</config-property-type>
        <config-property-value></config-property-value>
      </config-property>
      <config-property>
        <description>Should clean server shutdown trigger failover?</description>
        <config-property-name>FailoverOnServerShutdown</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value>true</config-pruperty-value>
      </config-property>
      <config-property>
        <description>The client id</description>
        <config-property-name>ClientID</config-property-name>
        <config-property-type>java.lang.String</config-property-type>
        <config-property-value></config-property-value>
      </config-property>-->
      <outbound-resourceadapter>
         <connection-definition>
            <managedconnectionfactory-class>org.hornetq.ra.HornetQRAManagedConnectionFactory</managedconnectionfactory-class>
            <config-property>
               <description>The default session type</description>
               <config-property-name>SessionDefaultType</config-property-name>
               <config-property-type>java.lang.String</config-property-type>
               <config-property-value>javax.jms.Queue</config-property-value>
            </config-property>
            <config-property>
               <description>Try to obtain a lock within specified number of seconds; less than or equal to 0 disable this functionality</description>
               <config-property-name>UseTryLock</config-property-name>
               <config-property-type>java.lang.Integer</config-property-type>
               <config-property-value>0</config-property-value>
            </config-property>
            <connectionfactory-interface>org.hornetq.ra.HornetQRAConnectionFactory</connectionfactory-interface>
            <connectionfactory-impl-class>org.hornetq.ra.HornetQRAConnectionFactoryImpl</connectionfactory-impl-class>
            <connection-interface>javax.jms.Session</connection-interface>
            <connection-impl-class>org.hornetq.ra.HornetQRASession</connection-impl-class>
         </connection-definition>
         <transaction-support>XATransaction</transaction-support>
         <authentication-mechanism>
            <authentication-mechanism-type>BasicPassword</authentication-mechanism-type>
            <credential-interface>javax.resource.spi.security.PasswordCredential</credential-interface>
         </authentication-mechanism>
         <reauthentication-support>false</reauthentication-support>
      </outbound-resourceadapter>
      <inbound-resourceadapter>
         <messageadapter>
            <messagelistener>
               <messagelistener-type>javax.jms.MessageListener</messagelistener-type>
               <activationspec>
                  <activationspec-class>org.hornetq.ra.inflow.HornetQActivationSpec</activationspec-class>
                  <required-config-property>
                      <config-property-name>destination</config-property-name>
                  </required-config-property>
               </activationspec>
            </messagelistener>
         </messageadapter>
      </inbound-resourceadapter>
   </resourceadapter>
</connector>

Files of 10.240.132.216 (Backup server):

hornetq-configuration.xml

<configuration xmlns="urn:hornetq"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
   <clustered>true</clustered>
   <log-delegate-factory-class-name>org.hornetq.integration.logging.Log4jLogDelegateFactory</log-delegate-factory-class-name>
   <bindings-directory>${jboss.server.data.dir}/hornetq/bindings</bindings-directory>
   <journal-directory>${jboss.server.data.dir}/hornetq/journal</journal-directory>
   <large-messages-directory>${jboss.server.data.dir}/hornetq/largemessages</large-messages-directory>
   <paging-directory>${jboss.server.data.dir}/hornetq/paging</paging-directory>
   <backup>true</backup>
   <shared-store>true</shared-store>
  <!-- <FailoverOnInitialConnection>true</FailoverOnInitialConnection>-->
   <connectors>
      <connector name="netty">
         <factory-class>org.hornetq.integration.transports.netty.NettyConnectorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:10.240.132.216}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </connector>
      <connector name="in-vm">
         <factory-class>org.hornetq.core.remoting.impl.invm.InVMConnectorFactory</factory-class>
      </connector>
   </connectors>
   <acceptors>
      <acceptor name="netty">
         <factory-class>org.hornetq.integration.transports.netty.NettyAcceptorFactory</factory-class>
         <param key="host"  value="${hornetq.remoting.netty.host:10.240.132.216}"/>
         <param key="port"  value="${hornetq.remoting.netty.port:5445}"/>
      </acceptor>
      <acceptor name="in-vm">
        <factory-class>org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory</factory-class>
        <param key="server-id" value="0"/>
      </acceptor>
<!--      <acceptor name="backup-acceptor">
        <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
        <param key="host" value="10.240.132.216"/>
        <param key="port" value="5445"/>
      </acceptor> -->
   </acceptors>
   <broadcast-groups>
      <broadcast-group name="bg-group1">
        <local-bind-address>10.240.132.216</local-bind-address>
         <group-address>224.0.0.1</group-address>
         <group-port>9876</group-port>
         <broadcast-period>5000</broadcast-period>
         <connector-ref connector-name="netty"/>
      </broadcast-group>
   </broadcast-groups>
   <discovery-groups>
      <discovery-group name="dg-group1">
         <group-address>224.0.0.1</group-address>
         <group-port>9876</group-port>
         <refresh-timeout>10000</refresh-timeout>
      </discovery-group>
   </discovery-groups>
   <cluster-connections>
      <cluster-connection name="my-cluster">
         <address>jms</address>
              <discovery-group-ref discovery-group-name="dg-group1"/>
      </cluster-connection>
   </cluster-connections>

   <security-settings>
      <security-setting match="#">
         <permission type="createTempQueue" roles="guest"/>
         <permission type="deleteTempQueue" roles="guest"/>
         <permission type="consume" roles="guest"/>
         <permission type="send" roles="guest"/>
      </security-setting>
   </security-settings>
   <address-settings>
      <!--default for catch all-->
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>-1</max-size-bytes>
         <page-size-bytes>10485760</page-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
      </address-setting>
   </address-settings>

   <queues>
        <queue name="jms.queue.indexRequestQueue">
            <address>jms.queue.indexRequestQueue</address>
            <durable>true</durable>
        </queue>
   </queues>
</configuration>

ra.xml

Identical to ra.xml of 10.240.132.215

Extract of ra.xml from 10.240.132.213 (Client)

<resourceadapter>
      <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>
      <config-property>
         <description>The transport type</description>
         <config-property-name>ConnectorClassName</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value>org.hornetq.integration.transports.netty.NettyConnectorFactory</config-property-value>
      </config-property>
      <config-property>
         <description>The transport configuration. These values must be in the form of key=val;key=val;</description>
         <config-property-name>ConnectionParameters</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value>host=10.240.132.215;port=5445</config-property-value>
      </config-property>
      <config-property>
        <description>Use XA?</description>
        <config-property-name>UseXA</config-property-name>
        <config-property-type>java.lang.Boolean</config-property-type>
        <config-property-value>false</config-property-value>
      </config-property>

jms-ds.xml

<connection-factories>
   <!--
    JMS Stuff
   -->
   <mbean code="org.jboss.jms.jndi.JMSProviderLoader" name="hornetq:service=JMSProviderLoader,name=JMSProvider">
      <attribute name="ProviderName">DefaultJMSProvider</attribute>
      <attribute name="ProviderAdapterClass">org.jboss.jms.jndi.JNDIProviderAdapter</attribute>
      <attribute name="FactoryRef">java:/XAConnectionFactory</attribute>
      <attribute name="QueueFactoryRef">java:/XAConnectionFactory</attribute>
      <attribute name="TopicFactoryRef">java:/XAConnectionFactory</attribute>
   </mbean>
   <!--
    JMS XA Resource adapter, use this to get transacted JMS in beans
   -->
   <no-tx-connection-factory>
      <jndi-name>JmsNonXA</jndi-name>
      <rar-name>hornetq-ra.rar</rar-name>
      <connection-definition>org.hornetq.ra.HornetQRAConnectionFactory</connection-definition>
      <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Queue</config-property>
      <config-property name="JmsProviderAdapterJNDI" type="java.lang.String">java:/DefauiltJMSProvider</config-property>
      <max-pool-size>20</max-pool-size>
   </no-tx-connection-factory>
</connection-factories>

When Live server is down, the client can't basically find the backup server and I get the following exceptions in the client's log,

2010-08-05 02:46:36,363 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371]) javax.jms.JMSException: Timed out waiting for response when sending packet 49
2010-08-05 02:46:36,363 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.hornetq.core.remoting.impl.ChannelImpl.sendBlocking(ChannelImpl.java:270)
2010-08-05 02:46:36,363 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.hornetq.core.client.impl.ClientSessionImpl.bindingQuery(ClientSessionImpl.java:363)
2010-08-05 02:46:36,363 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.hornetq.core.client.impl.DelegatingSession.bindingQuery(DelegatingSession.java:122)
2010-08-05 02:46:36,363 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.hornetq.jms.client.HornetQSession.createBrowser(HornetQSession.java:680)
2010-08-05 02:46:36,363 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.hornetq.jms.client.HornetQSession.createBrowser(HornetQSession.java:643)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.hornetq.ra.HornetQRASession.createBrowser(HornetQRASession.java:801)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at com.gettyimages.mq.qb.MessageLocator.findByDescribedObjectId(MessageLocator.java:19)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at com.getty.mms.mq.MQManagerBean.submitIndexRequest(MQManagerBean.java:41)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at sun.reflect.GeneratedMethodAccessor303.invoke(Unknown Source)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at java.lang.reflect.Method.invoke(Method.java:597)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:122)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.EJBContainerInvocationWrapper.invokeNext(EJBContainerInvocationWrapper.java:69)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.invoke(InterceptorSequencer.java:73)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.interceptors.aop.InterceptorSequencer.aroundInvoke(InterceptorSequencer.java:59)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at sun.reflect.GeneratedMethodAccessor302.invoke(Unknown Source)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at java.lang.reflect.Method.invoke(Method.java:597)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.advice.PerJoinpointAdvice.invoke(PerJoinpointAdvice.java:174)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.fillMethod(InvocationContextInterceptor.java:72)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_fillMethod_747757891.invoke(InvocationContextInterceptor_z_fillMethod_747757891.java)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor.setup(InvocationContextInterceptor.java:88)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.advice.org.jboss.ejb3.interceptors.aop.InvocationContextInterceptor_z_setup_747757891.invoke(InvocationContextInterceptor_z_setup_747757891.java)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:62)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,364 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:56)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.stateless.StatelessInstanceInterceptor.invoke(StatelessInstanceInterceptor.java:68)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:190)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.tx.NullInterceptor.invoke(NullInterceptor.java:42)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.security.Ejb3AuthenticationInterceptorv2.invoke(Ejb3AuthenticationInterceptorv2.java:186)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:41)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.BlockContainerShutdownInterceptor.invoke(BlockContainerShutdownInterceptor.java:67)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aspects.currentinvocation.CurrentInvocationInterceptor.invoke(CurrentInvocationInterceptor.java:67)
2010-08-05 02:46:36,365 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.stateless.StatelessContainer.dynamicInvoke(StatelessContainer.java:421)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.ejb3.session.InvokableContextClassProxyHack._dynamicInvoke(InvokableContextClassProxyHack.java:53)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:91)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:891)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.remoting.transport.socket.ServerThread.completeInvocation(ServerThread.java:744)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:697)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:524)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:232)
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371]) Caused by: HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 49]
2010-08-05 02:46:36,366 ERROR [STDERR] (WorkerThread#0[10.240.132.199:43371])   ... 61 more
2010-08-05 02:46:36,668 WARN  [org.hornetq.core.cluster.impl.DiscoveryGroupImpl] (hornetq-discovery-group-thread-dg-group2) There seem to be more than one broadcasters on the network broadcasting the same node id

I am not sure where the problem lies here.

I would like to specify one thing though, when using a standalone cliet to access the live-backup pair, I used

props.put(Context.PROVIDER_URL, "10.240.132.215:1099,10.240.132.216:1099");

In this scenario it managed to lookup the back server.

Could anyone please tell me how to solve it?

TIA,

Joydeep

1. Re: Problem with Live-backup

joydeep.sarkar Aug 6, 2010 10:25 AM (in response to joydeep.sarkar)

Hi,

Any thoughts about the same?

Regards,
Joydeep
Actions
2. Re: Problem with Live-backup

clebert.suconic Aug 6, 2010 11:04 AM (in response to joydeep.sarkar)

On this case it would be better you describe your system instead of throwing all your configs... probably nobody answered it before because nobody could figure out... and as you know we have limited resources here..

For instance I don't know how your clients are connecting to the server.

Since you're adding the RA, I assume you have MDBs...

So, how many servers you have?

One for the MDBs

And two for HornetQ? (Live, and backup)

The MDBs are connecting remotely?
Actions
3. Re: Problem with Live-backup

joydeep.sarkar Aug 6, 2010 11:41 AM (in response to clebert.suconic)
Hi Clebert,

I have 6 servers altogether and there are three levels. Each level containing two servers in clustered manner.
The first level is the Session bean (Server1 and 2, which is a HASingleton. This is running from all-with-hornetq instance's deploy-hasinglrton directory using deployer service) looking up for a MDB in the level 2 ( Server3 and 4, HornetQ Live-Backup).
Now, the problem that I am facing is when I am trying to lookup MDB in level 2 from level 1 using Session bean.

Following is the Session bean code snippet,
Stateless @RemoteBinding(jndiBinding = "remote/MQManagerBean") public class MQManagerBean implements MQManager { @Resource(mappedName = "java:/JmsNonXA") private ConnectionFactory connectionFactory; public MQManagerBean() { } public void submitIndexRequest(MMIndexingMQContext mmContext) { long describedObjectId = 0; IndexingQueuePartitionScheme qps = new IndexingQueuePartitionScheme(mmContext); String queueName = qps.getDestinationQueue(); System.out.println("index request called with mmContext = "+ mmContext); Connection conn = null; try { conn = connectionFactory.createConnection(); Session session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE); Queue queue = HornetQJMSClient.createQueue(queueName);

The HornetQJMSClient is being used as queue injection failed.
The MDBs are connecting remotely.

Regards,
Joydeep
Actions
4. Re: Problem with Live-backup

clebert.suconic Aug 6, 2010 12:00 PM (in response to joydeep.sarkar)

You need to specify the backup node at the Inbound connection on the MDBs (or at the ra.xml)

the jms-ds.xml is only used to outbound connections (i.e. MessageProducers or regular consumers when a jms.createSession is called).

Look at the MDB example that connects to a remote queue.
Actions

5. Re: Problem with Live-backup

joydeep.sarkar Aug 9, 2010 4:05 AM (in response to clebert.suconic)

Hi Clebert,

Thanks a lot for the response.

But I am a bit confused about the configuration part

You need to specify the backup node at the Inbound connection on the MDBs (or at the ra.xml)

The MDB is itself deployed in Live and Backup servers.

I guess I was not clear about the setup that I have. Allow me to descrive the setup once again.

Server 1,2 - Run session bean in HASingleton mode using deployer service and lookup for the MDB on Server 3/4.

Server 3,4 - Run MDB and queues are deployed in the same instances. Run in Live-Backup mode. Server3 is live and Server4 is the backup server.

The backup connector is already specified in the live server in hornetq-configuration.xml

<backup-connector-ref connector-name="backup-connector"/>

Do I need to further specify the connections in ra.xml as follows?

<resourceadapter>
      <resourceadapter-class>org.hornetq.ra.HornetQResourceAdapter</resourceadapter-class>
      <config-property>
         <description>The transport type</description>
         <config-property-name>ConnectorClassName</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</config-property-value>
      </config-property>
      <config-property>
         <description>The transport configuration. These values must be in the form of key=val;key=val;</description>
         <config-property-name>ConnectionParameters</config-property-name>
         <config-property-type>java.lang.String</config-property-type>
         <config-property-value>host=10.240.132.215;port=5445</config-property-value>-->

Please help.

Regards,

Joydeep

6. Re: Problem with Live-backup

joydeep.sarkar Aug 10, 2010 10:32 AM (in response to clebert.suconic)
Hello Clebert,

There was a configuration mismatch in live and backup servers.
In the live server the backup connector was specified as
<factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

Whereas, in the backup server the acceptor was defined as,
<factory-class>org.hornetq.integration.transports.netty.NettyAcceptorFactory</factory-class>

Now, when I send a request to the queue, both Live and Backup servers start processing the same.
And when the live server is down, backup server does not process the data at all. This behaviour is due to the JMSNonXA I think, as I have configured it with live server's IP only. Shall I use a multicast IP instead? But why both the servers are processing data at the same time?

Could you please help me on the same?

Regards,
Joydeep

Message was edited by: Joydeep Sarkar
Actions
7. Re: Problem with Live-backup

clebert.suconic Aug 10, 2010 12:10 PM (in response to joydeep.sarkar)

We're currently improving the HA code.

At the current version, you can't have any client connected directly on the backup. Having a client connected on the backup will immediately activate the backup.

All the clients should be connected to the live server, until the live server dies. The clients will then reconnect at the backup and activate it.
Actions

Go to original post