10 Replies Latest reply on Sep 5, 2012 2:02 PM by Hitz Jain

    Configure ESBs to use Clustered Queues

    Carl Lewis Newbie

      We are having difficulty configuring multiple esb applications to use clustered queues. Our environment is as follows:

       

      - Windows Server running AS 5.1 configured with 4 nodes (copied from the All configuration)

      - The server is multi-homed; each node has its own IP

      - ESB 4.7

       

      Node 1 contains an esb application that acts as a gateway; picks up files, emails, etc. and uses a JMSRouter to put the messages on a clustered queue.

      Nodes 2, 3, and 4 contain esb applications that monitor the clustered queue and process the messages. Each of these applications have a JMS Selector defined and only pull the messages they are interested in.

      In addition the cluster has been isolated using -g and -u options.

       

      We've tried two different configurations:

       

      1) Each application is configured to look at the queue on its own Node - jndi-URL="jnp://xxx.xxx.xxx.133", jndi-URL="jnp://xxx.xxx.xxx.134", jndi-URL="jnp://xxx.xxx.xxx.135", jndi-URL="jnp://xxx.xxx.xxx.136"

       

      The problem is that it does not appear the queue is actually clustered. The application on Node 1 puts a message on its local queue but the applications on Nodes 2, 3, 4 never see it in their local queues. The admin console on Node 1 shows 1 message in the queue, the console on all the other nodes shows 0 messages in their respective queues.

       

      2) All applications are configured to look at the queue defined in Node 1 - jndi-URL="jnp://${node1.bind.address}"

       

      This configuration works however, we have problems with this configuration when the -g option is used to name the partition; when Node 1 goes down or is not started first, the other nodes repeatedly throw exceptions and do not recover.

       

      13:46:47,756 INFO  [MessageAwareListener] State reached : false
      13:46:47,756 INFO  [JmsConnectionPool] Received NamingException, refreshing context.
      13:46:49,974 WARN  [MessageAwareListener] Error processing courier, backing off for 32000 milliseconds

       

      When the -g option is not used, DefaultPartition, the other nodes successfully recover and will reconnect if Node 1 comes back online.

       

      Option 1 Configuration details:

       

      Node 1:

       

      queue-service.xml

       

      <mbean code="org.jboss.jms.server.destination.QueueService" name="gateway.destination:service=Queue,name=esb_message_queue" xmbean-dd="xmdesc/Queue-xmbean.xml">
        <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
        <depends>jboss.messaging:service=PostOffice</depends>
        <attribute name="Clustered">true</attribute>
      </mbean>

       

       

       

      deployment.xml

       

      <jbossesb-deployment>
          <depends>jboss.jca:service=DataSourceBinding,name=JBossESBDS</depends>
          <depends>jboss.jca:name='mail-ra.rar',service=RARDeployment</depends>
          <depends>gateway.destination:service=Queue,name=esb_message_queue</depends>
      </jbossesb-deployment>

       

       

       

      jboss-esb.xml

       

      <?xml version = "1.0" encoding = "UTF-8"?>
      <jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd" parameterReloadSecs="5">
         <providers>
           <jms-provider name="GATEWAY-JMS" connection-factory="ConnectionFactory"
                         jndi-URL="jnp://${node1.bind.address}" jndi-context-factory="org.jnp.interfaces.NamingContextFactory"
                         jndi-pkg-prefix="org.jboss.naming:org.jnp.interfaces">

      .

      .

      .

         </providers>

      .

      .

      .

      <services>

      <service
              category="Gateway"
              name="FileService"
              description="File Service Queue" >
              <listeners>
                  <jms-listener name="JMS-ESBFileListener"
                      busidref="fileGatewayChannel"
                  />
              </listeners>
                  <actions mep="OneWay">
                          <action name="putMessageOnQueue">
                              <property name="jndi-context-factory" value="org.jnp.interfaces.NamingContextFactory"/>
                              <property name="jndi-URL" value="${node1.bind.address}"/>
                              <property name="jndi-pkg-prefix" value="org.jboss.naming:org.jnp.interfaces"/>
                              <property name="connection-factory" value="ConnectionFactory"/>
                              <property name="jndiName" value="queue/esb_message_queue"/>
                              <property name="unwrap" value="false"/>
                         </action>               
                  </actions>         
           </service>
          </services>
      </jbossesb>

       

       

      Node 2:

       

      queue-service.xml

       

      <?xml version="1.0" encoding="UTF-8" ?>
      <server>
      <mbean code="org.jboss.jms.server.destination.QueueService" name="report.destination:service=Queue,name=esb_message_queue" xmbean-dd="xmdesc/Queue-xmbean.xml">
        <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>
        <depends>jboss.messaging:service=PostOffice</depends>
        <attribute name="Clustered">true</attribute>
      </mbean>
      </server>

       

       

       

      deployment.xml

       

      <jbossesb-deployment>
          <depends>jboss.jca:service=DataSourceBinding,name=JBossESBDS</depends>
          <depends>jboss.jca:name='mail-ra.rar',service=RARDeployment</depends>
          <depends>report.destination:service=Queue,name=esb_message_queue</depends>
      </jbossesb-deployment>

       

       

       

      jboss-esb.xml

       

      <?xml version = "1.0" encoding = "UTF-8"?>
      <jbossesb xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd" parameterReloadSecs="5">
         <providers>
           <jms-provider name="REPORT-JMS" connection-factory="ConnectionFactory"
                         jndi-URL="jnp://${node2.bind.address}" jndi-context-factory="org.jnp.interfaces.NamingContextFactory"
                         jndi-pkg-prefix="org.jboss.naming:org.jnp.interfaces">

       

                <jms-bus busid="esbMessageQueueChannel">
                    <jms-message-filter
                        dest-type="QUEUE"
                        dest-name="queue/esb_message_queue"
                        selector="FILE_TYPE='REPORT'"
                    />
                </jms-bus>

      .

      .

      .

       

      Node 3 and 4 similar to Node 2.

       

      Any help would be appreciated.

       

      Thanks,

       

      Carl

        • 1. Re: Configure ESBs to use Clustered Queues
          Chad Hahn Newbie

          Try using ClusteredConnectionFactory

          • 2. Re: Configure ESBs to use Clustered Queues
            Kevin Conner Master

            The JBossMessaging clustering requires a database to be shared between all the nodes, how have you configured this?

             

            Kev

            • 3. Re: Configure ESBs to use Clustered Queues
              Ryan Hochstetler Newbie

              We have this working almost exactly as you desire.  You're likely missing one bit of configuration for JBoss Messaging.  This discussion is probably better placed over the JBM forum, but I have an interest in promoting clustered use of the ESB, so I'm going to answer here and hope we don't both get smacked on the hand.

               

              http://docs.jboss.org/jbossmessaging/docs/userguide-1.4.5.GA/html/c_configuration.html

               

              Following these steps will help you get Configuration #1 working, which I feel is the best option for your application if you intend to scale beyond your initial configuration.

              • 4. Re: Configure ESBs to use Clustered Queues
                Carl Lewis Newbie

                We're using SQLServer 2008 and have the same mssql-ds.xml configured for each node. We also have the same mssql-persistence-manager.xml defined in each of the 4 node's messaging directory:

                 

                <?xml version="1.0" encoding="UTF-8"?>

                 

                <!--

                     MS SQL Server persistence deployment descriptor.

                 

                     Tested with SQL Server 2005

                 

                     $Id$

                -->

                 

                <server>

                 

                   <!-- Persistence Manager MBean configuration

                       ======================================== -->

                 

                   <mbean code="org.jboss.messaging.core.jmx.JDBCPersistenceManagerService"

                      name="jboss.messaging:service=PersistenceManager"

                      xmbean-dd="xmdesc/JDBCPersistenceManager-xmbean.xml">

                 

                      <depends>jboss.jca:service=DataSourceBinding,name=DefaultDS</depends>

                 

                      <depends optional-attribute-name="TransactionManager">jboss:service=TransactionManager</depends>

                 

                      <!-- The datasource to use for the persistence manager -->

                 

                      <attribute name="DataSource">java:/DefaultDS</attribute>

                 

                      <!-- If true will attempt to create tables and indexes on every start-up -->

                 

                      <attribute name="CreateTablesOnStartup">true</attribute>

                 

                      <!-- If true then we will automatically detect and reject duplicate messages sent during failover -->

                 

                      <attribute name="DetectDuplicates">true</attribute>

                 

                      <!-- The size of the id cache to use when detecting duplicate messages -->

                 

                      <attribute name="IDCacheSize">500</attribute>

                 

                      <attribute name="SqlProperties"><![CDATA[

                   CREATE_DUAL=CREATE TABLE JBM_DUAL (DUMMY INTEGER, PRIMARY KEY (DUMMY))

                   CREATE_MESSAGE_REFERENCE=CREATE TABLE JBM_MSG_REF (MESSAGE_ID BIGINT, CHANNEL_ID BIGINT, TRANSACTION_ID BIGINT, STATE CHAR(1), ORD BIGINT, PAGE_ORD BIGINT, DELIVERY_COUNT INT, SCHED_DELIVERY BIGINT, PRIMARY KEY(MESSAGE_ID, CHANNEL_ID))

                   CREATE_IDX_MESSAGE_REF_TX=CREATE INDEX JBM_MSG_REF_TX ON JBM_MSG_REF (TRANSACTION_ID, STATE)

                   CREATE_MESSAGE=CREATE TABLE JBM_MSG (MESSAGE_ID BIGINT, RELIABLE CHAR(1), EXPIRATION BIGINT, TIMESTAMP BIGINT, PRIORITY TINYINT, TYPE TINYINT, HEADERS IMAGE, PAYLOAD IMAGE, PRIMARY KEY (MESSAGE_ID))

                   CREATE_TRANSACTION=CREATE TABLE JBM_TX (NODE_ID INTEGER, TRANSACTION_ID BIGINT, BRANCH_QUAL VARBINARY(254), FORMAT_ID INT, GLOBAL_TXID VARBINARY(254), PRIMARY KEY (TRANSACTION_ID))

                   CREATE_COUNTER=CREATE TABLE JBM_COUNTER (NAME VARCHAR(255), NEXT_ID BIGINT, PRIMARY KEY(NAME))

                   CREATE_ID_CACHE=CREATE TABLE JBM_ID_CACHE (NODE_ID INTEGER, CNTR INTEGER, JBM_ID VARCHAR(255), PRIMARY KEY(NODE_ID, CNTR))

                   INSERT_DUAL=INSERT INTO JBM_DUAL VALUES (1)

                   CHECK_DUAL=SELECT 1 FROM JBM_DUAL

                   INSERT_MESSAGE_REF=INSERT INTO JBM_MSG_REF (CHANNEL_ID, MESSAGE_ID, TRANSACTION_ID, STATE, ORD, PAGE_ORD, DELIVERY_COUNT, SCHED_DELIVERY) VALUES (?, ?, ?, ?, ?, ?, ?, ?)

                   DELETE_MESSAGE_REF=DELETE FROM JBM_MSG_REF WHERE MESSAGE_ID=? AND CHANNEL_ID=? AND STATE='C'

                   UPDATE_MESSAGE_REF=UPDATE JBM_MSG_REF SET TRANSACTION_ID=?, STATE='-' WHERE MESSAGE_ID=? AND CHANNEL_ID=? AND STATE='C'

                   UPDATE_PAGE_ORDER=UPDATE JBM_MSG_REF SET PAGE_ORD = ? WHERE MESSAGE_ID=? AND CHANNEL_ID=?

                   COMMIT_MESSAGE_REF1=UPDATE JBM_MSG_REF SET STATE='C', TRANSACTION_ID = NULL WHERE TRANSACTION_ID=? AND STATE='+'

                   COMMIT_MESSAGE_REF2=DELETE FROM JBM_MSG_REF WHERE TRANSACTION_ID=? AND STATE='-'

                   ROLLBACK_MESSAGE_REF1=DELETE FROM JBM_MSG_REF WHERE TRANSACTION_ID=? AND STATE='+'

                   ROLLBACK_MESSAGE_REF2=UPDATE JBM_MSG_REF SET STATE='C', TRANSACTION_ID = NULL WHERE TRANSACTION_ID=? AND STATE='-'

                   LOAD_PAGED_REFS=SELECT MESSAGE_ID, DELIVERY_COUNT, PAGE_ORD, SCHED_DELIVERY FROM JBM_MSG_REF WHERE CHANNEL_ID = ? AND PAGE_ORD BETWEEN ? AND ? ORDER BY PAGE_ORD

                   LOAD_UNPAGED_REFS=SELECT MESSAGE_ID, DELIVERY_COUNT, SCHED_DELIVERY FROM JBM_MSG_REF WHERE STATE = 'C' AND CHANNEL_ID = ? AND PAGE_ORD IS NULL ORDER BY ORD

                   LOAD_REFS=SELECT MESSAGE_ID, DELIVERY_COUNT, SCHED_DELIVERY FROM JBM_MSG_REF WHERE STATE = 'C' AND CHANNEL_ID = ? ORDER BY ORD

                   UPDATE_REFS_NOT_PAGED=UPDATE JBM_MSG_REF SET PAGE_ORD = NULL WHERE PAGE_ORD BETWEEN ? AND ? AND CHANNEL_ID=?

                   SELECT_MIN_MAX_PAGE_ORD=SELECT MIN(PAGE_ORD), MAX(PAGE_ORD) FROM JBM_MSG_REF WHERE CHANNEL_ID = ?

                   UPDATE_DELIVERY_COUNT=UPDATE JBM_MSG_REF SET DELIVERY_COUNT = ? WHERE MESSAGE_ID = ? AND CHANNEL_ID = ?

                   UPDATE_CHANNEL_ID=UPDATE JBM_MSG_REF SET CHANNEL_ID = ? WHERE CHANNEL_ID = ?

                   MOVE_REFERENCE=UPDATE JBM_MSG_REF SET CHANNEL_ID = ? WHERE MESSAGE_ID = ? AND CHANNEL_ID = ?

                   LOAD_MESSAGES=SELECT MESSAGE_ID, RELIABLE, EXPIRATION, TIMESTAMP, PRIORITY, HEADERS, PAYLOAD, TYPE FROM JBM_MSG

                   INSERT_MESSAGE=INSERT INTO JBM_MSG (MESSAGE_ID, RELIABLE, EXPIRATION, TIMESTAMP, PRIORITY, TYPE, HEADERS, PAYLOAD) VALUES (?, ?, ?, ?, ?, ?, ?, ?)

                   INSERT_MESSAGE_CONDITIONAL=INSERT INTO JBM_MSG (MESSAGE_ID, RELIABLE, EXPIRATION, TIMESTAMP, PRIORITY, TYPE) SELECT ?, ?, ?, ?, ?, ? FROM JBM_DUAL WHERE NOT EXISTS (SELECT MESSAGE_ID FROM JBM_MSG WHERE MESSAGE_ID = ?)

                   UPDATE_MESSAGE_4CONDITIONAL=UPDATE JBM_MSG SET HEADERS=?, PAYLOAD=? WHERE MESSAGE_ID=?

                   INSERT_MESSAGE_CONDITIONAL_FULL=INSERT INTO JBM_MSG (MESSAGE_ID, RELIABLE, EXPIRATION, TIMESTAMP, PRIORITY, TYPE, HEADERS, PAYLOAD) SELECT ?, ?, ?, ?, ?, ?, ?, ? FROM JBM_DUAL WHERE NOT EXISTS (SELECT MESSAGE_ID FROM JBM_MSG WHERE MESSAGE_ID = ?)

                   MESSAGE_ID_COLUMN=MESSAGE_ID

                   DELETE_MESSAGE=DELETE FROM JBM_MSG WHERE MESSAGE_ID = ? AND NOT EXISTS (SELECT JBM_MSG_REF.MESSAGE_ID FROM JBM_MSG_REF WHERE JBM_MSG_REF.MESSAGE_ID = ?)

                   INSERT_TRANSACTION=INSERT INTO JBM_TX (NODE_ID, TRANSACTION_ID, BRANCH_QUAL, FORMAT_ID, GLOBAL_TXID) VALUES(?, ?, ?, ?, ?)

                   DELETE_TRANSACTION=DELETE FROM JBM_TX WHERE NODE_ID = ? AND TRANSACTION_ID = ?

                   SELECT_PREPARED_TRANSACTIONS=SELECT TRANSACTION_ID, BRANCH_QUAL, FORMAT_ID, GLOBAL_TXID FROM JBM_TX WHERE NODE_ID = ?

                   SELECT_MESSAGE_ID_FOR_REF=SELECT MESSAGE_ID, CHANNEL_ID FROM JBM_MSG_REF WHERE TRANSACTION_ID = ? AND STATE = '+' ORDER BY ORD

                   SELECT_MESSAGE_ID_FOR_ACK=SELECT MESSAGE_ID, CHANNEL_ID FROM JBM_MSG_REF WHERE TRANSACTION_ID = ? AND STATE = '-' ORDER BY ORD

                   UPDATE_COUNTER=UPDATE JBM_COUNTER SET NEXT_ID = ? WHERE NAME=?

                   SELECT_COUNTER=SELECT NEXT_ID FROM JBM_COUNTER WITH (HOLDLOCK, ROWLOCK) WHERE NAME=?

                   INSERT_COUNTER=INSERT INTO JBM_COUNTER (NAME, NEXT_ID) VALUES (?, ?)

                   SELECT_ALL_CHANNELS=SELECT DISTINCT(CHANNEL_ID) FROM JBM_MSG_REF

                   UPDATE_TX=UPDATE JBM_TX SET NODE_ID=? WHERE NODE_ID=?

                   UPDATE_ID_IN_CACHE=UPDATE JBM_ID_CACHE SET JBM_ID = ? WHERE NODE_ID = ? AND CNTR = ?

                   INSERT_ID_IN_CACHE=INSERT INTO JBM_ID_CACHE (NODE_ID, CNTR, JBM_ID) VALUES (?, ?, ?)

                   LOAD_ID_CACHE=SELECT CNTR, JBM_ID FROM JBM_ID_CACHE WHERE NODE_ID = ?

                      ]]></attribute>

                 

                      <!-- The maximum number of parameters to include in a prepared statement -->

                 

                      <attribute name="MaxParams">500</attribute>

                 

                   </mbean>

                 

                   <!-- Messaging Post Office MBean configuration

                        ========================================= -->

                 

                   <mbean code="org.jboss.messaging.core.jmx.MessagingPostOfficeService"

                      name="jboss.messaging:service=PostOffice"

                      xmbean-dd="xmdesc/MessagingPostOffice-xmbean.xml">

                 

                      <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>

                 

                      <depends>jboss.jca:service=DataSourceBinding,name=DefaultDS</depends>

                 

                      <depends optional-attribute-name="TransactionManager">jboss:service=TransactionManager</depends>

                 

                      <!-- The name of the post office -->

                 

                      <attribute name="PostOfficeName">JMS post office</attribute>

                 

                      <!-- The datasource used by the post office to access it's binding information -->

                 

                      <attribute name="DataSource">java:/DefaultDS</attribute>

                 

                      <!-- If true will attempt to create tables and indexes on every start-up -->

                 

                      <attribute name="CreateTablesOnStartup">true</attribute>

                 

                      <attribute name="SqlProperties"><![CDATA[

                CREATE_POSTOFFICE_TABLE=CREATE TABLE JBM_POSTOFFICE (POSTOFFICE_NAME VARCHAR(255), NODE_ID SMALLINT, QUEUE_NAME VARCHAR(255), COND VARCHAR(1023), SELECTOR VARCHAR(1023), CHANNEL_ID INTEGER, CLSTERED CHAR(1), ALL_NODES CHAR(1), PRIMARY KEY(POSTOFFICE_NAME, NODE_ID, QUEUE_NAME))

                INSERT_BINDING=INSERT INTO JBM_POSTOFFICE (POSTOFFICE_NAME, NODE_ID, QUEUE_NAME, COND, SELECTOR, CHANNEL_ID, CLSTERED, ALL_NODES) VALUES (?, ?, ?, ?, ?, ?, ?, ?)

                DELETE_BINDING=DELETE FROM JBM_POSTOFFICE WHERE POSTOFFICE_NAME=? AND NODE_ID=? AND QUEUE_NAME=?

                LOAD_BINDINGS=SELECT QUEUE_NAME, COND, SELECTOR, CHANNEL_ID, CLSTERED, ALL_NODES FROM JBM_POSTOFFICE WHERE POSTOFFICE_NAME=? AND NODE_ID=?

                      ]]></attribute>

                 

                      <!-- This post office is non clustered. If you want a clustered post office then set to true -->

                 

                      <attribute name="Clustered">true</attribute>

                 

                      <!-- All the remaining properties only have to be specified if the post office is clustered.

                           You can safely comment them out if your post office is non clustered -->

                 

                      <!-- The JGroups group name that the post office will use -->

                 

                      <attribute name="GroupName">${jboss.messaging.groupname:MessagingPostOffice}</attribute>

                 

                      <!-- Max time to wait for state to arrive when the post office joins the cluster -->

                 

                      <attribute name="StateTimeout">30000</attribute>

                 

                      <!-- Max time to wait for a synchronous call to node members using the MessageDispatcher -->

                 

                      <attribute name="CastTimeout">30000</attribute>

                 

                      <!-- Set this to true if you want failover of connections to occur when a node is shut down -->

                 

                      <attribute name="FailoverOnNodeLeave">false</attribute>

                 

                      <depends optional-attribute-name="ChannelFactoryName">jboss.jgroups:service=ChannelFactory</depends>

                      <attribute name="ControlChannelName">jbm-control</attribute>

                      <attribute name="DataChannelName">jbm-data</attribute>

                      <attribute name="ChannelPartitionName">${jboss.partition.name:DefaultPartition}-JMS</attribute>

                   </mbean>

                 

                   <!-- Messaging JMS User Manager MBean config

                        ======================================= -->

                 

                   <mbean code="org.jboss.jms.server.plugin.JDBCJMSUserManagerService"

                      name="jboss.messaging:service=JMSUserManager"

                      xmbean-dd="xmdesc/JMSUserManager-xmbean.xml">

                 

                      <depends>jboss.jca:service=DataSourceBinding,name=DefaultDS</depends>

                 

                      <depends optional-attribute-name="TransactionManager">jboss:service=TransactionManager</depends>

                 

                      <attribute name="DataSource">java:/DefaultDS</attribute>

                 

                      <attribute name="CreateTablesOnStartup">true</attribute>

                 

                      <attribute name="SqlProperties"><![CDATA[

                CREATE_USER_TABLE=CREATE TABLE JBM_USER (USER_ID VARCHAR(32) NOT NULL, PASSWD VARCHAR(32) NOT NULL, CLIENTID VARCHAR(128), PRIMARY KEY(USER_ID))

                CREATE_ROLE_TABLE=CREATE TABLE JBM_ROLE (ROLE_ID VARCHAR(32) NOT NULL, USER_ID VARCHAR(32) NOT NULL, PRIMARY KEY(USER_ID, ROLE_ID))

                SELECT_PRECONF_CLIENTID=SELECT CLIENTID FROM JBM_USER WHERE USER_ID=?

                POPULATE.TABLES.1  = INSERT INTO JBM_USER (USER_ID, PASSWD) VALUES ('guest', 'guest')

                POPULATE.TABLES.2  = INSERT INTO JBM_USER (USER_ID, PASSWD) VALUES ('j2ee', 'j2ee')

                POPULATE.TABLES.3  = INSERT INTO JBM_USER (USER_ID, PASSWD, CLIENTID) VALUES ('john', 'needle', 'DurableSubscriberExample')

                POPULATE.TABLES.4  = INSERT INTO JBM_USER (USER_ID, PASSWD) VALUES ('nobody', 'nobody')

                POPULATE.TABLES.5  = INSERT INTO JBM_USER (USER_ID, PASSWD) VALUES ('dynsub', 'dynsub')

                POPULATE.TABLES.6  = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('guest','guest')

                POPULATE.TABLES.7  = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('j2ee','guest')

                POPULATE.TABLES.8  = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('john','guest')

                POPULATE.TABLES.9  = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('subscriber','john')

                POPULATE.TABLES.10 = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('publisher','john')

                POPULATE.TABLES.11 = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('publisher','dynsub')

                POPULATE.TABLES.12 = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('durpublisher','john')

                POPULATE.TABLES.13 = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('durpublisher','dynsub')

                POPULATE.TABLES.14 = INSERT INTO JBM_ROLE (ROLE_ID, USER_ID) VALUES ('noacc','nobody')

                      ]]></attribute>

                   </mbean>

                 

                </server>

                 

                 

                I checked the JBM_POSTOFFICE table and it contains the following rows:

                 

                jbm_postoffice.JPG

                 

                The file_gateway_queue and email_gateway_queue are only defined in the esb on Node1, the file_gateway_queue is for the JMSRouter and the email_gateway_queue is for the Mail_JCA_Gateway. The esb_message_queue is the queue the JMSRouter writes to and the other nodes read from so this is the queue that needs to be clustered. Looking at the jbm_postoffice table, I realize that the two gateway queues are clustered but probably shouldn't be since only the esb on Node1 uses them. I'll try changing that but it doesn't seem like that would cause a problem the way it is now.

                • 5. Re: Configure ESBs to use Clustered Queues
                  Carl Lewis Newbie

                  I double checked all the config files and I think we have everything covered.

                   

                   

                  PostOffice config defined in deploy\messaging\mssql-persistence-service.xml (Defined the same on all 4 nodes):

                   

                   

                  <!-- Messaging Post Office MBean configuration

                   

                           ========================================= -->

                   

                   

                      <mbean code="org.jboss.messaging.core.jmx.MessagingPostOfficeService"

                         name="jboss.messaging:service=PostOffice"

                         xmbean-dd="xmdesc/MessagingPostOffice-xmbean.xml">


                   

                         <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>


                   

                         <depends>jboss.jca:service=DataSourceBinding,name=DefaultDS</depends>

                        <depends optional-attribute-name="TransactionManager">jboss:service=TransactionManager</depends>

                   

                        <!-- The name of the post office -->

                        <attribute name="PostOfficeName">JMS post office</attribute>

                   

                         <!-- The datasource used by the post office to access it's binding information -->

                        <attribute name="DataSource">java:/DefaultDS</attribute>



                         <!-- If true will attempt to create tables and indexes on every start-up -->

                        <attribute name="CreateTablesOnStartup">true</attribute>

                   

                   

                         <attribute name="SqlProperties"><![CDATA[

                   

                  CREATE_POSTOFFICE_TABLE=CREATE TABLE JBM_POSTOFFICE (POSTOFFICE_NAME VARCHAR(255), NODE_ID SMALLINT, QUEUE_NAME VARCHAR(255), COND VARCHAR(1023), SELECTOR VARCHAR(1023), CHANNEL_ID INTEGER, CLSTERED CHAR(1), ALL_NODES CHAR(1), PRIMARY KEY(POSTOFFICE_NAME, NODE_ID, QUEUE_NAME))

                   

                  INSERT_BINDING=INSERT INTO JBM_POSTOFFICE (POSTOFFICE_NAME, NODE_ID, QUEUE_NAME, COND, SELECTOR, CHANNEL_ID, CLSTERED, ALL_NODES) VALUES (?, ?, ?, ?, ?, ?, ?, ?)

                   

                  DELETE_BINDING=DELETE FROM JBM_POSTOFFICE WHERE POSTOFFICE_NAME=? AND NODE_ID=? AND QUEUE_NAME=?

                   

                  LOAD_BINDINGS=SELECT QUEUE_NAME, COND, SELECTOR, CHANNEL_ID, CLSTERED, ALL_NODES FROM JBM_POSTOFFICE WHERE POSTOFFICE_NAME=? AND NODE_ID=?

                   

                         ]]></attribute>

                   

                        <!-- This post office is non clustered. If you want a clustered post office then set to true -->

                        <attribute name="Clustered">true</attribute>

                   

                        <!-- All the remaining properties only have to be specified if the post office is clustered.

                             You can safely comment them out if your post office is non clustered -->

                   

                   

                         <!-- The JGroups group name that the post office will use -->

                        <attribute name="GroupName">${jboss.messaging.groupname:MessagingPostOffice}</attribute>

                   

                    

                   

                         <!-- Max time to wait for state to arrive when the post office joins the cluster -->

                        <attribute name="StateTimeout">30000</attribute>

                   

                    

                   

                         <!-- Max time to wait for a synchronous call to node members using the MessageDispatcher -->

                        <attribute name="CastTimeout">30000</attribute>

                   

                        

                   

                         <!-- Set this to true if you want failover of connections to occur when a node is shut down -->

                        <attribute name="FailoverOnNodeLeave">false</attribute>

                   

                   

                         <depends optional-attribute-name="ChannelFactoryName">jboss.jgroups:service=ChannelFactory</depends>

                   

                         <attribute name="ControlChannelName">jbm-control</attribute>

                   

                         <attribute name="DataChannelName">jbm-data</attribute>

                   

                         <attribute name="ChannelPartitionName">${jboss.partition.name:DefaultPartition}-JMS</attribute>

                   

                      </mbean>

                   

                   

                  The following destination is defined on all 4 nodes:

                   

                   

                  <mbean code="org.jboss.jms.server.destination.QueueService" name="trax.conduit.destination:service=Queue,name=esb_message_queue" xmbean-dd="xmdesc/Queue-xmbean.xml">


                    <depends optional-attribute-name="ServerPeer">jboss.messaging:service=ServerPeer</depends>


                    <depends>jboss.messaging:service=PostOffice</depends>


                    <attribute name="Clustered">true</attribute>


                  </mbean>

                   

                   

                  Each node is started with the following options:

                   

                   

                  # Application parameters.  Add parameters as needed starting from 1

                  wrapper.app.parameter.1=org.jboss.Main

                  wrapper.app.parameter.2=--configuration=node1

                  wrapper.app.parameter.3=-b xxx.xxx.xxx.133

                  wrapper.app.parameter.4=-Djboss.messaging.ServerPeerID=1

                  wrapper.app.parameter.5=-u 239.255.100.111

                  wrapper.app.parameter.6=-g QAPartition

                  wrapper.app.parameter.7=-Djboss.jgroups.enable_diagnostics=false

                  wrapper.app.parameter.8=-Djboss.messaging.groupname=QAPostOffice

                   


                  # Application parameters.  Add parameters as needed starting from 1

                  wrapper.app.parameter.1=org.jboss.Main

                  wrapper.app.parameter.2=--configuration=node2

                  wrapper.app.parameter.3=-b xxx.xxx.xxx.134

                  wrapper.app.parameter.4=-Djboss.messaging.ServerPeerID=2

                  wrapper.app.parameter.5=-u 239.255.100.111

                  wrapper.app.parameter.6=-g QAPartition

                  wrapper.app.parameter.7=-Djboss.jgroups.enable_diagnostics=false

                  wrapper.app.parameter.8=-Djboss.messaging.groupname=QAPostOffice

                   


                  # Application parameters.  Add parameters as needed starting from 1

                  wrapper.app.parameter.1=org.jboss.Main

                  wrapper.app.parameter.2=--configuration=node3

                  wrapper.app.parameter.3=-b xxx.xxx.xxx.135

                  wrapper.app.parameter.4=-Djboss.messaging.ServerPeerID=3

                  wrapper.app.parameter.5=-u 239.255.100.111

                  wrapper.app.parameter.6=-g QAPartition

                  wrapper.app.parameter.7=-Djboss.jgroups.enable_diagnostics=false

                  wrapper.app.parameter.8=-Djboss.messaging.groupname=QAPostOffice

                   


                  # Application parameters.  Add parameters as needed starting from 1

                  wrapper.app.parameter.1=org.jboss.Main

                  wrapper.app.parameter.2=--configuration=node4

                  wrapper.app.parameter.3=-b xxx.xxx.xxx.136 

                  wrapper.app.parameter.4=-Djboss.messaging.ServerPeerID=4

                  wrapper.app.parameter.5=-u 239.255.100.111

                  wrapper.app.parameter.6=-g QAPartition

                  wrapper.app.parameter.7=-Djboss.jgroups.enable_diagnostics=false

                  wrapper.app.parameter.8=-Djboss.messaging.groupname=QAPostOffice

                   

                   

                  The sucker password has been changed and is the same across all nodes.

                   

                   

                  We're using a database backed persistence manager (details in my previous post).

                   

                   

                  I'm not sure what we're missing, I think everything is configured correctly.

                  • 6. Re: Configure ESBs to use Clustered Queues
                    Ryan Hochstetler Newbie

                    In your first post, you showed that the JMX domain of the clustered queue was different.

                     

                    Node 1: <mbean code="org.jboss.jms.server.destination.QueueService"  name="gateway.destination:service=Queue,name=esb_message_queue"

                    Node 2-4: <mbean code="org.jboss.jms.server.destination.QueueService"  name="report.destination:service=Queue,name=esb_message_queue"

                     

                    In the final post, you listed the domain as trax.conduit.destination.  Is it the same on all four nodes now?  I don't know that this is required, but it seems likely that JBM would use the full ObjectName to identify the queues across JVMs.

                     

                    Also, I'm getting a broken image link for the jpeg you uploaded.  I assume that there's something wrong with the JBoss Community software.

                     

                    The only thing I can find different with your configuration is that we do not set the jboss.messaging.groupname property.  We just accept the default.  However, this shouldn't cause a problem.  Though, there is a strangecomment and bit of (backward) code in MessagingPostOfficeService.java.  You might try unsetting the group name.

                     

                    MessagingPostOfficeService.java from JBoss 5.1.0.GA sources

                    // http://jira.jboss.org/jira/browse/JBMESSAGING-1244
                    // if the channelPartitionName is set, we use it to identify the control and
                    // data channels. Otherwise, we fall back to use groupName
                    String channelName = (channelPartitionName == null)? channelPartitionName : groupName;
                    

                     

                    Yeah....I had to get someone else to read it and confirm that I had not developed selective dyslexia.

                    • 7. Re: Configure ESBs to use Clustered Queues
                      Carl Lewis Newbie

                      Sorry about the inconsistency. Each node has a unique JMX domain but the queue name is the same. My first post kind of implies that nodes 2-4 were all named report.destination, but they each have unique JMX domain names. I will try setting them all the same across the four nodes.

                       

                      The broken image link was a nicely formatted image of the contents of our jbm_postoffice table. Here's the raw delimited data.

                       

                      JMS post office;1;CallbackQueue;queue.CallbackQueue;<null>;9;Y;N
                      JMS post office;2;CallbackQueue;queue.CallbackQueue;<null>;757;Y;N
                      JMS post office;3;CallbackQueue;queue.CallbackQueue;<null>;767;Y;N
                      JMS post office;4;CallbackQueue;queue.CallbackQueue;<null>;107;Y;N
                      JMS post office;1;DataCollectorQueue;queue.DataCollectorQueue;<null>;6;Y;N
                      JMS post office;2;DataCollectorQueue;queue.DataCollectorQueue;<null>;754;Y;N
                      JMS post office;3;DataCollectorQueue;queue.DataCollectorQueue;<null>;764;Y;N
                      JMS post office;4;DataCollectorQueue;queue.DataCollectorQueue;<null>;104;Y;N
                      JMS post office;1;DeadMessageQueue;queue.DeadMessageQueue;<null>;5;Y;N
                      JMS post office;2;DeadMessageQueue;queue.DeadMessageQueue;<null>;753;Y;N
                      JMS post office;3;DeadMessageQueue;queue.DeadMessageQueue;<null>;763;Y;N
                      JMS post office;4;DeadMessageQueue;queue.DeadMessageQueue;<null>;103;Y;N
                      JMS post office;1;DLQ;queue.DLQ;<null>;1;Y;N
                      JMS post office;2;DLQ;queue.DLQ;<null>;751;Y;N
                      JMS post office;3;DLQ;queue.DLQ;<null>;761;Y;N
                      JMS post office;4;DLQ;queue.DLQ;<null>;102;Y;N
                      JMS post office;1;email_gateway_queue;queue.email_gateway_queue;<null>;4;Y;N
                      JMS post office;1;esb_message_queue;queue.esb_message_queue;<null>;0;Y;N
                      JMS post office;2;esb_message_queue;queue.esb_message_queue;<null>;752;Y;N
                      JMS post office;3;esb_message_queue;queue.esb_message_queue;<null>;762;Y;N
                      JMS post office;4;esb_message_queue;queue.esb_message_queue;<null>;100;Y;N
                      JMS post office;1;ExpiryQueue;queue.ExpiryQueue;<null>;3;Y;N
                      JMS post office;2;ExpiryQueue;queue.ExpiryQueue;<null>;750;Y;N
                      JMS post office;3;ExpiryQueue;queue.ExpiryQueue;<null>;760;Y;N
                      JMS post office;4;ExpiryQueue;queue.ExpiryQueue;<null>;101;Y;N
                      JMS post office;1;file_gateway_queue;queue.file_gateway_queue;<null>;2;Y;N
                      JMS post office;1;OperationsCollectorQueue;queue.OperationsCollectorQueue;<null>;7;Y;N
                      JMS post office;2;OperationsCollectorQueue;queue.OperationsCollectorQueue;<null>;755;Y;N
                      JMS post office;3;OperationsCollectorQueue;queue.OperationsCollectorQueue;<null>;765;Y;N
                      JMS post office;4;OperationsCollectorQueue;queue.OperationsCollectorQueue;<null>;105;Y;N
                      JMS post office;1;OperationsResultCollectorQueue;queue.OperationsResultCollectorQueue;<null>;8;Y;N
                      JMS post office;2;OperationsResultCollectorQueue;queue.OperationsResultCollectorQueue;<null>;756;Y;N
                      JMS post office;3;OperationsResultCollectorQueue;queue.OperationsResultCollectorQueue;<null>;766;Y;N
                      JMS post office;4;OperationsResultCollectorQueue;queue.OperationsResultCollectorQueue;<null>;106;Y;N

                       

                       

                      I'll also try unsetting the jboss.messaging.groupname to see if that makes any difference.

                       

                      Thanks,

                       

                      Carl

                      • 8. Re: Configure ESBs to use Clustered Queues
                        Carl Lewis Newbie

                        Still not working. It probably has to do with the fact that Nodes 2-4 have JMS Selectors defined so they only pick up certain types of messages. My assumption was that each node would have its own local copy of every message but that wouldn't make sense since each node would end up with a subset of messages that never get processed. In order to cluster and use selectors I think what we really need is the Server Side Load Balancing and Message Redistribution capabilities in HornetQ.

                         

                        From the HornetQ 2.0 User Manual

                         

                        <cluster-connections>
                            <cluster-connection name="my-cluster">
                                <address>jms</address>
                                <retry-interval>500</retry-interval>
                                <use-duplicate-detection>true</use-duplicate-detection>
                                <forward-when-no-consumers>false</forward-when-no-consumers>
                                <max-hops>1</max-hops>
                                <discovery-group-ref discovery-group-name="my-discovery-group"/>
                            </cluster-connection>
                        </cluster-connections>
                          

                         

                        forward-when-no-consumers. This parameter determines                         whether messages will be distributed round robin between other nodes of the                         cluster irrespective of whether there are matching or                         indeed any consumers on other nodes. If this is set to false then HornetQ will only                         forward messages to other nodes of the cluster if the address to which they                         are being forwarded has queues which have consumers, and if those consumers                         have message filters (selectors) at least one of those selectors must match                         the message.

                         

                        <address-settings>     
                           <address-setting match="jms.#">
                              <redistribution-delay>0</redistribution-delay>
                           </address-setting>
                        </address-settings>

                         

                        Another important part of clustering is message redistribution. Earlier we learned how             server side message load balancing round robins messages across the cluster. If forward-when-no-consumers is false, then messages won't be forwarded to             nodes which don't have matching consumers, this is great and ensures that messages don't             arrive on a queue which has no consumers to consume them, however there is a situation             it doesn't solve: What happens if the consumers on a queue close after the messages have             been sent to the node? If there are no consumers on the queue the message won't get             consumed and we have a starvation situation.

                        This is where message redistribution comes in. With message redistribution HornetQ             can be configured to automatically redistribute messages             from queues which have no consumers back to other nodes in the cluster which do have             matching consumers.

                         

                         

                        So for now, we'll go with a single queue configuration until HornetQ is available in the ESB.

                         

                        Thanks

                         

                        Carl

                        • 9. Re: Configure ESBs to use Clustered Queues
                          Carl Lewis Newbie

                          I just wanted to provide an update on this issue. We've upgraded our environment to ESB 4.9 and HornetQ 2.1.2  to take advantage of the foward-when-no-consumers and message redistribution features of HornetQ. Clustering is now working correctly with our configuration which makes use of jms selectors.

                          • 10. Re: Configure ESBs to use Clustered Queues
                            Hitz Jain Newbie

                            Hi Carl,

                             

                            I know itz been long since you have worked on this, but would you know what exactly you did to get this clustering working.

                            It would be great if you could share the config snippet.

                             

                            -