Urgent! Clustered-queue on 2 nodes failed

[glenwood]

My company would like to update our jms implementation from jbossmq to JBM2,I configured a simple cluster environment on 2 nodes with jboss5.1,and followed the instructions of JBM2 to create all-with-jbm2 configuration on 2 nodes,I created 2 queue consumers on 2 nodes and created one queue producer on one node,after starting the jboss on both servers,one server which have both consumer and producer can receive the messages sent by producer,but another server was not able to get any messages,can't figure out why,maybe the cluster configuration of jboss5 has conflict with messaging cluster configuration?

Following are the configuration files on server1,

jbm-configuration.xml

<configuration xmlns="urn:jboss:messaging"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:jboss:messaging /schema/jbm-configuration.xsd">

true



<factory-class>org.jboss.messaging.integration.transports.netty.NettyConnectorFactory</factory-class>






<factory-class>org.jboss.messaging.integration.transports.netty.NettyAcceptorFactory</factory-class>




<broadcast-groups>
<broadcast-group name="hebei-broadcast-group1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref connector-name="netty-connector1"/>
</broadcast-group>
</broadcast-groups>

<discovery-groups>
<discovery-group name="hebei-discovery-group1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>

<cluster-connections>
<cluster-connection name="hebei-cluster">
jms
<forward-when-no-consumers>true</forward-when-no-consumers>
<discovery-group-ref discovery-group-name="hebei-discovery-group1"/>
</cluster-connection>
</cluster-connections>

<security-settings>
<security-setting match="#">




</security-setting>
</security-settings>

<address-settings>
<!--default for catch all-->
<address-setting match="#">
false
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<distribution-policy-class>org.jboss.messaging.core.server.impl.RoundRobinDistributor</distribution-policy-class>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<redistribution-delay>0</redistribution-delay>
</address-setting>
</address-settings>

--------------------------------------------------
jbm-jboss-bean.xml

<?xml version="1.0" encoding="UTF-8"?>





<!-- JNDI server. Disable this if you don't want JNDI -->




1099
192.168.110.16
1098
192.168.110.16


<!-- MBean server -->




<!-- The core configuration -->


<!-- The security manager -->






<!-- The core server -->
















<!-- The JMS server -->









------------------------------------------------------------------
jbm-jms.xml

<configuration xmlns="urn:jboss:messaging"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:jboss:messaging /schema/jbm-jms.xsd">

<connection-factory name="ConnectionFactory">
<connector-ref connector-name="netty-connector1"/>




</connection-factory>

<connection-factory name="ClusteredConnectionFactory">
<connector-ref connector-name="netty" backup-connector-name="netty-backup"/>






</connection-factory>













------------------------------------------------
jbm-user.xml

<configuration xmlns="urn:jboss:messaging" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:jboss:messaging /schema/jbm-users.xsd">
<!-- the default user. this is used where username is null-->




---------------------------------------------------
Following are the configuration file on server2,

jbm-configuration.xml

<configuration xmlns="urn:jboss:messaging"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:jboss:messaging /schema/jbm-configuration.xsd">

true



<factory-class>org.jboss.messaging.integration.transports.netty.NettyConnectorFactory</factory-class>






<factory-class>org.jboss.messaging.integration.transports.netty.NettyAcceptorFactory</factory-class>




<broadcast-groups>
<broadcast-group name="hebei-broadcast-group1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<broadcast-period>2000</broadcast-period>
<connector-ref connector-name="netty-connector2"/>
</broadcast-group>
</broadcast-groups>

<discovery-groups>
<discovery-group name="hebei-discovery-group1">
<group-address>231.7.7.7</group-address>
<group-port>9876</group-port>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>

<cluster-connections>
<cluster-connection name="hebei-cluster">
jms
<forward-when-no-consumers>true</forward-when-no-consumers>
<discovery-group-ref discovery-group-name="hebei-discovery-group1"/>
</cluster-connection>
</cluster-connections>

<security-settings>
<security-setting match="#">




</security-setting>
</security-settings>

<address-settings>
<!--default for catch all-->
<address-setting match="#">
false
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>-1</max-size-bytes>
<page-size-bytes>10485760</page-size-bytes>
<distribution-policy-class>org.jboss.messaging.core.server.impl.RoundRobinDistributor</distribution-policy-class>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
<redistribution-delay>0</redistribution-delay>
</address-setting>
</address-settings>


-------------------------------------------------------
jbm-jboss-bean.xml


<?xml version="1.0" encoding="UTF-8"?>




<!-- JNDI server. Disable this if you don't want JNDI -->




2099
192.168.110.19
2098
192.168.110.19


<!-- MBean server -->




<!-- The core configuration -->


<!-- The security manager -->




<!-- The core server -->
















<!-- The JMS server -->









-------------------------------------------
jbm-jms.xml

<configuration xmlns="urn:jboss:messaging"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:jboss:messaging /schema/jbm-jms.xsd">

<connection-factory name="ConnectionFactory">
<connector-ref connector-name="netty-connector2"/>




</connection-factory>

<connection-factory name="ClusteredConnectionFactory">
<connector-ref connector-name="netty" backup-connector-name="netty-backup"/>






</connection-factory>













---------------------------------------------------------
jbm-user.xml

<configuration xmlns="urn:jboss:messaging" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:jboss:messaging /schema/jbm-users.xsd">
<!-- the default user. this is used where username is null-->






Could anyone can find what's going on,I really need to make it work ASAP as I spent too much time on this ,my boss push me too hard :(

[glenwood]

not sure why xml file didn't get display correctly,I try one more time,hopefully this time works

Following is the configuration on server1,

jbm-configuration.xml

<configuration xmlns="urn:jboss:messaging"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:jboss:messaging /schema/jbm-configuration.xsd">

 <clustered>true</clustered>

 <connectors>
 <connector name="netty-connector1">
 <factory-class>org.jboss.messaging.integration.transports.netty.NettyConnectorFactory</factory-class>
 <param key="jbm.remoting.netty.port" value="5445" type="Integer"/>
 </connector>
 </connectors>

 <acceptors>
 <acceptor name="netty-acceptor">
 <factory-class>org.jboss.messaging.integration.transports.netty.NettyAcceptorFactory</factory-class>
 <param key="jbm.remoting.netty.port" value="5445" type="Integer"/>
 </acceptor>
 </acceptors>

 <broadcast-groups>
 <broadcast-group name="hebei-broadcast-group1">
 <group-address>231.7.7.7</group-address>
 <group-port>9876</group-port>
 <broadcast-period>2000</broadcast-period>
 <connector-ref connector-name="netty-connector1"/>
 </broadcast-group>
 </broadcast-groups>

 <discovery-groups>
 <discovery-group name="hebei-discovery-group1">
 <group-address>231.7.7.7</group-address>
 <group-port>9876</group-port>
 <refresh-timeout>10000</refresh-timeout>
 </discovery-group>
 </discovery-groups>

 <cluster-connections>
 <cluster-connection name="hebei-cluster">
 <address>jms</address>
 <forward-when-no-consumers>true</forward-when-no-consumers>
 <discovery-group-ref discovery-group-name="hebei-discovery-group1"/>
 </cluster-connection>
 </cluster-connections>

 <security-settings>
 <security-setting match="#">
 <permission type="createTempQueue" roles="guest,publisher"/>
 <permission type="deleteTempQueue" roles="guest,publisher"/>
 <permission type="consume" roles="guest,publisher"/>
 <permission type="send" roles="guest,publisher"/>
 </security-setting>
 </security-settings>

 <address-settings>
 <!--default for catch all-->
 <address-setting match="#">
 <clustered>false</clustered>
 <dead-letter-address>jms.queue.DLQ</dead-letter-address>
 <expiry-address>jms.queue.ExpiryQueue</expiry-address>
 <redelivery-delay>0</redelivery-delay>
 <max-size-bytes>-1</max-size-bytes>
 <page-size-bytes>10485760</page-size-bytes>
 <distribution-policy-class>org.jboss.messaging.core.server.impl.RoundRobinDistributor</distribution-policy-class>
 <message-counter-history-day-limit>10</message-counter-history-day-limit>
 <redistribution-delay>0</redistribution-delay>
 </address-setting>
 </address-settings>
</configuration>

jbm-jboss-beans.xml

<?xml version="1.0" encoding="UTF-8"?>

<deployment xmlns="urn:jboss:bean-deployer:2.0">

<bean name="Naming" class="org.jnp.server.NamingBeanImpl"/>

 <!-- JNDI server. Disable this if you don't want JNDI -->
 <bean name="JNDIServer" class="org.jnp.server.Main">
 <property name="namingInfo">
 <inject bean="Naming"/>
 </property>
 <property name="port">1099</property>
 <property name="bindAddress">192.168.110.16</property>
 <property name="rmiPort">1098</property>
 <property name="rmiBindAddress">192.168.110.16</property>
 </bean>

 <!-- MBean server -->
 <bean name="MBeanServer" class="javax.management.MBeanServer">
 <constructor factoryClass="java.lang.management.ManagementFactory"
 factoryMethod="getPlatformMBeanServer"/>
 </bean>

 <!-- The core configuration -->
 <bean name="Configuration" class="org.jboss.messaging.core.config.impl.FileConfiguration"/>

 <!-- The security manager -->
 <bean name="JBMSecurityManager" class="org.jboss.messaging.core.security.impl.JBMSecurityManagerImpl">
 <start ignored="true"/>
 <stop ignored="true"/>
 </bean>


 <!-- The core server -->
 <bean name="MessagingServer" class="org.jboss.messaging.core.server.impl.MessagingServerImpl">
 <constructor>
 <parameter>
 <inject bean="Configuration"/>
 </parameter>
 <parameter>
 <inject bean="MBeanServer"/>
 </parameter>
 <parameter>
 <inject bean="JBMSecurityManager"/>
 </parameter>
 </constructor>
 <start ignored="true"/>
 <stop ignored="true"/>
 </bean>

 <!-- The JMS server -->
 <bean name="JMSServerManager" class="org.jboss.messaging.jms.server.impl.JMSServerManagerImpl">
 <constructor>
 <parameter>
 <inject bean="MessagingServer"/>
 </parameter>
 </constructor>
 </bean>

</deployment>

jbm-jms.xml

<configuration xmlns="urn:jboss:messaging"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:jboss:messaging /schema/jbm-jms.xsd">

 <connection-factory name="ConnectionFactory">
 <connector-ref connector-name="netty-connector1"/>
 <entries>
 <entry name="ConnectionFactory"/>
 <entry name="XAConnectionFactory"/>
 </entries>
 </connection-factory>

 <connection-factory name="ClusteredConnectionFactory">
 <connector-ref connector-name="netty" backup-connector-name="netty-backup"/>
 <entries>
 <entry name="/ClusteredConnectionFactory"/>
 <entry name="/ClusteredXAConnectionFactory"/>
 <entry name="java:/ClusteredConnectionFactory"/>
 <entry name="java:/ClusteredXAConnectionFactory"/>
 </entries>
 </connection-factory>

 <queue name="DLQ">
 <entry name="/queue/DLQ"/>
 </queue>
 <queue name="ExpiryQueue">
 <entry name="/queue/ExpiryQueue"/>
 </queue>
 <queue name="BUYTICKET">
 <entry name="/queue/BUYTICKET"/>
 </queue>


</configuration>

jbm-users.xml

<configuration xmlns="urn:jboss:messaging" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:jboss:messaging /schema/jbm-users.xsd">
 <!-- the default user. this is used where username is null-->
 <defaultuser name="guest" password="guest">
 <role name="guest"/>
 </defaultuser>
</configuration>

The following are the configuration on server2

jbm-configuration.xml

<configuration xmlns="urn:jboss:messaging"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:jboss:messaging /schema/jbm-configuration.xsd">

 <clustered>true</clustered>

 <connectors>
 <connector name="netty-connector2">
 <factory-class>org.jboss.messaging.integration.transports.netty.NettyConnectorFactory</factory-class>
 <param key="jbm.remoting.netty.port" value="5446" type="Integer"/>
 </connector>
 </connectors>

 <acceptors>
 <acceptor name="netty-acceptor">
 <factory-class>org.jboss.messaging.integration.transports.netty.NettyAcceptorFactory</factory-class>
 <param key="jbm.remoting.netty.port" value="5446" type="Integer"/>
 </acceptor>
 </acceptors>

 <broadcast-groups>
 <broadcast-group name="hebei-broadcast-group1">
 <group-address>231.7.7.7</group-address>
 <group-port>9876</group-port>
 <broadcast-period>2000</broadcast-period>
 <connector-ref connector-name="netty-connector2"/>
 </broadcast-group>
 </broadcast-groups>

 <discovery-groups>
 <discovery-group name="hebei-discovery-group1">
 <group-address>231.7.7.7</group-address>
 <group-port>9876</group-port>
 <refresh-timeout>10000</refresh-timeout>
 </discovery-group>
 </discovery-groups>

 <cluster-connections>
 <cluster-connection name="hebei-cluster">
 <address>jms</address>
 <forward-when-no-consumers>true</forward-when-no-consumers>
 <discovery-group-ref discovery-group-name="hebei-discovery-group1"/>
 </cluster-connection>
 </cluster-connections>

 <security-settings>
 <security-setting match="#">
 <permission type="createTempQueue" roles="guest"/>
 <permission type="deleteTempQueue" roles="guest"/>
 <permission type="consume" roles="guest"/>
 <permission type="send" roles="guest"/>
 </security-setting>
 </security-settings>

 <address-settings>
 <!--default for catch all-->
 <address-setting match="#">
 <clustered>false</clustered>
 <dead-letter-address>jms.queue.DLQ</dead-letter-address>
 <expiry-address>jms.queue.ExpiryQueue</expiry-address>
 <redelivery-delay>0</redelivery-delay>
 <max-size-bytes>-1</max-size-bytes>
 <page-size-bytes>10485760</page-size-bytes>
 <distribution-policy-class>org.jboss.messaging.core.server.impl.RoundRobinDistributor</distribution-policy-class>
 <message-counter-history-day-limit>10</message-counter-history-day-limit>
 <redistribution-delay>0</redistribution-delay>
 </address-setting>
 </address-settings>

</configuration>

jbm-jboss-beans.xml

<?xml version="1.0" encoding="UTF-8"?>

<deployment xmlns="urn:jboss:bean-deployer:2.0">
<bean name="Naming" class="org.jnp.server.NamingBeanImpl"/>

 <!-- JNDI server. Disable this if you don't want JNDI -->
 <bean name="JNDIServer" class="org.jnp.server.Main">
 <property name="namingInfo">
 <inject bean="Naming"/>
 </property>
 <property name="port">2099</property>
 <property name="bindAddress">192.168.110.19</property>
 <property name="rmiPort">2098</property>
 <property name="rmiBindAddress">192.168.110.19</property>
 </bean>

 <!-- MBean server -->
 <bean name="MBeanServer" class="javax.management.MBeanServer">
 <constructor factoryClass="java.lang.management.ManagementFactory"
 factoryMethod="getPlatformMBeanServer"/>
 </bean>

 <!-- The core configuration -->
 <bean name="Configuration" class="org.jboss.messaging.core.config.impl.FileConfiguration"/>

 <!-- The security manager -->
 <bean name="JBMSecurityManager" class="org.jboss.messaging.core.security.impl.JBMSecurityManagerImpl">
 <start ignored="true"/>
 <stop ignored="true"/>
 </bean>
 <!-- The core server -->
 <bean name="MessagingServer" class="org.jboss.messaging.core.server.impl.MessagingServerImpl">
 <constructor>
 <parameter>
 <inject bean="Configuration"/>
 </parameter>
 <parameter>
 <inject bean="MBeanServer"/>
 </parameter>
 <parameter>
 <inject bean="JBMSecurityManager"/>
 </parameter>
 </constructor>
 <start ignored="true"/>
 <stop ignored="true"/>
 </bean>

 <!-- The JMS server -->
 <bean name="JMSServerManager" class="org.jboss.messaging.jms.server.impl.JMSServerManagerImpl">
 <constructor>
 <parameter>
 <inject bean="MessagingServer"/>
 </parameter>
 </constructor>
 </bean>

</deployment>

jbm-jms.xml

<configuration xmlns="urn:jboss:messaging"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:jboss:messaging /schema/jbm-jms.xsd">

 <connection-factory name="ConnectionFactory">
 <connector-ref connector-name="netty-connector2"/>
 <entries>
 <entry name="ConnectionFactory"/>
 <entry name="XAConnectionFactory"/>
 </entries>
 </connection-factory>

 <connection-factory name="ClusteredConnectionFactory">
 <connector-ref connector-name="netty" backup-connector-name="netty-backup"/>
 <entries>
 <entry name="/ClusteredConnectionFactory"/>
 <entry name="/ClusteredXAConnectionFactory"/>
 <entry name="java:/ClusteredConnectionFactory"/>
 <entry name="java:/ClusteredXAConnectionFactory"/>
 </entries>
 </connection-factory>

 <queue name="DLQ">
 <entry name="/queue/DLQ"/>
 </queue>
 <queue name="ExpiryQueue">
 <entry name="/queue/ExpiryQueue"/>
 </queue>
 <queue name="BUYTICKET">
 <entry name="/queue/BUYTICKET"/>
 </queue>


</configuration>

jbm-users.xml

<configuration xmlns="urn:jboss:messaging" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:jboss:messaging /schema/jbm-users.xsd">
 <!-- the default user. this is used where username is null-->
 <defaultuser name="guest" password="guest">
 <role name="guest"/>
 </defaultuser>
</configuration>

[glenwood]

BTW,my jndi property file is below

server1

java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.provider.url=jnp://192.168.110.16:1099
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces

server2

java.naming.factory.initial=org.jnp.interfaces.NamingContextFactory
java.naming.provider.url=jnp://192.168.110.19:2099
java.naming.factory.url.pkgs=org.jboss.naming:org.jnp.interfaces

[ataylor]

Have you tried the clustering examples, do they work for you.

[glenwood]

Ataylor, I didn't run the examples ship with JBM2,cause I would like to deploy a web application on jboss5 in a real cluster Env,so I just simplely copy the configuration file from clustering sample and made a litter changes to reflect my testing jboss Env.

I wonder if by any chance,the jboss clustering configure have conflicts with JBM2 clustering configure?

[ataylor]

Ataylor, I didn't run the examples ship with JBM2,cause I would like to deploy a web application on jboss5 in a real cluster Env,so I just simplely copy the configuration file from clustering sample and made a litter changes to reflect my testing jboss Env.

This would show whether you had any network issues, the cluster connections use UDP and may not be finding each other.

I wonder if by any chance,the jboss clustering configure have conflicts with JBM2 clustering configure?

Unlikely, since they are implemented differently.

[glenwood]

Our jboss5 clustering works fine,2 nodes can find each other,if jboss cluster uses UDP as well?

I already confirmed with our network engineer,UDP was allowed on our testing Env,I will create a simple socket UDP application to verify this through.

BTW,for connection factory look up,I wonder which one I should use, ConnectionFactory or XAConnectionFactory? I don't use transaction in the code.

[glenwood]

I has verified UDP broadcast is working,both servers can receive UDP message,so there should be no networking issue within our jboss cluster Env

[timfox]

You're not specify your hostname on your acceptors, this will default to localhost which is clearly not visible from other machines.

[glenwood]

Timfox,thanks! I just basically neglected this setting.

I have another question,I would like to implement this solution via JBM2 that we have a cluster with multiple nodes been connected each other,we deploy queue on every node with same queue name to form a messaging cluster,we deploy queue producer on several nodes and also deploy consumer on some nodes as well with messaging load balance(server side),our requirement is once one node down(completely server down) which has consumer ,the other consumers will consume the messaging sent to down server,once this server up ,it will resume to consume the queue messages as messages been load balanced to this node again.

I have configured the messaging cluster and queue messaging redistribution,but seems it didn't work as I expected.

Sometime,after I shutdown server2,and sent 10 messages to queue,server1 only consume 5 of them,looks like redistribution failed.Once I restarted the server2,it got rest of 5 messages!

I also keep getting this WARN message after I shutdown/restart 2nd server 3 times,
WARNING [DiscoveryGroupImpl] There seem to be more than one broadcasters on the network broadcasting the same node id

Do you have any suggestion?Thanks in advance!

[venkadesan]

Hello Glenwood,

Can i know what you did to fix the issue?

Since we too face similar kind of issue. We have created two queues in cluster and everything works fine JBoss 5.0.0 GA whereas the same code does not work in JBoss 5.1.0 GA where the messages are sent to two queues in round robin (say out of 10 messages, 5 goes to node1 and 5 goes to node 2). When we try to consume, consumer pointing to node1 can consume only 5 from node 1 and nothing from node2.

Thanks in advance.

Regards,
Venkat

[josey]

It appears that I, too, am facing the same issue as described by venkadesan.

the messages are sent to two queues in round robin (say out of 10 messages, 5 goes to node1 and 5 goes to node 2). When we try to consume, consumer pointing to node1 can consume only 5 from node 1 and nothing from node2.

[josey]

Forgot to mention, I am also running AS 5.1.0 GA with the JbossMessaging that came with it (I believe it is 1.4.3).

[ataylor]

Again, I would suggest running the clustering examples!

[gaohoward]

Also beware of this:

https://jira.jboss.org/jira/browse/JBMESSAGING-1713