0 Replies Latest reply on Sep 25, 2012 11:08 PM by jacklund

    Problem clustering statically connected queues in JBoss 7.1.1.Final

    jacklund
    jacklund Sep 25, 2012 11:08 PM

      I'm trying to set up a cluster of four JBoss servers, with one server having a message producer, and the other three having queues which are clustered, and which have the message consumers. My machine names are jboss1, jboss2, and jboss4 for the consumers, and jboss3 is the producer. The queues are also on jboss1, jboss2, and jboss4. I'm in an environment where I can't use multicast, so I'm having to use static connections to cluster the queue servers.

       

      I originally started with just machines 1-3 clustered together, which worked great - they all were set up as a domain with jboss1 as the controller, and the queues on jboss1 and 2 clustered together so that messages sent from the producer were load-balanced across both queues.

       

      However, when I tried to add jboss4 into the mix, I can't seem to get its queue to cluster with the other ones. I've got it set up to connect to the netty ports on jboss1 and 2, and I've made sure the connections are there, which they are, but when they are all started up, and jboss3 sends messages, they only end up on the queues for jboss1 and jboss2. Here are the relevant config entries, and I'm including the logs for all three servers as attachments.

       

      jboss1 has the following messaging set up (in profile 'full-ha'):

       

              
<subsystem xmlns="urn:jboss:domain:messaging:1.1">
            
<hornetq-server>
                
<clustered>true</clustered>
                
<persistence-enabled>true</persistence-enabled>
                
<cluster-password>some_password</cluster-password>
                
<journal-file-size>102400</journal-file-size>
                
<journal-min-files>2</journal-min-files>


                
<connectors>
                    
<netty-connector name="netty" socket-binding="messaging"/>
                    
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                        
<param key="batch-delay" value="50"/>
                    
</netty-connector>
                    
<netty-connector name="jboss2-connector" socket-binding="jboss2"/>
                    
<in-vm-connector name="in-vm" server-id="0"/>
                
</connectors>


                
<acceptors>
                    
<netty-acceptor name="netty" socket-binding="messaging"/>
                    
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                        
<param key="batch-delay" value="50"/>
                        
<param key="direct-deliver" value="false"/>
                    
</netty-acceptor>
                    
<in-vm-acceptor name="in-vm" server-id="0"/>
                
</acceptors>


                
<broadcast-groups>
                    
<broadcast-group name="bg-group1">
                        
<group-address>231.7.7.7</group-address>
                        
<group-port>9876</group-port>
                        
<broadcast-period>5000</broadcast-period>
                        
<connector-ref>
                            
netty
                        
</connector-ref>
                    
</broadcast-group>
                
</broadcast-groups>


                
<discovery-groups>
                    
<discovery-group name="dg-group1">
                        
<group-address>231.7.7.7</group-address>
                        
<group-port>9876</group-port>
                        
<refresh-timeout>10000</refresh-timeout>
                    
</discovery-group>
                
</discovery-groups>


                
<cluster-connections>
                    
<cluster-connection name="my-cluster">
                        
<address>jms</address>
                        
<connector-ref>netty</connector-ref>
                        
<static-connectors>
                            
<connector-ref>
                                
jboss2-connector
                            
</connector-ref>
                        
</static-connectors>
                    
</cluster-connection>
                
</cluster-connections>


                
<security-settings>
                    
<security-setting match="#">
                        
<permission type="send" roles="guest"/>
                        
<permission type="consume" roles="guest"/>
                        
<permission type="createNonDurableQueue" roles="guest"/>
                        
<permission type="deleteNonDurableQueue" roles="guest"/>
                    
</security-setting>
                
</security-settings>


                
<address-settings>
                    
<address-setting match="#">
                        
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        
<redelivery-delay>0</redelivery-delay>
                        
<max-size-bytes>10485760</max-size-bytes>
                        
<address-full-policy>BLOCK</address-full-policy>
                        
<message-counter-history-day-limit>10</message-counter-history-day-limit>
                        
<redistribution-delay>1000</redistribution-delay>
                    
</address-setting>
                
</address-settings>


                
<jms-connection-factories>
                    
<connection-factory name="InVmConnectionFactory">
                        
<connectors>
                            
<connector-ref connector-name="in-vm"/>
                        
</connectors>
                        
<entries>
                            
<entry name="java:/ConnectionFactory"/>
                        
</entries>
                    
</connection-factory>
                    
<connection-factory name="RemoteConnectionFactory">
                        
<connectors>
                            
<connector-ref connector-name="netty"/>
                        
</connectors>
                        
<entries>
                            
<entry name="RemoteConnectionFactory"/>
                            
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        
</entries>
                    
</connection-factory>
                    
<pooled-connection-factory name="hornetq-ra">
                        
<transaction mode="xa"/>
                        
<connectors>
                            
<connector-ref connector-name="in-vm"/>
                        
</connectors>
                        
<entries>
                            
<entry name="java:/JmsXA"/>
                        
</entries>
                        
<block-on-durable-send>false</block-on-durable-send>
                    
</pooled-connection-factory>
                
</jms-connection-factories>


                
<jms-destinations>
                    
<jms-queue name="testQueue">
                        
<entry name="queue/test"/>
                        
<entry name="java:jboss/exported/jms/queue/test"/>
                    
</jms-queue>
                    
<jms-queue name="deviceAPIQueue">
                        
<entry name="java:/queue/deviceAPI"/>
                        
<durable>true</durable>
                    
</jms-queue>
                    
<jms-topic name="testTopic">
                        
<entry name="topic/test"/>
                        
<entry name="java:jboss/exported/jms/topic/test"/>
                    
</jms-topic>
                
</jms-destinations>
            
</hornetq-server>
        
</subsystem>


And here's the socket bindings section:

    
<socket-binding-group name="full-ha-sockets" default-interface="public">
        
<socket-binding name="ajp" port="8009"/>
        
<socket-binding name="http" port="8080"/>
        
<socket-binding name="https" port="8443"/>
        
<socket-binding name="jacorb" interface="unsecure" port="3528"/>
        
<socket-binding name="jacorb-ssl" interface="unsecure" port="3529"/>
        
<socket-binding name="jgroups-diagnostics" port="0" multicast-address="224.0.75.75" multicast-port="7500"/>
        
<socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        
<socket-binding name="jgroups-tcp" port="7600"/>
        
<socket-binding name="jgroups-tcp-fd" port="57600"/>
        
<socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        
<socket-binding name="jgroups-udp-fd" port="54200"/>
        
<socket-binding name="messaging" port="5445"/>
        
<socket-binding name="messaging-throughput" port="5455"/>
        
<socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
        
<socket-binding name="osgi-http" interface="management" port="8090"/>
        
<socket-binding name="remoting" port="4447"/>
        
<socket-binding name="txn-recovery-environment" port="4712"/>
        
<socket-binding name="txn-status-manager" port="4713"/>
        
<outbound-socket-binding name="mail-smtp">
            
<remote-destination host="localhost" port="25"/>
        
</outbound-socket-binding>
        
<outbound-socket-binding name="jboss2">
            
<remote-destination host="jboss2" port="5445"/>
        
</outbound-socket-binding>
    
</socket-binding-group>


The messaging section for jboss2:

        
<subsystem xmlns="urn:jboss:domain:messaging:1.1">
            
<hornetq-server>
                
<clustered>true</clustered>
                
<cluster-password>some_password</cluster-password>
                
<persistence-enabled>true</persistence-enabled>
                
<journal-file-size>102400</journal-file-size>
                
<journal-min-files>2</journal-min-files>
                
<connectors>
                    
<netty-connector name="netty" socket-binding="messaging"/>
                    
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                        
<param key="batch-delay" value="50"/>
                    
</netty-connector>
                    
<netty-connector name="jboss1-connector" socket-binding="jboss1"/>
                    
<in-vm-connector name="in-vm" server-id="0"/>
                
</connectors>
                
<acceptors>
                    
<netty-acceptor name="netty" socket-binding="messaging"/>
                    
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                        
<param key="batch-delay" value="50"/>
                        
<param key="direct-deliver" value="false"/>
                    
</netty-acceptor>
                    
<in-vm-acceptor name="in-vm" server-id="0"/>
                
</acceptors>
                
<broadcast-groups>
                    
<broadcast-group name="bg-group1">
                        
<group-address>231.7.7.7</group-address>
                        
<group-port>9876</group-port>
                        
<broadcast-period>5000</broadcast-period>
                        
<connector-ref>netty</connector-ref>
                    
</broadcast-group>
                
</broadcast-groups>
                
<discovery-groups>
                    
<discovery-group name="dg-group1">
                        
<group-address>231.7.7.7</group-address>
                        
<group-port>9876</group-port>
                        
<refresh-timeout>10000</refresh-timeout>
                    
</discovery-group>
                
</discovery-groups>
                
<cluster-connections>
                    
<cluster-connection name="my-cluster">
                        
<address>jms</address>
                        
<connector-ref>netty</connector-ref>
                        
<static-connectors>
                            
<connector-ref>jboss1-connector</connector-ref>
                        
</static-connectors>
                    
</cluster-connection>
                
</cluster-connections>
                
<security-settings>
                    
<security-setting match="#">
                        
<permission type="send" roles="guest"/>
                        
<permission type="consume" roles="guest"/>
                        
<permission type="createNonDurableQueue" roles="guest"/>
                        
<permission type="deleteNonDurableQueue" roles="guest"/>
                    
</security-setting>
                
</security-settings>
                
<address-settings>
                    
<!--default for catch all-->
                    
<address-setting match="#">
                        
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        
<redelivery-delay>0</redelivery-delay>
                        
<redistribution-delay>1000</redistribution-delay>
                        
<max-size-bytes>10485760</max-size-bytes>
                        
<address-full-policy>BLOCK</address-full-policy>
                        
<message-counter-history-day-limit>10</message-counter-history-day-limit>
                    
</address-setting>
                
</address-settings>
                
<jms-connection-factories>
                    
<connection-factory name="InVmConnectionFactory">
                        
<connectors>
                            
<connector-ref connector-name="in-vm"/>
                        
</connectors>
                        
<entries>
                            
<entry name="java:/ConnectionFactory"/>
                        
</entries>
                    
</connection-factory>
                    
<connection-factory name="RemoteConnectionFactory">
                        
<connectors>
                            
<connector-ref connector-name="netty"/>
                        
</connectors>
                        
<entries>
                            
<entry name="RemoteConnectionFactory"/>
                            
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        
</entries>
                    
</connection-factory>
                    
<pooled-connection-factory name="hornetq-ra">
                        
<transaction mode="xa"/>
                        
<block-on-durable-send>false</block-on-durable-send>
                        
<connectors>
                            
<connector-ref connector-name="in-vm"/>
                        
</connectors>
                        
<entries>
                            
<entry name="java:/JmsXA"/>
                        
</entries>
                    
</pooled-connection-factory>
                
</jms-connection-factories>
                
<jms-destinations>
                    
<jms-queue name="testQueue">
                        
<entry name="queue/test"/>
                        
<entry name="java:jboss/exported/jms/queue/test"/>
                    
</jms-queue>
                    
<jms-queue name="deviceAPIQueue">
                        
<entry name="java:/queue/deviceAPI"/>
                        
<durable>true</durable>
                    
</jms-queue>
                    
<jms-topic name="testTopic">
                        
<entry name="topic/test"/>
                        
<entry name="java:jboss/exported/jms/topic/test"/>
                    
</jms-topic>
                
</jms-destinations>
            
</hornetq-server>
        
</subsystem>

       

      And the socket bindings:

       

          
<socket-binding-group name="full-ha-sockets" default-interface="public">
        
<!-- Needed for server groups using the 'full-ha' profile  -->
        
<socket-binding name="ajp" port="8009"/>
        
<socket-binding name="http" port="8080"/>
        
<socket-binding name="https" port="8443"/>
        
<socket-binding name="jacorb" interface="unsecure" port="3528"/>
        
<socket-binding name="jacorb-ssl" interface="unsecure" port="3529"/>
        
<socket-binding name="jgroups-diagnostics" port="0" multicast-address="224.0.75.75" multicast-port="7500"/>
        
<socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        
<socket-binding name="jgroups-tcp" port="7600"/>
        
<socket-binding name="jgroups-tcp-fd" port="57600"/>
        
<socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        
<socket-binding name="jgroups-udp-fd" port="54200"/>
        
<socket-binding name="messaging" port="5445"/>
        
<socket-binding name="messaging-throughput" port="5455"/>
        
<socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
        
<socket-binding name="osgi-http" interface="management" port="8090"/>
        
<socket-binding name="remoting" port="4447"/>
        
<socket-binding name="txn-recovery-environment" port="4712"/>
        
<socket-binding name="txn-status-manager" port="4713"/>
        
<outbound-socket-binding name="mail-smtp">
            
<remote-destination host="localhost" port="25"/>
        
</outbound-socket-binding>
        
<outbound-socket-binding name="jboss1">
            
<remote-destination host="jboss1" port="5445"/>
        
</outbound-socket-binding>
    
</socket-binding-group>

</socket-binding-groups>

       

      And for jboss4:

       

              
<subsystem xmlns="urn:jboss:domain:messaging:1.1">
            
<hornetq-server>
                
<clustered>true</clustered>
                
<cluster-password>some_password</cluster-password>
                
<persistence-enabled>true</persistence-enabled>
                
<journal-file-size>102400</journal-file-size>
                
<journal-min-files>2</journal-min-files>
                
<connectors>
                    
<netty-connector name="netty" socket-binding="messaging"/>
                    
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
                        
<param key="batch-delay" value="50"/>
                    
</netty-connector>
                    
<netty-connector name="jboss1-connector" socket-binding="jboss1"/>
                    
<netty-connector name="jboss2-connector" socket-binding="jboss2"/>
                    
<in-vm-connector name="in-vm" server-id="0"/>
                
</connectors>
                
<acceptors>
                    
<netty-acceptor name="netty" socket-binding="messaging"/>
                    
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
                        
<param key="batch-delay" value="50"/>
                        
<param key="direct-deliver" value="false"/>
                    
</netty-acceptor>
                    
<in-vm-acceptor name="in-vm" server-id="0"/>
                
</acceptors>
                
<broadcast-groups>
                    
<broadcast-group name="bg-group1">
                        
<group-address>231.7.7.7</group-address>
                        
<group-port>9876</group-port>
                        
<broadcast-period>5000</broadcast-period>
                        
<connector-ref>netty</connector-ref>
                    
</broadcast-group>
                
</broadcast-groups>
                
<discovery-groups>
                    
<discovery-group name="dg-group1">
                        
<group-address>231.7.7.7</group-address>
                        
<group-port>9876</group-port>
                        
<refresh-timeout>10000</refresh-timeout>
                    
</discovery-group>
                
</discovery-groups>
                
<cluster-connections>
                    
<cluster-connection name="my-cluster">
                        
<address>jms</address>
                        
<connector-ref>netty</connector-ref>
                        
<static-connectors>
                            
<connector-ref>jboss1-connector</connector-ref>
                            
<connector-ref>jboss2-connector</connector-ref>
                        
</static-connectors>
                    
</cluster-connection>
                
</cluster-connections>
                
<security-settings>
                    
<security-setting match="#">
                        
<permission type="send" roles="guest"/>
                        
<permission type="consume" roles="guest"/>
                        
<permission type="createNonDurableQueue" roles="guest"/>
                        
<permission type="deleteNonDurableQueue" roles="guest"/>
                    
</security-setting>
                
</security-settings>
                
<address-settings>
                    
<!--default for catch all-->
                    
<address-setting match="#">
                        
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        
<redelivery-delay>0</redelivery-delay>
                        
<redistribution-delay>1000</redistribution-delay>
                        
<max-size-bytes>10485760</max-size-bytes>
                        
<address-full-policy>BLOCK</address-full-policy>
                        
<message-counter-history-day-limit>10</message-counter-history-day-limit>
                    
</address-setting>
                
</address-settings>
                
<jms-connection-factories>
                    
<connection-factory name="InVmConnectionFactory">
                        
<connectors>
                            
<connector-ref connector-name="in-vm"/>
                        
</connectors>
                        
<entries>
                            
<entry name="java:/ConnectionFactory"/>
                        
</entries>
                    
</connection-factory>
                    
<connection-factory name="RemoteConnectionFactory">
                        
<connectors>
                            
<connector-ref connector-name="netty"/>
                        
</connectors>
                        
<entries>
                            
<entry name="RemoteConnectionFactory"/>
                            
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        
</entries>
                    
</connection-factory>
                    
<pooled-connection-factory name="hornetq-ra">
                        
<transaction mode="xa"/>
                        
<block-on-durable-send>false</block-on-durable-send>
                        
<connectors>
                            
<connector-ref connector-name="in-vm"/>
                        
</connectors>
                        
<entries>
                            
<entry name="java:/JmsXA"/>
                        
</entries>
                    
</pooled-connection-factory>
                
</jms-connection-factories>
                
<jms-destinations>
                    
<jms-queue name="testQueue">
                        
<entry name="queue/test"/>
                        
<entry name="java:jboss/exported/jms/queue/test"/>
                    
</jms-queue>
                    
<jms-queue name="deviceAPIQueue">
                        
<entry name="java:/queue/deviceAPI"/>
                        
<durable>true</durable>
                    
</jms-queue>
                    
<jms-topic name="testTopic">
                        
<entry name="topic/test"/>
                        
<entry name="java:jboss/exported/jms/topic/test"/>
                    
</jms-topic>
                
</jms-destinations>
            
</hornetq-server>
        
</subsystem>

       

      and

       

          
<socket-binding-group name="full-ha-sockets" default-interface="public">
        
<!-- Needed for server groups using the 'full-ha' profile  -->
        
<socket-binding name="ajp" port="8009"/>
        
<socket-binding name="http" port="8080"/>
        
<socket-binding name="https" port="8443"/>
        
<socket-binding name="jacorb" interface="unsecure" port="3528"/>
        
<socket-binding name="jacorb-ssl" interface="unsecure" port="3529"/>
        
<socket-binding name="jgroups-diagnostics" port="0" multicast-address="224.0.75.75" multicast-port="7500"/>
        
<socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        
<socket-binding name="jgroups-tcp" port="7600"/>
        
<socket-binding name="jgroups-tcp-fd" port="57600"/>
        
<socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        
<socket-binding name="jgroups-udp-fd" port="54200"/>
        
<socket-binding name="messaging" port="5445"/>
        
<socket-binding name="messaging-throughput" port="5455"/>
        
<socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
        
<socket-binding name="osgi-http" interface="management" port="8090"/>
        
<socket-binding name="remoting" port="4447"/>
        
<socket-binding name="txn-recovery-environment" port="4712"/>
        
<socket-binding name="txn-status-manager" port="4713"/>
        
<outbound-socket-binding name="mail-smtp">
            
<remote-destination host="localhost" port="25"/>
        
</outbound-socket-binding>
        
<outbound-socket-binding name="jboss1">
            
<remote-destination host="jboss1" port="5445"/>
        
</outbound-socket-binding>
    
</socket-binding-group>
        
<outbound-socket-binding name="jboss2">
            
<remote-destination host="jboss2" port="5445"/>
        
</outbound-socket-binding>
    
</socket-binding-group>

       

      Note that I'm defining the cluster connections without using factory-class, per https://issues.jboss.org/browse/AS7-4414. I'm assuming this is the workaround which is referred to in that bug report. It worked well for two machines clustered together, anyway.

       

      Any help would be greatly appreciated!

       

      -Jack