8 Replies Latest reply on Sep 9, 2014 3:44 PM by ebmmaxis2007

    Correct way to configure discovery and broadcast groups with TCP

    ebmmaxis2007

      I am using JBOSS EAP 6.2 with AS 7.3.  AS 7.3 does not recognize the <jgroups-file> element available, so I have to use the <jgroups-stack> element.

      UDP multicast broadcast and discovery group configuration works fine, of course.  But, I cannot get the TCP to work with the discovery groups and broadcast groups.

      I have the following:

       

           <connectors>

            <netty-connector name="netty" socket-binding="messaging"/> <!-- port 5445 -->

           </connectors>

       

                 <stack name="tcp">

                        <transport type="TCP" socket-binding="jgroups-tcp"/>

                        <protocol type="TCPPING">

                             <property name="initial_hosts">192.168.2.14[7600], 192.168.2.14[7700]</property>

                        </protocol>

                        <protocol type="MERGE2"/>

                        <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

                        <protocol type="FD"/>

                        <protocol type="VERIFY_SUSPECT"/>

                        <protocol type="pbcast.NAKACK"/>

                        <protocol type="UNICAST2"/>

                        <protocol type="pbcast.STABLE"/>

                        <protocol type="pbcast.GMS"/>

                        <protocol type="UFC"/>

                        <protocol type="MFC"/>

                        <protocol type="FRAG2"/>

                   <protocol type="RSVP"/>

          </stack>

       

                      <broadcast-groups>

                          <broadcast-group name="bg-group1">

                              <jgroups-stack>tcp</jgroups-stack>

                              <jgroups-channel>my-cluster</jgroups-channel>

                              <broadcast-period>5000</broadcast-period>

                              <connector-ref>

                                  netty

                              </connector-ref>

                          </broadcast-group>

                      </broadcast-groups>

       

                      <discovery-groups>

                          <discovery-group name="dg-group1">

                              <jgroups-stack>tcp</jgroups-stack>

                              <jgroups-channel>my-cluster</jgroups-channel>

                              <refresh-timeout>10000</refresh-timeout>

                          </discovery-group>

                      </discovery-groups>

                      <cluster-connections>

                          <cluster-connection name="my-cluster">

                              <address>jms</address>

                              <connector-ref>netty</connector-ref>

                              <discovery-group-ref discovery-group-name="dg-group1"/>

                          </cluster-connection>

                      </cluster-connections>

       

       

      When starting the live server, this is what I the following from the log:

       

      INFO  [stdout] (MSC service thread 1-6) GMS: address=nodeA/my-cluster, cluster=my-cluster, physical address=192.168.2.14:7600

      (MSC service thread 1-6) HQ224000: Failure in initialisation: java.lang.IllegalStateException: channel is closed

        at org.jgroups.JChannel.checkClosed(JChannel.java:865) [jgroups-3.2.7.Final.jar:3.2.7.Final]

        at org.jgroups.JChannel._preConnect(JChannel.java:520) [jgroups-3.2.7.Final.jar:3.2.7.Final]

        at org.jgroups.JChannel.connect(JChannel.java:277) [jgroups-3.2.7.Final.jar:3.2.7.Final]

        at org.jgroups.JChannel.connect(JChannel.java:268) [jgroups-3.2.7.Final.jar:3.2.7.Final]

        at org.hornetq.api.core.JGroupsBroadcastGroupConfiguration$JGroupsBroadcastEndpoint$JChannelWrapper.connect(JGroupsBroadcastGroupConfiguration.java:353) [hornetq-core-client-2.3.21.jar:]

        at org.hornetq.api.core.JGroupsBroadcastGroupConfiguration$JGroupsBroadcastEndpoint.internalOpen(JGroupsBroadcastGroupConfiguration.java:203) [hornetq-core-client-2.3.21.jar:]

        at org.hornetq.api.core.JGroupsBroadcastGroupConfiguration$JGroupsBroadcastEndpoint.openBroadcaster(JGroupsBroadcastGroupConfiguration.java:182) [hornetq-core-client-2.3.21.jar:]

        at org.hornetq.core.server.cluster.impl.BroadcastGroupImpl.start(BroadcastGroupImpl.java:108) [hornetq-server-2.3.21.jar:]

        at org.hornetq.core.server.cluster.ClusterManager.start(ClusterManager.java:268) [hornetq-server-2.3.21.jar:]

        at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1658) [hornetq-server-2.3.21.jar:]

        at org.hornetq.core.server.impl.HornetQServerImpl.access$1400(HornetQServerImpl.java:178) [hornetq-server-2.3.21.jar:]

        at org.hornetq.core.server.impl.HornetQServerImpl$SharedNothingLiveActivation.run(HornetQServerImpl.java:2831) [hornetq-server-2.3.21.jar:]

        at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:435) [hornetq-server-2.3.21.jar:]

        at org.hornetq.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:490) [hornetq-jms-server-2.3.21.jar:]

        at org.jboss.as.messaging.jms.JMSService.start(JMSService.java:74) [jboss-as-messaging-7.2.0.Final.jar:7.2.0.Final]

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

        at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]

        at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

       

      What is the correct way to configure this?

      What does the <jboss-channel> need to be set to?  Why is the channel 'closed?'

      Is the tcp stack set up correctly?

      Any help will be greatly appreciated.

        • 1. Re: Correct way to configure discovery and broadcast groups with TCP
          jbertram

          Please attach your full server configuration file.

           

          Also, if your goal is simply to have TCP based static clustering then I would recommend you drop the JGroups stuff, broadcast group, and discovery group and simply configure static cluster connections (see the bottom of section 38.3.1).  In my opinion this is much simpler and clearer than using JGroups.

          • 2. Re: Correct way to configure discovery and broadcast groups with TCP
            ebmmaxis2007

            Just uploaded. I added in 'static-connectors' to that uploaded file, but didn't try it out yet.

             

            Due to the documentation, the tcp jgroups configuration can be quite dynamic in the voting process of the cluster, i.e. which nodes will be kicked out of the cluster due to no ping replies and other criteria.  Will the static connectors still provide this same functionality?

             

            Per the documenation:

            Much like in the shared-store case, when the live server stops or crashes, its replicating backup will become active and take over its duties. Specifically, the backup will become active when it loses connection to its live server. This can be problematic because this can also happen because of a temporary network problem. In order to address this issue, the backup will try to determine whether it still can connect to the other servers in the cluster. If it can connect to more than half the servers, it will become active, if more than half the servers also disappeared with the live, the backup will wait and try reconnecting with the live. This avoids a split brain situation.


            I may be mistaken, but I thought that the jgroups tcp configuration could actually affect the way quorum vote is determined before a backup takes over as live.

            • 3. Re: Correct way to configure discovery and broadcast groups with TCP
              jbertram

              I believe you are mistaken about how HornetQ leverages JGroups.  JGroups is only used for server discovery.  Once other cluster servers have been discovered the cluster nodes connect to each other via Netty TCP connections.  JGroups isn't used for any kind of cluster monitoring, pinging, kicking, quorum voting, etc.  This is all done internally by HornetQ.

              • 4. Re: Correct way to configure discovery and broadcast groups with TCP
                ebmmaxis2007

                Hmmm... Ok.  So, the better approach is to bypass discovery and broadcasting and only use static connectors?  Please elaborate if you don't mind.  The documentation is somewhat sparse on this subject.  It only states to use static connectors, but does not explain how they replace jgroup broadcasting and discovery.

                • 5. Re: Correct way to configure discovery and broadcast groups with TCP
                  gaohoward

                  There is a related bug regarding using jgroups for discovery:

                   

                  https://issues.jboss.org/browse/HORNETQ-1389

                  • 6. Re: Correct way to configure discovery and broadcast groups with TCP
                    gaohoward

                    One possible way to verify if your tcp stack is ok or not is try using the stack to create a JChannel. If your tck stack is not correct then the JChannel (or its subclasses) may not be created in EAP.

                    • 7. Re: Correct way to configure discovery and broadcast groups with TCP
                      jbertram

                      Ok.  So, the better approach is to bypass discovery and broadcasting and only use static connectors?

                      That depends on your goal.  Is your goal to use TCP instead of UDP for server discovery and configure a simple, non-dynamic list of nodes in the cluster?  If so, then using static connectors is a good fit for you.

                       

                      The documentation is somewhat sparse on this subject.  It only states to use static connectors, but does not explain how they replace jgroup broadcasting and discovery.

                      Read section 38.2.1 of the HornetQ User Guide.  Once you understand what discovery groups and broadcast groups are for I think you'll see why they aren't necessary when using static connectors.

                      1 of 1 people found this helpful
                      • 8. Re: Correct way to configure discovery and broadcast groups with TCP
                        ebmmaxis2007

                        Alright. I got everything working with static connections.   Indeed, it is much clearer.  I am still curious as to how to configure the broadcast and discovery group with the <jgroups-stack> element.