12 Replies Latest reply on May 17, 2013 6:09 AM by akostadinov

    Lost in the documentation UDP->TCP

      I hope I don't sound too much like an idiot here but I'm having problems establishing clustering/loadbalancing between two servers.

      I have installed 5.1.0.GA have done a few changes in the configuration mostly involved with getting my DefaultDS set correctly.

      I have Read through the bulk of the Clustering Guide PDF and the wiki here: http://www.jboss.org/community/wiki/MajorClusteringChangesinJBossAS5 and here: http://www.jboss.org/community/wiki/JBossHAJNDIImpl and here: http://www.jboss.org/community/wiki/JBossHA

      Also Configure ports wiki: http://www.jboss.org/community/wiki/ConfigurePorts

      To start with almost all the docs including the "Major Clustering Changes in JBoss AS 5" refer to "cluster-service.xml" the only place I found that explained the absence was: http://www.jboss.org/community/wiki/AS5HighLevelClusteringServiceConfigurationChanges

      Which pointed me to "deploy/cluster/ha-jndi-jboss-beans.xml" which I had already found. Looking through this config it appears to me that there isn't a "PartitionConfig" anymore, but rather some new properties like "discoveryDisabled" and "autoDiscoveryAddress" which utilize the ServiceBindingManager and binding names of "AutoDiscovery" and "AutoDiscoveryInterface"

      So I've done some leg work here attempting to search/dig and answer my own question. I am going to have 2 clusters in 4 physical machines that exist on the same network. I am using the -g command line switch to isolate the partitions. Where I am stuck however is the network admin has set up the VLAN in such a way that MCast is not an option.

      I need to set up the cluster discovery to utilize TCP/TCPPing, the number of servers is static as are the IPs.

      However I can't find anything in the documentation covering how to move from mcast to unicast in the new microcontainer configuration. It seems to me I have to make changes in hajndi-jboss-beans.xml to cover the mode change and in bindings-jboss-beans.xml to define the proper bind addresses and ports.

      Help would be appreciated, even if it's a point to some documentation I missed.

      -Andrew

        • 1. Re: Lost in the documentation UDP->TCP

          Digging a little further I've looked through the HANamingService class and it's dependencies and I don't see properties that I would expect to see if TCP/unicast/TCPPing were supported.

          For example an "InitialHosts" or a "TransportProtocol" is cluster currently limited to UDP/MCast?

          • 2. Re: Lost in the documentation UDP->TCP
            brian.stansberry

            Good question. This topic deserves a wiki page of its own.

            Simplest way to solve your problem:

            1) Include the following in your command line args that you pass to JBoss:

            -Djboss.default.jgroups.stack=tcp


            That will change most of the services that use a JGroups channel to use one based on the "tcp" stack instead of the default "udp" stack.

            2) The actual JGroups configurations for the "tcp" stack and the "udp" stack are found in the deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml file. For more on the service that configures, see [1]

            You can't use multicast, but the "tcp" config actually uses multicast for it's peer-discovery function. It does this because not using multicast requires configuring the addresses/ports of all the possible nodes in the cluster, which is of course environment-specific and doesn't just work out of the box. But, the "tcp" config in jgroups-channelfactory-stacks.xml tries to help. It includes two discovery protocol configurations, one "MPING" that is uncommented and one "TCPPING" that is commented. You need to comment out MPING and uncommment TCPPING.

            For more on MPING and TCPPING see [2] and [3].

            3) As noted above, not using multicast requires configuring the addresses/ports of all the possible nodes in the cluster. You need to compile this information. The port is the value of the "tcp" stack's TCP.start_port configuration, by default 7600. The address for each node is usually the value you pass to -b when you start JBoss; can also be the value you assign to -Djgroups.bind_addr if you're using that technique to tell JGroups to use a different interface than the -b one. You can also use hostnames instead of addresses.

            Gather all these into a comma-delimited list and pass them to JBoss at startup via a system property:

            -Djgroups.tcpping.initial_hosts=192.168.0.100[7600],192.168.0.101[7600],192.168.0.102[7600]


            (Edit: when I originally posted the above it was incorrect; see next post below. I've corrected the mistake.)

            4) If you are running clustered JBoss Messaging, JBM has their own UDP multicast-based JGroups config they use. You'll need to change that to use the "tcp" config. This is configured in the deploy/messaging/XXX-persistence-service.xml file. (The value of XXX will depend on what RDBMS you are using for message persistence.) Change:

             <mbean code="org.jboss.messaging.core.jmx.MessagingPostOfficeService" ...>
            
             . . .
            
             <attribute name="ControlChannelName">jbm-control</attribute>
            
             . . .
             </mbean>
            


            to



             <mbean code="org.jboss.messaging.core.jmx.MessagingPostOfficeService" ...">
            
             . . .
            
             <attribute name="ControlChannelName">tcp</attribute>
            
             . . .
             </mbean>
            


            [1] http://www.jboss.org/community/wiki/JGroupsChannelFactoryandSharedTransportinJBossAS5

            [2] http://www.jboss.org/community/docs/DOC-10897

            [3] http://www.jboss.org/community/docs/DOC-10915

            • 3. Re: Lost in the documentation UDP->TCP

              Thank you, this appears to be exactly the information I needed.

              • 4. Re: Lost in the documentation UDP->TCP

                Ok to update this thread in case anyone else is looking for the same solution. I followed the post by bstansberry and modified my init script. Ended up with this run command:

                /opt/jboss/bin/run.sh -c all -b 0.0.0.0 -g cameldev -Djboss.default.jgroups.stack=tcp -Djgroups.tcpping.initial_hosts=cameldev1:7600,cameldev2:7600
                


                Which critically fails the server start. I looked over the default values in jgroups-channelfactory-stacks.xml and realized the fault.

                Functional run command:
                /opt/jboss/bin/run.sh -c default -b 0.0.0.0 -g cameldev -Djboss.default.jgroups.stack=tcp -Djgroups.tcpping.initial_hosts=cameldev1[7600],cameldev2[7600]
                


                The key difference being how the ports are identifed.

                Also in case it helps someone, here is my modified init script
                #!/bin/sh
                #
                # $Id: jboss_init_redhat.sh 81068 2008-11-14 15:14:35Z dimitris@jboss.org $
                #
                # JBoss Control Script
                # chkconfig: - 85 15
                # description: Runs JBoss service
                #
                
                #define where jboss is - this is the directory containing directories log, bin, conf etc
                JBOSS_HOME=${JBOSS_HOME:-"/opt/jboss"}
                
                #Define location for console log
                JBOSS_CONSOLE="$JBOSS_HOME/server/default/log/jboss.log"
                
                #define the user under which jboss will run, or use 'RUNASIS' to run as the current user
                JBOSS_USER=${JBOSS_USER:-"jboss"}
                
                #make sure java is in your path
                JAVAPTH=${JAVAPTH:-"/usr/java/jdk1.6.0_14"}
                
                #configuration to use, usually one of 'minimal', 'default', 'all'
                JBOSS_CONF=${JBOSS_CONF:-"default"}
                
                #IP Address to Bind to 0.0.0.0 binds all IP default is loopback only
                JBOSS_HOST="0.0.0.0"
                
                #Partition Name for Clustering
                JBOSS_PARTION="cameldev"
                
                #JGroups Stack leave blank for default UDP stack
                JGROUPS_STACK="tcp"
                
                #Cluster Hosts if Stack is tcp and TCPPing is enabled in jgroups-channelfactory-stacks.xml
                JGROUPS_HOSTS="cameldev1[7600],cameldev2[7600]"
                
                #If JBOSS_HOST specified, use -b to bind jboss services to that address
                JBOSS_BIND_ADDR=${JBOSS_HOST:+"-b $JBOSS_HOST"}
                
                #If JBOSS_PARTION specified, use -g to bind jboss services to that address
                JBOSS_PARTION_NAME=${JBOSS_PARTION:+"-g $JBOSS_PARTION"}
                
                #Set command line flag for JGroups stack
                JGROUPS_STACK_LINE=${JGROUPS_STACK:+"-Djboss.default.jgroups.stack=$JGROUPS_STACK"}
                
                #Set command line flag for JGroup Inital Hosts
                JGROUPS_HOSTS_LINE=${JGROUPS_HOSTS:+"-Djgroups.tcpping.initial_hosts=$JGROUPS_HOSTS"}
                
                #define the classpath for the shutdown class
                JBOSSCP=${JBOSSCP:-"$JBOSS_HOME/bin/shutdown.jar:$JBOSS_HOME/client/jnet.jar"}
                
                #define the script to use to start jboss
                JBOSSSH=${JBOSSSH:-"$JBOSS_HOME/bin/run.sh -c $JBOSS_CONF $JBOSS_BIND_ADDR $JBOSS_PARTION_NAME $JGROUPS_STACK_LINE $JGROUPS_HOSTS_LINE"}
                
                if [ "$JBOSS_USER" = "RUNASIS" ]; then
                 SUBIT=""
                else
                 SUBIT="su - $JBOSS_USER -c "
                fi
                
                if [ -n "$JBOSS_CONSOLE" -a ! -d "$JBOSS_CONSOLE" ]; then
                 # ensure the file exists
                 touch $JBOSS_CONSOLE
                 if [ ! -z "$SUBIT" ]; then
                 chown $JBOSS_USER $JBOSS_CONSOLE
                 fi
                fi
                
                if [ -n "$JBOSS_CONSOLE" -a ! -f "$JBOSS_CONSOLE" ]; then
                 echo "WARNING: location for saving console log invalid: $JBOSS_CONSOLE"
                 echo "WARNING: ignoring it and using /dev/null"
                 JBOSS_CONSOLE="/dev/null"
                fi
                
                #define what will be done with the console log
                JBOSS_CONSOLE=${JBOSS_CONSOLE:-"/dev/null"}
                
                JBOSS_CMD_START="cd $JBOSS_HOME/bin; $JBOSSSH"
                JBOSS_CMD_STOP=${JBOSS_CMD_STOP:-"java -classpath $JBOSSCP org.jboss.Shutdown --shutdown"}
                
                if [ -z "`echo $PATH | grep $JAVAPTH`" ]; then
                 export PATH=$PATH:$JAVAPTH
                fi
                
                if [ ! -d "$JBOSS_HOME" ]; then
                 echo JBOSS_HOME does not exist as a valid directory : $JBOSS_HOME
                 exit 1
                fi
                
                echo JBOSS_CMD_START = $JBOSS_CMD_START
                
                case "$1" in
                start)
                 cd $JBOSS_HOME/bin
                 if [ -z "$SUBIT" ]; then
                 eval $JBOSS_CMD_START >${JBOSS_CONSOLE} 2>&1 &
                 else
                 $SUBIT "$JBOSS_CMD_START >${JBOSS_CONSOLE} 2>&1 &"
                 fi
                 ;;
                stop)
                 if [ -z "$SUBIT" ]; then
                 $JBOSS_CMD_STOP
                 else
                 $SUBIT "$JBOSS_CMD_STOP"
                 fi
                 ;;
                restart)
                 $0 stop
                 $0 start
                 ;;
                *)
                 echo "usage: $0 (start|stop|restart|help)"
                esac
                



                • 5. Re: Lost in the documentation UDP->TCP
                  brian.stansberry

                  Thanks, giddion for the correction and the details. Yep, in my comment format for the initial_hosts list was incorrect.

                  Note that if you use -b 0.0.0.0, JGroups will not bind to all interfaces. Rather it ends up using the value returned by InetAddress.getLocalHost().getHostName(). You can control this what interface is used by using -Djgroups.bind_addr=xxx.

                  • 6. Re: Lost in the documentation UDP->TCP

                    I know about the -b but the system only has one network interface so it's not an issue.

                    • 7. Re: Lost in the documentation UDP->TCP
                      matt.hibb

                      This post has been very useful in helping me configure clustering using Amazon EC2. I would just like to clarify a point regarding JBM and using TCP.

                      I have changed "ControlChannelName" from "jbm-control" to "tcp", but what about "DataChannelName" which uses "jbm-data"?
                      Looking at the jgroups stack config for "jbm-data", it to uses MPING. Does this need adjusting to use TCPPING as well?

                      Thanks

                      • 8. Re: Lost in the documentation UDP->TCP
                        brian.stansberry

                        Yes, if your environment can't support multicast for discovery you'll need to replace MPING with TCPPING. Something like:

                        <TCPPING timeout="3000"
                         initial_hosts="${jbm.data.tcpping.initial_hosts}"
                         port_range="1"
                         num_initial_members="3"/>


                        Key thing is don't reuse the jgroups.tcpping.initial_hosts system property as you'll need different ports (7900 instead of 7600).

                        • 9. Re: Lost in the documentation UDP->TCP

                          Hello:

                           

                          Great thread - very informative.  However, I am having trouble setting up two separate TCP channels in JBoss 5.1, one for jbm-control, and one for jbm-data.  The jbm-control one seems to work fine - it clusters properly.  However, when I try to form the jbm-data channel with TCP and TCPPING, it doesn't form properly, and hence jms messages are not being load balanced across the cluster.

                           

                          What is weird is that I took Brian's advice stated above and created a separate initial_hosts property for jbm-data with different port #'s (7900) separate from the 'tcp' channel. When the cluster comes up, the two TCP channels seem to 'hook up' to each other - in the jmx-console, if I go to the jboss.jgroups section, select the "JMS-CTRL,protocol=TCP", and invoke "print connections", it shows the two separate channels connecting to each other - this is definitely not good.  Also, when I view the "JMS-DATA,type=channel", the "viewAsString" property only shows one machine (in the "JMS-CTRL,type=channel" view it shows all nodes of the cluster properly).  For some reason it seems the TCPPING in the jbm-data channel is not doing what it should - instead it is pinging the host/port pairs in the jbm-control channel.

                           

                          Anyone out there have two separate TCP channels working correctly, TCPPING and all?   Thanks for any help.

                           

                           

                          Below is a snippet of my jgroups-channelfactory-stacks.xml file - pretty straightforward:

                           

                          <stack name="tcp">
                                  <config>
                                      <TCP
                                           singleton_name="tcp"
                                           start_port="7600"
                                          
                                           ...
                                          
                                           />
                                      <TCPPING timeout="3000"
                                               initial_hosts="host1[7600],host2[7600],host3[7600]"
                                               port_range="1"
                                               num_initial_members="3"/>
                                              
                                          ...
                                         
                                  </config>
                              </stack>
                             
                              <stack name="jbm-data">
                                  <config>
                                      <TCP singleton_name="jbm-data"
                                           start_port="7900"
                                          
                                           ...
                                          
                                           />
                                      <TCPPING timeout="3000"
                                               initial_hosts="host1[7900],host2[7900],host3[7900]"
                                               port_range="1"
                                               num_initial_members="3"/>
                                              
                                           ...
                                          
                                  </config>
                              </stack>

                          • 10. Re: Lost in the documentation UDP->TCP
                            dbschofield

                            Don't set the system property jgroups.tcpping.initial_hosts when trying to configure tcpping if you will have more than one channel using tcpping.  From what I understand this value will take precedence over the initial_hosts values in jgroups-channelfactory-stacks.xml.  The result is cross-talking between two channels that you probably didn't mean to happen.

                            • 11. Re: Lost in the documentation UDP->TCP
                              kmajeed

                              I am also tyring to setup a JBoss 5.0 Cluster  with JMS Topic deployed in  deploy-hasingleton (tcp unicast), my first instance starts  successfully but when  second node starts i am getting this exception,  one thing weird i have  noticed is that sometimes second node starts  smoothly, i dont get this  exception - i had to retry 4-5 times to start  the server. Can anyone  help me in this?

                               

                              Stacktrace:

                               

                              javax.jms.InvalidDestinationException: No such destination: JBossTopic[MyMessage] has it been deployed?
                                   at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegateInternal(ServerSessionEndpoint.java:1838)
                                   at org.jboss.jms.server.endpoint.ServerSessionEndpoint.createConsumerDelegate(ServerSessionEndpoint.java:252)
                                     at    org.jboss.jms.server.endpoint.advised.SessionAdvised.org$jboss$jms$server$endpoint$advised$SessionAdvised$createConsumerDelegate$aop(SessionAdvised.java:94)
                                   at

                               

                              Khurram

                              • 12. Re: Lost in the documentation UDP->TCP
                                akostadinov

                                For anybody interested, jgroups supports other discovery protocols like jdbc, GOSSIP, S3_PING and a couple others that can be used for automatic discovery of nodes. See "Using UDP without IP multicasting" in jgroups advanced docs for other info.