JBoss AS 7.1 Clustering behind Amazon ELB

[redmond007]

I'm in the process of setting up a JBoss 7.1 HA cluster in standalone mode using Amazon ELB as a load balancer.  I understand that Amazon EC2 does not support UDP multicast, so it's necessary to modify the jgroups default-stack to use something else for peer-to-peer communication, something like TCP unicast.

I tried to modify my jgroups configuration to use TCP unicast instead of UDP multicast by modifying the jgroups subsystem configuration in my standalone-ha.xml like

        <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcp">

clustering still does not seem to be working.  Do I need to do anything else to enable jgroups over TCP?  I have a couple of more general clustering related questions as well.  Namely- JBoss clustering documentation states that two nodes started on the same network will automatically form a cluster.  My EC2 instances are in different geographic regions, so I'm not sure if they consider themselves to be in the same virtual network or not - how can I be sure that they connect to each other?  Bela Ban mentioned a way to cluster geographically distributed servers in the HA cluster presentation but did not go into configuration specifics.  Is there a way to specify cluster members by IP in cluster config?  I found some ways to do this in older versions of JBoss but not in AS7.1.  Also how can I tell from server.log which nodes are connected to the cluster?  Thanks for any advice.

[kazaag]

Hi,

The default tcp stack configuration use the MPING discorvery protocole; it use multicast so won't work on EC2.  You should replace the MPING protocule by another discovery; look at the jGroups documentation to have a view on the different avalable protocoles: http://www.jgroups.org/manual-3.x/html/protlist.html#DiscoveryProtocols

In you case you can use TCPPING if the list of member are static or S3_PING if you need elastic cluster.

To know if a node join the cluster you should search for line contening "cluster view"; it will list the nodes of the cluseter the node is connected to.

[rhusar]

Francois is right. Also take a look at TCPGossip discovery protocol using GossipRouter appliance http://www.jgroups.org/manual-3.x/html/protlist.html#d0e4353

[redmond007]

Thanks for your help.  Our nodes have static IPs so we probably need to configure tcp as default transport protocol and use TCPPING as a discovery protocol.  I know in older versions of JBoss this is done via the JBOSS/server/CONFIG/deploy/cluster/jgroupschannelfactory.sar/META-INF/jgroups-channelfactorystacks.xml file - which is an xml configuration file that uses the same type of configuration syntax described in the jgroups wiki. 

I'm having trouble translating the way the jgroups wiki describes configuring the TCPPING discovery protocol

<TCP bind_port="7800" /> +
<TCPPING initial_hosts="HostA[7800],HostB[7800]" port_range="2"
         timeout="3000" num_initial_members="3" />

to the way jgroups configuration has been organized (with socket binding information externalized to a different section of the config) in JBoss 7.1.  I tried changing the default jgroups config to

        <subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcp">
            <stack name="udp">
                <transport type="UDP" socket-binding="jgroups-udp" diagnostics-socket-binding="jgroups-diagnostics"/>
                <protocol type="PING"/>
                <protocol type="MERGE2"/>
                <protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
                <protocol type="FD"/>
                <protocol type="VERIFY_SUSPECT"/>
                <protocol type="BARRIER"/>
                <protocol type="pbcast.NAKACK"/>
                <protocol type="UNICAST2"/>
                <protocol type="pbcast.STABLE"/>
                <protocol type="pbcast.GMS"/>
                <protocol type="UFC"/>
                <protocol type="MFC"/>
                <protocol type="FRAG2"/>
            </stack>
            <stack name="tcp">
                <transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>
                <protocol type="TCPPING" socket-binding="jgroups-tcpping"/>
                <protocol type="MERGE2"/>
                <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
                <protocol type="FD"/>
                <protocol type="VERIFY_SUSPECT"/>
                <protocol type="BARRIER"/>
                <protocol type="pbcast.NAKACK"/>
                <protocol type="UNICAST2"/>
                <protocol type="pbcast.STABLE"/>
                <protocol type="pbcast.GMS"/>
                <protocol type="UFC"/>
                <protocol type="MFC"/>
                <protocol type="FRAG2"/>
            </stack>
        </subsystem>

    <interfaces>
        <interface name="management">
            <inet-address value="${jboss.bind.address.management:10.211.46.9}"/>
        </interface>
        <interface name="public">
            <inet-address value="${jboss.bind.address:10.211.46.9}"/>
        </interface>
    </interfaces>


    <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="http" port="80"/>
        <socket-binding name="https" port="443"/>
        <socket-binding name="jgroups-diagnostics" port="0" multicast-address="224.0.75.75" multicast-port="7500"/>
        <socket-binding name="jgroups-tcpping" initial_hosts="10.80.38.10[7600],10.211.46.9[7600]" port_range="2" timeout="3000" num_initial_members="2"/>
        <socket-binding name="jgroups-tcp" port="7600"/>
        <socket-binding name="jgroups-tcp-fd" port="57600"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="54200"/>
        <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
        <socket-binding name="osgi-http" interface="management" port="8090"/>
        <socket-binding name="remoting" port="4447"/>
        <socket-binding name="txn-recovery-environment" port="4712"/>
        <socket-binding name="txn-status-manager" port="4713"/>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
    </socket-binding-group>

but JBoss is unable to start with this configuration - JBoss boot.log is complaining about the way I have configured the socket-binding for 'jgroups-tcpping'

(it says 'initial_hosts' is an unexpected attribute). 

10:21:23,602 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
10:21:24,023 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
10:21:24,086 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
10:21:24,086 DEBUG [org.jboss.as.config] Configured system properties:
          awt.toolkit = sun.awt.windows.WToolkit
          file.encoding = Cp1252
          file.encoding.pkg = sun.io
          file.separator = \
          java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
          java.awt.printerjob = sun.awt.windows.WPrinterJob
          java.class.path = D:\jboss-as-7.1.0.Final\jboss-modules.jar
          java.class.version = 50.0
          java.endorsed.dirs = C:\Program Files\Java\jdk1.6.0_29\jre\lib\endorsed
          java.ext.dirs = C:\Program Files\Java\jdk1.6.0_29\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
          java.home = C:\Program Files\Java\jdk1.6.0_29\jre
          java.io.tmpdir = D:/file_transfers/
          java.library.path = C:\Program Files\Java\jdk1.6.0_29\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;;.
          java.runtime.name = Java(TM) SE Runtime Environment
          java.runtime.version = 1.6.0_29-b11
          java.specification.name = Java Platform API Specification
          java.specification.vendor = Sun Microsystems Inc.
          java.specification.version = 1.6
          java.util.logging.manager = org.jboss.logmanager.LogManager
          java.vendor = Sun Microsystems Inc.
          java.vendor.url = http://java.sun.com/
          java.vendor.url.bug = http://java.sun.com/cgi-bin/bugreport.cgi
          java.version = 1.6.0_29
          java.vm.info = mixed mode
          java.vm.name = Java HotSpot(TM) 64-Bit Server VM
          java.vm.specification.name = Java Virtual Machine Specification
          java.vm.specification.vendor = Sun Microsystems Inc.
          java.vm.specification.version = 1.0
          java.vm.vendor = Sun Microsystems Inc.
          java.vm.version = 20.4-b02
          javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
          javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory
          javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory
          javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory
          javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory
          javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory
          javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory
          javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory
          javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory
          javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory
          jboss.home.dir = D:\jboss-as-7.1.0.Final
          jboss.host.name = ip-0ad32e4f
          jboss.modules.dir = D:\jboss-as-7.1.0.Final\modules
          jboss.modules.system.pkgs = org.jboss.byteman
          jboss.node.name = ip-0ad32e4f
          jboss.qualified.host.name = ip-0ad32e4f
          jboss.server.base.dir = D:\jboss-as-7.1.0.Final\standalone
          jboss.server.config.dir = D:\jboss-as-7.1.0.Final\standalone\configuration
          jboss.server.data.dir = D:\jboss-as-7.1.0.Final\standalone\data
          jboss.server.deploy.dir = D:\jboss-as-7.1.0.Final\standalone\data\content
          jboss.server.log.dir = D:\jboss-as-7.1.0.Final\standalone\log
          jboss.server.name = ip-0ad32e4f
          jboss.server.temp.dir = D:\jboss-as-7.1.0.Final\standalone\tmp
          line.separator = 


          logging.configuration = file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties
          module.path = D:\jboss-as-7.1.0.Final\modules
          org.apache.commons.logging.Log = org.apache.commons.logging.impl.NoOpLog
          org.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR = false
          org.dom4j.DocumentFactory.singleton.strategy = org.dom4j.util.PerThreadSingleton
          org.jboss.boot.log.file = D:\jboss-as-7.1.0.Final\standalone\log\boot.log
          org.jboss.resolver.warning = true
          org.xml.sax.driver = __redirected.__XMLReaderFactory
          os.arch = amd64
          os.name = Windows Server 2008 R2
          os.version = 6.1
          path.separator = ;
          program.name = standalone.bat
          sun.arch.data.model = 64
          sun.boot.class.path = C:\Program Files\Java\jdk1.6.0_29\jre\lib\resources.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\rt.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\sunrsasign.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\jce.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\modules\jdk.boot.jar;C:\Program Files\Java\jdk1.6.0_29\jre\classes
          sun.boot.library.path = C:\Program Files\Java\jdk1.6.0_29\jre\bin
          sun.cpu.endian = little
          sun.cpu.isalist = amd64
          sun.desktop = windows
          sun.io.unicode.encoding = UnicodeLittle
          sun.java.command = D:\jboss-as-7.1.0.Final\jboss-modules.jar -mp D:\jboss-as-7.1.0.Final\modules -logmodule org.jboss.logmanager -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=D:\jboss-as-7.1.0.Final -server-config standalone-ha.xml
          sun.java.launcher = SUN_STANDARD
          sun.jnu.encoding = Cp1252
          sun.management.compiler = HotSpot 64-Bit Tiered Compilers
          sun.os.patch.level = Service Pack 1
          sun.rmi.dgc.client.gcInterval = 3600000
          sun.rmi.dgc.server.gcInterval = 3600000
          user.country = US
          user.dir = D:\jboss-as-7.1.0.Final\bin
          user.home = C:\
          user.language = en
          user.name = IP-0AD32E4F$
          user.timezone = America/Chicago
          user.variant = 
10:21:24,101 DEBUG [org.jboss.as.config] VM Arguments: -Dprogram.name=standalone.bat -Xrs -Xms256M -Xmx768M -XX:MaxPermSize=512M -Dlogging.configuration=file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties -Dorg.jboss.boot.log.file=/opt/jboss-as-7.0.0.Final/standalone/log/boot.log -Djboss.home.dir=/opt/jboss-as-7.0.0.Final -Dorg.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR=false -Xss256k -Dorg.dom4j.DocumentFactory.singleton.strategy=org.dom4j.util.PerThreadSingleton -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog -XX:+UseParallelGC -Djava.io.tmpdir=D:/file_transfers/ -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.boot.log.file=D:\jboss-as-7.1.0.Final\standalone\log\boot.log -Dlogging.configuration=file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties 
10:21:25,334 ERROR [org.jboss.as.controller] JBAS014601: Error booting the container: java.lang.RuntimeException: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
          at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:161) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
          at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29]
Caused by: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
          at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:125) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:187) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.as.server.ServerService.boot(ServerService.java:261) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:155) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
          ... 1 more
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[402,9]
Message: JBAS014788: Unexpected attribute 'initial_hosts' encountered
          at org.jboss.as.controller.parsing.ParseUtils.unexpectedAttribute(ParseUtils.java:104) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.as.controller.parsing.CommonXml.parseSocketBinding(CommonXml.java:773) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.as.server.parsing.StandaloneXml.parseSocketBindingGroup_1_1(StandaloneXml.java:856) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.as.server.parsing.StandaloneXml.readServerElement_1_1(StandaloneXml.java:341) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:126) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:100) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
          at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
          at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
          at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:117) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
          ... 4 more


10:21:25,365 INFO  [org.jboss.as] JBAS015950: JBoss AS 7.1.0.Final "Thunder" stopped in 7ms
r

This seems to me the most logical way to configure the socket binding for TCPPING given the jgroups documentation.  Am I doing this correctly?  I'm not sure how else to try it.

[rhusar]

Something is wrong. I am trying to find out why.

[redmond007]

Thanks, we're not able to downgrade right now.  Is there another approach we can try?  or any timeframe on when TCPPING will be supported?

[rhusar]

Update: The TCPPING should be supported, sounds like a bug.

Hi Red,

as I mentioned earlier and I checked now TCP_GOSSIP is supported. (There is also an issue, the protocol in AS7 is TCP_GOSSIP but its supposed to be TCPGOSSIP in reality).

Rad

Message was edited by: Radoslav Husar

[rhusar]

        <socket-binding name="jgroups-tcpping" initial_hosts="10.80.38.10[7600],10.211.46.9[7600]" port_range="2" timeout="3000" num_initial_members="2"/>

I am looking at the post again carefully, the problem is here. The attribute initial_hosts is part of the PROTOCOL definition. socket-binding mechanism is only for specifing sockets.

So you need to fix to:

                <protocol type="TCPPING" initial_hosts="10.80.38.10[7600],10.211.46.9[7600]" port_range="2" timeout="3000" num_initial_members="2"/>

...or something very similar.

[redmond007]

Great.  How should my societ-binding for 'jgroups-tcpping' look?  should I leave port at '0' assign it the same port as tcp?

        <socket-binding name="jgroups-tcpping" port="7600"/>

or do I even need to set a socket-binding for tcpping?

[redmond007]

Fyi I just tried it again taking your advice and setting TCPPING properties in the protocol definition

<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcp">
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp" diagnostics-socket-binding="jgroups-diagnostics"/>
<protocol type="PING"/>
<protocol type="MERGE2"/>
<protocol type="FD_SOCK" socket-binding="jgroups-udp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="BARRIER"/>
<protocol type="pbcast.NAKACK"/>
<protocol type="UNICAST2"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>
<protocol type="TCPPING" initial_hosts="10.80.38.13[7600],10.211.46.9[7600]" port_range="2" timeout="3000" num_initial_members="2"/>
<protocol type="MERGE2"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="BARRIER"/>
<protocol type="pbcast.NAKACK"/>
<protocol type="UNICAST2"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>

I did not include a socket-binding config for tcpping.  JBoss is still unable to start - I get a message in boot.log that 'initial hosts' is an unexpected attribute.  Maybe I need a socket-binding config?

12:18:06,549 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
12:18:07,079 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
12:18:07,142 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
12:18:07,142 DEBUG [org.jboss.as.config] Configured system properties:
awt.toolkit = sun.awt.windows.WToolkit
file.encoding = Cp1252
file.encoding.pkg = sun.io
file.separator = \
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.awt.printerjob = sun.awt.windows.WPrinterJob
java.class.path = D:\jboss-as-7.1.0.Final\jboss-modules.jar
java.class.version = 50.0
java.endorsed.dirs = C:\Program Files\Java\jdk1.6.0_29\jre\lib\endorsed
java.ext.dirs = C:\Program Files\Java\jdk1.6.0_29\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
java.home = C:\Program Files\Java\jdk1.6.0_29\jre
java.io.tmpdir = D:/file_transfers/
java.library.path = C:\Program Files\Java\jdk1.6.0_29\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;;.
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_29-b11
java.specification.name = Java Platform API Specification
java.specification.vendor = Sun Microsystems Inc.
java.specification.version = 1.6
java.util.logging.manager = org.jboss.logmanager.LogManager
java.vendor = Sun Microsystems Inc.
java.vendor.url = http://java.sun.com/
java.vendor.url.bug = http://java.sun.com/cgi-bin/bugreport.cgi
java.version = 1.6.0_29
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Sun Microsystems Inc.
java.vm.specification.version = 1.0
java.vm.vendor = Sun Microsystems Inc.
java.vm.version = 20.4-b02
javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory
javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory
javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory
javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory
javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory
javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory
javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory
javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory
javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory
jboss.home.dir = D:\jboss-as-7.1.0.Final
jboss.host.name = ip-0ad32e4f
jboss.modules.dir = D:\jboss-as-7.1.0.Final\modules
jboss.modules.system.pkgs = org.jboss.byteman
jboss.node.name = ip-0ad32e4f
jboss.qualified.host.name = ip-0ad32e4f
jboss.server.base.dir = D:\jboss-as-7.1.0.Final\standalone
jboss.server.config.dir = D:\jboss-as-7.1.0.Final\standalone\configuration
jboss.server.data.dir = D:\jboss-as-7.1.0.Final\standalone\data
jboss.server.deploy.dir = D:\jboss-as-7.1.0.Final\standalone\data\content
jboss.server.log.dir = D:\jboss-as-7.1.0.Final\standalone\log
jboss.server.name = ip-0ad32e4f
jboss.server.temp.dir = D:\jboss-as-7.1.0.Final\standalone\tmp
line.separator = 
logging.configuration = file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties
module.path = D:\jboss-as-7.1.0.Final\modules
org.apache.commons.logging.Log = org.apache.commons.logging.impl.NoOpLog
org.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR = false
org.dom4j.DocumentFactory.singleton.strategy = org.dom4j.util.PerThreadSingleton
org.jboss.boot.log.file = D:\jboss-as-7.1.0.Final\standalone\log\boot.log
org.jboss.resolver.warning = true
org.xml.sax.driver = __redirected.__XMLReaderFactory
os.arch = amd64
os.name = Windows Server 2008 R2
os.version = 6.1
path.separator = ;
program.name = standalone.bat
sun.arch.data.model = 64
sun.boot.class.path = C:\Program Files\Java\jdk1.6.0_29\jre\lib\resources.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\rt.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\sunrsasign.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\jce.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\modules\jdk.boot.jar;C:\Program Files\Java\jdk1.6.0_29\jre\classes
sun.boot.library.path = C:\Program Files\Java\jdk1.6.0_29\jre\bin
sun.cpu.endian = little
sun.cpu.isalist = amd64
sun.desktop = windows
sun.io.unicode.encoding = UnicodeLittle
sun.java.command = D:\jboss-as-7.1.0.Final\jboss-modules.jar -mp D:\jboss-as-7.1.0.Final\modules -logmodule org.jboss.logmanager -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=D:\jboss-as-7.1.0.Final -server-config standalone-ha.xml
sun.java.launcher = SUN_STANDARD
sun.jnu.encoding = Cp1252
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
sun.os.patch.level = Service Pack 1
sun.rmi.dgc.client.gcInterval = 3600000
sun.rmi.dgc.server.gcInterval = 3600000
user.country = US
user.dir = D:\jboss-as-7.1.0.Final\bin
user.home = C:\
user.language = en
user.name = IP-0AD32E4F$
user.timezone = America/Chicago
user.variant = 
12:18:07,142 DEBUG [org.jboss.as.config] VM Arguments: -Dprogram.name=standalone.bat -Xrs -Xms256M -Xmx768M -XX:MaxPermSize=512M -Dlogging.configuration=file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties -Dorg.jboss.boot.log.file=/opt/jboss-as-7.0.0.Final/standalone/log/boot.log -Djboss.home.dir=/opt/jboss-as-7.0.0.Final -Dorg.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR=false -Xss256k -Dorg.dom4j.DocumentFactory.singleton.strategy=org.dom4j.util.PerThreadSingleton -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog -XX:+UseParallelGC -Djava.io.tmpdir=D:/file_transfers/ -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.boot.log.file=D:\jboss-as-7.1.0.Final\standalone\log\boot.log -Dlogging.configuration=file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties 
12:18:08,327 ERROR [org.jboss.as.controller] JBAS014601: Error booting the container: java.lang.RuntimeException: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:161) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29]
Caused by: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:125) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:187) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.ServerService.boot(ServerService.java:261) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:155) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
... 1 more
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[277,17]
Message: JBAS014788: Unexpected attribute 'initial_hosts' encountered
at org.jboss.as.controller.parsing.ParseUtils.unexpectedAttribute(ParseUtils.java:104) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.clustering.jgroups.subsystem.JGroupsSubsystemXMLReader_1_1.parseProtocol(JGroupsSubsystemXMLReader_1_1.java:283)
at org.jboss.as.clustering.jgroups.subsystem.JGroupsSubsystemXMLReader_1_1.parseStack(JGroupsSubsystemXMLReader_1_1.java:146)
at org.jboss.as.clustering.jgroups.subsystem.JGroupsSubsystemXMLReader_1_1.readElement(JGroupsSubsystemXMLReader_1_1.java:90)
at org.jboss.as.clustering.jgroups.subsystem.JGroupsSubsystemXMLReader_1_1.readElement(JGroupsSubsystemXMLReader_1_1.java:50)
at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.staxmapper.XMLExtendedStreamReaderImpl.handleAny(XMLExtendedStreamReaderImpl.java:69) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.parseServerProfile(StandaloneXml.java:893) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readServerElement_1_1(StandaloneXml.java:329) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:126) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:100) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:117) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
... 4 more
12:18:08,327 INFO  [org.jboss.as] JBAS015950: JBoss AS 7.1.0.Final "Thunder" stopped in 7ms

[kazaag]

You should put the protocol parameter in inner property node in the protocol node.

<protocol name="TCPPING">

     <property name="initial_hosts">

         10.80.38.13[7600],10.211.46.9[7600]

     </property>

     ...

</protocol>

On a slide note I don't think you need a port range.  It is used in the case there is several jGroups node on the same machine, then it scan in the port range for other jGroups node http://www.jgroups.org/manual-3.x/html/protlist.html#TCPPING_Prot .  If you put a port range to 2, it will scan port 7600, 7601 and 7602.  If you don't plan to run several jboss on the same machine (with port ofset of 1 or 2) put it to 0.

[redmond007]

Francois- thanks.  I still have problems though.  I tried taking your advice and formatting tcpping properties inside of inner property nodes nested inside the  property node.

<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcp">
<stack name="udp">
<transport type="UDP" socket-binding="jgroups-udp" diagnostics-socket-binding="jgroups-diagnostics"/>
<protocol type="BARRIER"/>
<protocol type="pbcast.NAKACK"/>
<protocol type="UNICAST2"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
<stack name="tcp">
<transport type="TCP" socket-binding="jgroups-tcp" diagnostics-socket-binding="jgroups-diagnostics"/>
<protocol type="TCPPING"/>
<property name="initial_hosts">
10.80.38.10[7600],10.211.46.9[7600]
</property>
<property name="port_range">
0
</property>
<property name="num_initial_members">
2
</property>
</protocol>
<protocol type="MERGE2"/>
<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
<protocol type="FD"/>
<protocol type="VERIFY_SUSPECT"/>
<protocol type="BARRIER"/>
<protocol type="pbcast.NAKACK"/>
<protocol type="UNICAST2"/>
<protocol type="pbcast.STABLE"/>
<protocol type="pbcast.GMS"/>
<protocol type="UFC"/>
<protocol type="MFC"/>
<protocol type="FRAG2"/>
</stack>
</subsystem>

JBoss is unable to start up - server.log is complaining that 'property' is an unexpected element.  Maybe I omitted something?

13:41:36,046 INFO  [org.jboss.modules] JBoss Modules version 1.1.1.GA
13:41:36,405 INFO  [org.jboss.msc] JBoss MSC version 1.0.2.GA
13:41:36,483 INFO  [org.jboss.as] JBAS015899: JBoss AS 7.1.0.Final "Thunder" starting
13:41:36,483 DEBUG [org.jboss.as.config] Configured system properties:
awt.toolkit = sun.awt.windows.WToolkit
file.encoding = Cp1252
file.encoding.pkg = sun.io
file.separator = \
java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
java.awt.printerjob = sun.awt.windows.WPrinterJob
java.class.path = D:\jboss-as-7.1.0.Final\jboss-modules.jar
java.class.version = 50.0
java.endorsed.dirs = C:\Program Files\Java\jdk1.6.0_29\jre\lib\endorsed
java.ext.dirs = C:\Program Files\Java\jdk1.6.0_29\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
java.home = C:\Program Files\Java\jdk1.6.0_29\jre
java.io.tmpdir = D:/file_transfers/
java.library.path = C:\Program Files\Java\jdk1.6.0_29\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;;.
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_29-b11
java.specification.name = Java Platform API Specification
java.specification.vendor = Sun Microsystems Inc.
java.specification.version = 1.6
java.util.logging.manager = org.jboss.logmanager.LogManager
java.vendor = Sun Microsystems Inc.
java.vendor.url = http://java.sun.com/
java.vendor.url.bug = http://java.sun.com/cgi-bin/bugreport.cgi
java.version = 1.6.0_29
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Sun Microsystems Inc.
java.vm.specification.version = 1.0
java.vm.vendor = Sun Microsystems Inc.
java.vm.version = 20.4-b02
javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
javax.xml.datatype.DatatypeFactory = __redirected.__DatatypeFactory
javax.xml.parsers.DocumentBuilderFactory = __redirected.__DocumentBuilderFactory
javax.xml.parsers.SAXParserFactory = __redirected.__SAXParserFactory
javax.xml.stream.XMLEventFactory = __redirected.__XMLEventFactory
javax.xml.stream.XMLInputFactory = __redirected.__XMLInputFactory
javax.xml.stream.XMLOutputFactory = __redirected.__XMLOutputFactory
javax.xml.transform.TransformerFactory = __redirected.__TransformerFactory
javax.xml.validation.SchemaFactory:http://www.w3.org/2001/XMLSchema = __redirected.__SchemaFactory
javax.xml.xpath.XPathFactory:http://java.sun.com/jaxp/xpath/dom = __redirected.__XPathFactory
jboss.home.dir = D:\jboss-as-7.1.0.Final
jboss.host.name = ip-0ad32e4f
jboss.modules.dir = D:\jboss-as-7.1.0.Final\modules
jboss.modules.system.pkgs = org.jboss.byteman
jboss.node.name = ip-0ad32e4f
jboss.qualified.host.name = ip-0ad32e4f
jboss.server.base.dir = D:\jboss-as-7.1.0.Final\standalone
jboss.server.config.dir = D:\jboss-as-7.1.0.Final\standalone\configuration
jboss.server.data.dir = D:\jboss-as-7.1.0.Final\standalone\data
jboss.server.deploy.dir = D:\jboss-as-7.1.0.Final\standalone\data\content
jboss.server.log.dir = D:\jboss-as-7.1.0.Final\standalone\log
jboss.server.name = ip-0ad32e4f
jboss.server.temp.dir = D:\jboss-as-7.1.0.Final\standalone\tmp
line.separator = 
logging.configuration = file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties
module.path = D:\jboss-as-7.1.0.Final\modules
org.apache.commons.logging.Log = org.apache.commons.logging.impl.NoOpLog
org.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR = false
org.dom4j.DocumentFactory.singleton.strategy = org.dom4j.util.PerThreadSingleton
org.jboss.boot.log.file = D:\jboss-as-7.1.0.Final\standalone\log\boot.log
org.jboss.resolver.warning = true
org.xml.sax.driver = __redirected.__XMLReaderFactory
os.arch = amd64
os.name = Windows Server 2008 R2
os.version = 6.1
path.separator = ;
program.name = standalone.bat
sun.arch.data.model = 64
sun.boot.class.path = C:\Program Files\Java\jdk1.6.0_29\jre\lib\resources.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\rt.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\sunrsasign.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\jce.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.6.0_29\jre\lib\modules\jdk.boot.jar;C:\Program Files\Java\jdk1.6.0_29\jre\classes
sun.boot.library.path = C:\Program Files\Java\jdk1.6.0_29\jre\bin
sun.cpu.endian = little
sun.cpu.isalist = amd64
sun.desktop = windows
sun.io.unicode.encoding = UnicodeLittle
sun.java.command = D:\jboss-as-7.1.0.Final\jboss-modules.jar -mp D:\jboss-as-7.1.0.Final\modules -logmodule org.jboss.logmanager -jaxpmodule javax.xml.jaxp-provider org.jboss.as.standalone -Djboss.home.dir=D:\jboss-as-7.1.0.Final -server-config standalone-ha.xml
sun.java.launcher = SUN_STANDARD
sun.jnu.encoding = Cp1252
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
sun.os.patch.level = Service Pack 1
sun.rmi.dgc.client.gcInterval = 3600000
sun.rmi.dgc.server.gcInterval = 3600000
user.country = US
user.dir = D:\jboss-as-7.1.0.Final\bin
user.home = C:\
user.language = en
user.name = IP-0AD32E4F$
user.timezone = America/Chicago
user.variant = 
13:41:36,545 DEBUG [org.jboss.as.config] VM Arguments: -Dprogram.name=standalone.bat -Xrs -Xms256M -Xmx768M -XX:MaxPermSize=512M -Dlogging.configuration=file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties -Dorg.jboss.boot.log.file=/opt/jboss-as-7.0.0.Final/standalone/log/boot.log -Djboss.home.dir=/opt/jboss-as-7.0.0.Final -Dorg.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR=false -Xss256k -Dorg.dom4j.DocumentFactory.singleton.strategy=org.dom4j.util.PerThreadSingleton -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.NoOpLog -XX:+UseParallelGC -Djava.io.tmpdir=D:/file_transfers/ -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Djboss.modules.system.pkgs=org.jboss.byteman -Dorg.jboss.boot.log.file=D:\jboss-as-7.1.0.Final\standalone\log\boot.log -Dlogging.configuration=file:D:\jboss-as-7.1.0.Final/standalone/configuration/logging.properties 
13:41:37,793 ERROR [org.jboss.as.controller] JBAS014601: Error booting the container: java.lang.RuntimeException: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:161) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_29]
Caused by: org.jboss.as.controller.persistence.ConfigurationPersistenceException: JBAS014676: Failed to parse configuration
at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:125) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:187) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.ServerService.boot(ServerService.java:261) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:155) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
... 1 more
Caused by: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[278,5]
Message: JBAS014789: Unexpected element '{urn:jboss:domain:jgroups:1.1}property' encountered
at org.jboss.as.controller.parsing.ParseUtils.unexpectedElement(ParseUtils.java:85) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.clustering.jgroups.subsystem.JGroupsSubsystemXMLReader_1_1.parseStack(JGroupsSubsystemXMLReader_1_1.java:150)
at org.jboss.as.clustering.jgroups.subsystem.JGroupsSubsystemXMLReader_1_1.readElement(JGroupsSubsystemXMLReader_1_1.java:90)
at org.jboss.as.clustering.jgroups.subsystem.JGroupsSubsystemXMLReader_1_1.readElement(JGroupsSubsystemXMLReader_1_1.java:50)
at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.staxmapper.XMLExtendedStreamReaderImpl.handleAny(XMLExtendedStreamReaderImpl.java:69) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.parseServerProfile(StandaloneXml.java:893) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readServerElement_1_1(StandaloneXml.java:329) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:126) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.as.server.parsing.StandaloneXml.readElement(StandaloneXml.java:100) [jboss-as-server-7.1.0.Final.jar:7.1.0.Final]
at org.jboss.staxmapper.XMLMapperImpl.processNested(XMLMapperImpl.java:110) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.staxmapper.XMLMapperImpl.parseDocument(XMLMapperImpl.java:69) [staxmapper-1.1.0.Final.jar:1.1.0.Final]
at org.jboss.as.controller.persistence.XmlConfigurationPersister.load(XmlConfigurationPersister.java:117) [jboss-as-controller-7.1.0.Final.jar:7.1.0.Final]
... 4 more
13:41:37,809 INFO  [org.jboss.as] JBAS015950: JBoss AS 7.1.0.Final "Thunder" stopped in 7ms

[kazaag]

You forget to remove the / in the oppening protocol tag:

<protocol type="TCPPING"/>
...
</protocol>

[redmond007]

Fantastic - it is working.  THank you both for your advice!