Hi, Jay.
Thanks for the response - I really appreciate it! I am running the host and domain controllers on the same box (not really by design, but because the instructions referenced above set it up that way). I went ahead and tried your first suggestion anyway, and got an error where the host controller couldn't connect. When I tried your second suggestion, I got my same old error. Here's the ps output:
jboss 10797 10794 0 19:50 ? 00:00:00 /bin/sh /usr/local/jboss/bin/domain.sh -Djboss.bind.address.management=localhost -c domain.xml
jboss 10847 10797 0 19:50 ? 00:00:03 java -D[Process Controller] -server -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -Dorg.jboss.boot.log.file=/usr/local/jboss/domain/log/process-controller.log -Dlogging.configuration=file:/usr/local/jboss/domain/configuration/logging.properties -jar /usr/local/jboss/jboss-modules.jar -mp /usr/local/jboss/modules org.jboss.as.process-controller -jboss-home /usr/local/jboss -jvm java -mp /usr/local/jboss/modules -- -Dorg.jboss.boot.log.file=/usr/local/jboss/domain/log/host-controller.log -Dlogging.configuration=file:/usr/local/jboss/domain/configuration/logging.properties -server -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -- -default-jvm java -Djboss.bind.address.management=localhost -c domain.xml
jboss 10863 10847 0 19:50 ? 00:00:19 java -D[Host Controller] -Dorg.jboss.boot.log.file=/usr/local/jboss/domain/log/host-controller.log -Dlogging.configuration=file:/usr/local/jboss/domain/configuration/logging.properties -server -Xms64m -Xmx512m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.domain.default.config=domain.xml -Djboss.host.default.config=host.xml -jar /usr/local/jboss/jboss-modules.jar -mp /usr/local/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /usr/local/jboss/modules --pc-address localhost.localdomain --pc-port 42201 -default-jvm java -Djboss.bind.address.management=localhost -c domain.xml -Djboss.home.dir=/usr/local/jboss
jboss 10939 10847 1 19:50 ? 00:00:59 /usr/java/jdk1.7.0_05/jre/bin/java -D[Server:server-one] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms64m -Xmx512m -server -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djboss.host.default.config=host.xml -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.domain.default.config=domain.xml -Djava.net.preferIPv4Stack=true -D[Host Controller]=true -Djboss.bind.address.management=localhost -Dcom.m2mci.config.dir=/usr/local/etc/Nephele -Djboss.home.dir=/usr/local/jboss -Djboss.server.log.dir=/usr/local/jboss/domain/servers/server-one/log -Djboss.server.temp.dir=/usr/local/jboss/domain/servers/server-one/tmp -Djboss.server.data.dir=/usr/local/jboss/domain/servers/server-one/data -Dorg.jboss.boot.log.file=/usr/local/jboss/domain/servers/server-one/log/boot.log -Dlogging.configuration=file:/usr/local/jboss/domain/configuration/logging.properties -jar /usr/local/jboss/jboss-modules.jar -mp /usr/local/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server
jboss 10944 10847 1 19:50 ? 00:00:58 /usr/java/jdk1.7.0_05/jre/bin/java -D[Server:server-two] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms64m -Xmx512m -server -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djboss.host.default.config=host.xml -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.domain.default.config=domain.xml -Djava.net.preferIPv4Stack=true -D[Host Controller]=true -Djboss.bind.address.management=localhost -Dcom.m2mci.config.dir=/usr/local/etc/Nephele -Djboss.home.dir=/usr/local/jboss -Djboss.server.log.dir=/usr/local/jboss/domain/servers/server-two/log -Djboss.server.temp.dir=/usr/local/jboss/domain/servers/server-two/tmp -Djboss.server.data.dir=/usr/local/jboss/domain/servers/server-two/data -Dorg.jboss.boot.log.file=/usr/local/jboss/domain/servers/server-two/log/boot.log -Dlogging.configuration=file:/usr/local/jboss/domain/configuration/logging.properties -jar /usr/local/jboss/jboss-modules.jar -mp /usr/local/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server
jboss 10964 10847 0 19:50 ? 00:00:16 /usr/java/jdk1.7.0_05/jre/bin/java -D[Server:server-three] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms64m -Xmx512m -server -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.awt.headless=true -Djboss.host.default.config=host.xml -Djboss.modules.system.pkgs=org.jboss.byteman -Djboss.domain.default.config=domain.xml -Djava.net.preferIPv4Stack=true -D[Host Controller]=true -Djboss.bind.address.management=localhost -Dcom.m2mci.config.dir=/usr/local/etc/Nephele -Djboss.home.dir=/usr/local/jboss -Djboss.server.log.dir=/usr/local/jboss/domain/servers/server-three/log -Djboss.server.temp.dir=/usr/local/jboss/domain/servers/server-three/tmp -Djboss.server.data.dir=/usr/local/jboss/domain/servers/server-three/data -Dorg.jboss.boot.log.file=/usr/local/jboss/domain/servers/server-three/log/boot.log -Dlogging.configuration=file:/usr/local/jboss/domain/configuration/logging.properties -jar /usr/local/jboss/jboss-modules.jar -mp /usr/local/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server
And the output:
19:50:51,057 INFO [org.jboss.as.process.Server:server-three.status] (ProcessController-threads - 4) JBAS012017: Starting process 'Server:server-three'
[Server:server-three] 19:50:56,840 INFO [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Server:server-one] 19:50:57,937 INFO [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Server:server-two] 19:50:59,726 INFO [org.jboss.modules] (main) JBoss Modules version 1.1.1.GA
[Server:server-three] 19:51:01,130 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Server:server-three] 19:51:02,297 INFO [org.jboss.as] (MSC service thread 1-1) JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
[Server:server-one] 19:51:02,614 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Server:server-one] 19:51:03,856 INFO [org.jboss.as] (MSC service thread 1-2) JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
[Server:server-three] 19:51:04,178 INFO [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.3.GA
[Server:server-three] 19:51:04,371 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.3.GA
[Server:server-three] 19:51:04,507 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.2.3.GA
[Server:server-two] 19:51:05,395 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.2.GA
[Server:server-one] 19:51:07,105 INFO [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.3.GA
[Server:server-two] 19:51:07,486 INFO [org.jboss.as] (MSC service thread 1-1) JBAS015899: JBoss AS 7.1.1.Final "Brontes" starting
[Server:server-one] 19:51:07,927 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.3.GA
[Server:server-one] 19:51:09,378 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.2.3.GA
[Server:server-two] 19:51:10,967 INFO [org.xnio] (MSC service thread 1-1) XNIO Version 3.0.3.GA
[Server:server-two] 19:51:11,208 INFO [org.xnio.nio] (MSC service thread 1-1) XNIO NIO Implementation Version 3.0.3.GA
[Server:server-two] 19:51:11,356 INFO [org.jboss.remoting] (MSC service thread 1-1) JBoss Remoting version 3.2.3.GA
[Server:server-three] 19:51:26,014 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC00001: Failed to start service jboss.host.controller.client: org.jboss.msc.service.StartException in service jboss.host.controller.client: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
[Server:server-three] at org.jboss.as.server.mgmt.domain.HostControllerServerClient.start(HostControllerServerClient.java:161) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
[Server:server-three] at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
[Server:server-three] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
[Server:server-three] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_05]
[Server:server-three] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_05]
[Server:server-three] at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_05]
[Server:server-three] Caused by: java.net.ConnectException: JBAS012144: Could not connect to remote://localhost:9999. The connection timed out
[Server:server-three] at org.jboss.as.protocol.ProtocolChannelClient.connectSync(ProtocolChannelClient.java:155) [jboss-as-protocol-7.1.1.Final.jar:7.1.1.Final]
[Server:server-three] at org.jboss.as.server.mgmt.domain.HostControllerServerConnection.openChannel(HostControllerServerConnection.java:158) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
[Server:server-three] at org.jboss.as.server.mgmt.domain.HostControllerServerConnection.connect(HostControllerServerConnection.java:86) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
[Server:server-three] at org.jboss.as.server.mgmt.domain.HostControllerServerClient.start(HostControllerServerClient.java:135) [jboss-as-server-7.1.1.Final.jar:7.1.1.Final]
[Server:server-three] ... 5 more
[Server:server-three]
And, like before, if I restart it, it pretty much randomly errors out on servers one, two, or three (or multiple of them). I usually have to restart it 10 or 15 times to get them all to come up correctly.
I don't think it's anything like having the wrong interface configured, because it would probably never connect at all, rather than connecting randomly. This really feels like a race condition of some sort.
I don't know if this matters as well, but I'm running this on an Amazon EC2 instance running Fedora 17. UDP multicast doesn't work there, but I got the JGroups stuff working using a GossipRouter, and that all seems to work correctly. The EC2 instance I'm running on is pretty low-powered (we're just testing this configuration for now, we'll probably beef it up later).
I'm going ahead and posting my domain.xml and host.xml files, hopefully there's something glaringly dopey I'm doing in there to cause this.
Thanks again.
-Jack