Restart JBoss Domain Controller and JBREM000200: Remote connection failed

[jorsol]

I have a Domain Controller in one server and various slave hosts on diferent servers, if I restart the Domain Controller then all the slave Host Controllers can't connect again:

[Host Controller] 15:31:29,029 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-master:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000201: Received invalid message on Remoting connection 26f29218 to /192.168.56.99:57823

[Host Controller] 15:31:29,033 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-master:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000201: Received invalid message on Remoting connection 44a47a56 to /192.168.56.97:49686

[Host Controller] 15:34:34,543 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-master:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000201: Received invalid message on Remoting connection 0111b863 to /192.168.56.101:37882

Is this a bug? or some kind of limitation for the cluster?

[jaikiran]

Which exact version of JBoss EAP is this (assuming you are indeed using JBoss EAP)? How are you restarting the domain controller?

[jorsol]

JBoss EAP 6.1.1 using a kill 15 TERM signal to the PID and tested it using jboss-cli shutdown.

I have to stop all the hosts including the master, and then start first the master and later the host to be able to gain control.

[jorsol]

Any news about this?

[wdfink]

If you do the test with one slave only and add trace level for org.jboss.remoting could you figure out the differences between a sucessful and failed connect?

[jorsol]

If I have 2 or more host controllers then they can't connect again to de domain controller, when I have only one it connects succesfully.

[wdfink]

If I use "kill -s TERM <pid>" to kill the host-controller process, it will be imediately restarted from the process controller and all slaves are successful reconected.

If I stop the domain (I start  from a shell) by Ctrl-C and start it again it may take a few seconds but the slave hosts are reconnecting as well.

I'm not sure what might be wrong in your environment.

Could you share the details, start commands and domain config you use?

I simple use a EAP6.1.1 copy it to different locations/machines, use the host -master -slave.xml configure one DC and add 3 slaves (without real servers)

[jorsol]

Ok, I have a single domain-controller and start it with with ./jboss/bin/domain.sh --domain-config=domain.xml --host-config=host-master.xml

the content of host-master.xml points the dc to local, the name of the host controller is master.

<domain-controller>

<local/>

</domain-controller>

And 4 Host Controllers that practically have the same configuration, they are named like host1, host2, host3, host4

./jboss/bin/domain.sh --domain-config=domain.xml --host-config=host-slave.xml

The content of host-slave.xml I have something like this:

<security-realm name="SlaveRealm">

<server-identities>

<secret value="Hashed###"/>

</server-identities>

</security-realm>

<domain-controller>

<remote host="${jboss.domain.master.address:192.168.2.96}" port="${jboss.domain.master.port:9999}" security-realm="SlaveRealm" username="slaveuser"/>

</domain-controller>

In the DC created a Management User called slaveuser wich is used to connect the HC to the DC.

I start the master (DC) first, then start host1, host2, host3, host4, they connect successfully to the master, but if I restart (stop and start) the DC then all the HC get lost and can't connect again to the DC.

The HC sends this then the DC is restarted:

In the first scenario the the DC is restarted I have the following log:

[Host Controller] 11:09:10,914 WARN  [org.jboss.as.host.controller] (Remoting "jboss-server4:MANAGEMENT" read-1) JBAS010914: Conexión cerrada al controlador host remoto. Tratando de re-conectarse.

[Host Controller] 11:10:51,876 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 2613a550 to 192.168.2.96/172.16.2.96:9999

[Server:host4-server1] 2013-10-28 11:11:29,567 WARN  [org.jgroups.protocols.TP$ProtocolAdapter] (Incoming-6,shared=udp) dropping unicast message to wrong destination host3:host3-server1/web; my local_addr is host4:host4-server1/web

[Server:host4-server1] 2013-10-28 11:11:29,567 WARN  [org.jgroups.protocols.TP$ProtocolAdapter] (Incoming-7,shared=udp) dropping unicast message to wrong destination host1:host1-server1/web; my local_addr is host4:host4-server1/web

[Server:host4-server1] 2013-10-28 11:11:29,567 WARN  [org.jgroups.protocols.TP$ProtocolAdapter] (Incoming-5,shared=udp) dropping unicast message to wrong destination host2:host2-server1/web; my local_addr is host4:host4-server1/web

[Host Controller] 11:13:24,466 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

[Host Controller] 11:13:24,475 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

[Host Controller] 11:13:24,477 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

[Host Controller] 11:13:24,552 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 547735be to 192.168.2.96/192.168.2.96:9999

In a second scenario, if I start the master, then host1, and then restart the master (DC) then the host1 connects successfully to the master, if start host2 to 4 then all connect fine.

I hope this help to identify the problem.

[wdfink]

Could you attach the domain.xml and the different host-*.xml (you need to edit the comment for that)

Also how the environment look like, I suppose you have different machines, what IP you use and what kind of OS?

[jorsol]

The enviroment is 5 virtual machines running on VMware ESXi 5.1 with Ubuntu Server 12.04 LTS as gest OS,

The IP of the VMs are:

192.168.2.96 - master

192.168.2.97 - host1

192.168.2.98 - host2

192.168.2.99 - host3

192.168.2.100 - host4

The master starts with host-master.xml and the slaves with host-slave.xml

the diferences with the slaves are simply the name (host1, host2, host3, host4) and the servers they contain are called host1-server1, host2-server1, host3-server1, host4-server1...

In another VM only have Apache 2.2 with mod_cluster with ip 192.168.2.95

[clichybi]

Hi,

as I face a similar issue, I like ask if you have opened a JIRA to get this fixed.

Cheers,

Carsten