10 Replies Latest reply on Jul 4, 2014 5:37 AM by Carsten Lichy-Bittendorf

    Restart JBoss Domain Controller and JBREM000200: Remote connection failed

    Jorge Solorzano Newbie

      I have a Domain Controller in one server and various slave hosts on diferent servers, if I restart the Domain Controller then all the slave Host Controllers can't connect again:

       

      [Host Controller] 15:31:29,029 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-master:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000201: Received invalid message on Remoting connection 26f29218 to /192.168.56.99:57823

      [Host Controller] 15:31:29,033 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-master:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000201: Received invalid message on Remoting connection 44a47a56 to /192.168.56.97:49686

      [Host Controller] 15:34:34,543 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-master:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000201: Received invalid message on Remoting connection 0111b863 to /192.168.56.101:37882

       

      Is this a bug? or some kind of limitation for the cluster?

        • 1. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
          jaikiran pai Master

          Which exact version of JBoss EAP is this (assuming you are indeed using JBoss EAP)? How are you restarting the domain controller?

          • 2. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
            Jorge Solorzano Newbie

            JBoss EAP 6.1.1 using a kill 15 TERM signal to the PID and tested it using jboss-cli shutdown.

             

            I have to stop all the hosts including the master, and then start first the master and later the host to be able to gain control.

            • 4. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
              Wolf-Dieter Fink Master

              If you do the test with one slave only and add trace level for org.jboss.remoting could you figure out the differences between a sucessful and failed connect?

              • 5. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
                Jorge Solorzano Newbie

                If I have 2 or more host controllers then they can't connect again to de domain controller, when I have only one it connects succesfully.

                • 6. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
                  Wolf-Dieter Fink Master

                  If I use "kill -s TERM <pid>" to kill the host-controller process, it will be imediately restarted from the process controller and all slaves are successful reconected.

                  If I stop the domain (I start  from a shell) by Ctrl-C and start it again it may take a few seconds but the slave hosts are reconnecting as well.

                  I'm not sure what might be wrong in your environment.

                  Could you share the details, start commands and domain config you use?

                  I simple use a EAP6.1.1 copy it to different locations/machines, use the host -master -slave.xml configure one DC and add 3 slaves (without real servers)

                  • 7. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
                    Jorge Solorzano Newbie

                    Ok, I have a single domain-controller and start it with with ./jboss/bin/domain.sh --domain-config=domain.xml --host-config=host-master.xml

                    the content of host-master.xml points the dc to local, the name of the host controller is master.

                    <domain-controller>
                    <local/>
                    </domain-controller>

                     

                    And 4 Host Controllers that practically have the same configuration, they are named like host1, host2, host3, host4

                    ./jboss/bin/domain.sh --domain-config=domain.xml --host-config=host-slave.xml

                    The content of host-slave.xml I have something like this:

                     

                    <security-realm name="SlaveRealm">
                    <server-identities>
                    <secret value="Hashed###"/>
                    </server-identities>
                    </security-realm>
                    <domain-controller>
                    <remote host="${jboss.domain.master.address:192.168.2.96}" port="${jboss.domain.master.port:9999}" security-realm="SlaveRealm" username="slaveuser"/>
                    </domain-controller>

                     

                    In the DC created a Management User called slaveuser wich is used to connect the HC to the DC.

                     

                    I start the master (DC) first, then start host1, host2, host3, host4, they connect successfully to the master, but if I restart (stop and start) the DC then all the HC get lost and can't connect again to the DC.

                     

                    The HC sends this then the DC is restarted:

                    In the first scenario the the DC is restarted I have the following log:

                    [Host Controller] 11:09:10,914 WARN  [org.jboss.as.host.controller] (Remoting "jboss-server4:MANAGEMENT" read-1) JBAS010914: Conexión cerrada al controlador host remoto. Tratando de re-conectarse.

                    [Host Controller] 11:10:51,876 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 2613a550 to 192.168.2.96/172.16.2.96:9999

                    [Server:host4-server1] 2013-10-28 11:11:29,567 WARN  [org.jgroups.protocols.TP$ProtocolAdapter] (Incoming-6,shared=udp) dropping unicast message to wrong destination host3:host3-server1/web; my local_addr is host4:host4-server1/web

                    [Server:host4-server1] 2013-10-28 11:11:29,567 WARN  [org.jgroups.protocols.TP$ProtocolAdapter] (Incoming-7,shared=udp) dropping unicast message to wrong destination host1:host1-server1/web; my local_addr is host4:host4-server1/web

                    [Server:host4-server1] 2013-10-28 11:11:29,567 WARN  [org.jgroups.protocols.TP$ProtocolAdapter] (Incoming-5,shared=udp) dropping unicast message to wrong destination host2:host2-server1/web; my local_addr is host4:host4-server1/web

                    [Host Controller] 11:13:24,466 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

                    [Host Controller] 11:13:24,475 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

                    [Host Controller] 11:13:24,477 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer

                    [Host Controller] 11:13:24,552 ERROR [org.jboss.remoting.remote.connection] (Remoting "jboss-server4:MANAGEMENT" read-1) JBREM000200: Remote connection failed: java.io.IOException: JBREM000202: Abrupt close on Remoting connection 547735be to 192.168.2.96/192.168.2.96:9999

                     

                    In a second scenario, if I start the master, then host1, and then restart the master (DC) then the host1 connects successfully to the master, if start host2 to 4 then all connect fine.

                     

                    I hope this help to identify the problem.

                    • 8. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
                      Wolf-Dieter Fink Master

                      Could you attach the domain.xml and the different host-*.xml (you need to edit the comment for that)

                      Also how the environment look like, I suppose you have different machines, what IP you use and what kind of OS?

                      • 9. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
                        Jorge Solorzano Newbie

                        The enviroment is 5 virtual machines running on VMware ESXi 5.1 with Ubuntu Server 12.04 LTS as gest OS,

                         

                        The IP of the VMs are:

                        192.168.2.96 - master

                        192.168.2.97 - host1

                        192.168.2.98 - host2

                        192.168.2.99 - host3

                        192.168.2.100 - host4

                         

                        The master starts with host-master.xml and the slaves with host-slave.xml

                        the diferences with the slaves are simply the name (host1, host2, host3, host4) and the servers they contain are called host1-server1, host2-server1, host3-server1, host4-server1...

                         

                        In another VM only have Apache 2.2 with mod_cluster with ip 192.168.2.95

                        • 10. Re: Restart JBoss Domain Controller and JBREM000200: Remote connection failed
                          Carsten Lichy-Bittendorf Newbie

                          Hi,

                           

                          as I face a similar issue, I like ask if you have opened a JIRA to get this fixed.

                           

                          Cheers,

                           

                          Carsten