7 Replies Latest reply on May 26, 2014 9:32 AM by vandamo

    Domain behavior - Server Nodes down when Host Controller dies

    vandamo

      Hi,

      We're currently testing out JBoss EAP 6.2 in domain mode in the context of migrating our Applications currently running either with Jetty or WebSphere Application Server 7.

       

      During our test, we noticed that if the Host Controller dies for one reason or another, the server nodes he manages are dying also.

      Same behavior if the Process Controller dies.

       

      Is this dependency a requirement for proper management of the server nodes, or is it possible to sever this dependency?

       

      Depending on this, the general domain structure we are going to setup will be different.

      Typically if we cannot avoid this behavior, we'd go for multiple Host Controllers per linux vm, to ensure that failure at Host Controller level related to ??? for Project A cannot result in failure of processes for Project B.

      While if it can be avoided, we'd go with one host controller per box.

       

      Any input or links related to the subject, would be greatly appreciated.

        • 1. Re: Domain behavior - Server Nodes down when Host Controller dies
          wdfink

          Not sure what you do.

          Normally the HC will be restarted from the ProcessController.

          The server instances are keep running and handle application requests.

           

          Which version do you use exact and how the HC die, did you kill it?

          • 2. Re: Re: Domain behavior - Server Nodes down when Host Controller dies
            vandamo

            I'm not the one who performed the installation of JBoss nor its domain configuration, so it is possible that something in the setup could cause this,

            although I find it unlikely (as the server groups and server nodes are created through the admin interface)


            OS: Red Hat Enterprise Linux Server release 6.5 (Santiago)

            JBoss:

                product-version=6.2.1.GA

                release-codename=Janus

                management-major-version=1

                release-version=7.3.1.Final-redhat-3

             

            I indeed did a kill of the process (see below)

             

            Sorry for the fairly verbose information, here the original running processes, the kill performed, and the new resulting processes, key for me being the node that is also a new process:

             

             

            Test Server Node - Host Controller Dependency

            jbossusr    16477    1  0 May16 ?        00:00:00 /bin/sh /apps/jbossEap/6.2.0/bin/domain.sh -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2/domain -Djboss.domain.config.dir=/data/DevDomain2/domain/configuration --domain-config=domain.xml --host-config=host-master.xml -bmanagement linux1 -Djboss.management.native.port=15013 -Djboss.management.http.port=15011

            jbossusr    16609 16477  0 May16 ?        00:10:04 /apps/java/jdk1.7.0_25/bin/java -D[Process Controller] -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.boot.log.file=/data/DevDomain2/domain/log/process-controller.log -Dlogging.configuration=file:/data/DevDomain2/domain/configuration/logging.properties -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules org.jboss.as.process-controller -jboss-home /apps/jbossEap/6.2.0 -jvm /apps/java/jdk1.7.0_25//bin/java -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -- -Dorg.jboss.boot.log.file=/data/DevDomain2/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -- -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2/domain -Djboss.domain.config.dir=/data/DevDomain2/domain/configuration --domain-config=domain.xml --host-config=host-master.xml -bmanagement linux1 -Djboss.management.native.port=15013 -Djboss.management.http.port=15011

            jbossusr    16634 16609  0 May16 ?        00:15:51 /apps/java/jdk1.7.0_25/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/data/DevDomain2/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules --pc-address 127.0.0.1 --pc-port 36612 -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2/domain -Djboss.domain.config.dir=/data/DevDomain2/domain/configuration --domain-config=domain.xml --host-config=host-master.xml -bmanagement linux1 -Djboss.management.native.port=15013 -Djboss.management.http.port=15011 -Djboss.home.dir=/apps/jbossEap/6.2.0

             

             

            jbossusr      727    1  0 May21 ?        00:00:00 /bin/sh /apps/jbossEap/6.2.0/bin/domain.sh -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=linux1 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 --host-config=host-slave.xml -Djboss.node.name=DevDomain2HostController1 -b localhost --backup

            jbossusr      874  727  0 May21 ?        00:04:48 /apps/java/jdk1.7.0_25/bin/java -D[Process Controller] -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.boot.log.file=/data/DevDomain2HostController1/domain/log/process-controller.log -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/configuration/logging.properties -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules org.jboss.as.process-controller -jboss-home /apps/jbossEap/6.2.0 -jvm /apps/java/jdk1.7.0_25//bin/java -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -- -Dorg.jboss.boot.log.file=/data/DevDomain2HostController1/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -- -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=linux1 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 --host-config=host-slave.xml -Djboss.node.name=DevDomain2HostController1 -b localhost --backup

            jbossusr      928  874  0 May21 ?        00:06:27 /apps/java/jdk1.7.0_25/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/data/DevDomain2HostController1/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules --pc-address 127.0.0.1 --pc-port 52370 -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=linux1 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 --host-config=host-slave.xml -Djboss.node.name=DevDomain2HostController1 -b localhost --backup -Djboss.home.dir=/apps/jbossEap/6.2.0

             

             

            jbossusr    18328  874  2 10:34 ?        00:00:53 /apps/java/jdk1.7.0_25/bin/java -D[Server:project2_node1] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms512m -Xmx512m -Djboss.bind.address=localhost -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=linux1 -DLOG_DIR=/data/logs -Djava.awt.headless=true -Djboss.node.name=DevDomain2HostController1 -Djboss.modules.system.pkgs=org.jboss.byteman -Dspring.profiles.active=jndi-mq,stpmock,jboss -Djava.net.preferIPv4Stack=true -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 -Djboss.server.log.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/log -Djboss.server.temp.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/tmp -Djboss.server.data.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/data -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/servers/project2_node1/data/logging.properties -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server

             

             

            --KILL Host Controller--------

            kill 928

            -------------------------------

             

             

            jbossusr    3882  874 41 11:32 ?        00:00:07 /apps/java/jdk1.7.0_25//bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/data/DevDomain2HostController1/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules --pc-address 127.0.0.1 --pc-port 52370 -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=lonsl1101432 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 --host-config=host-slave.xml -Djboss.node.name=DevDomain2HostController1 -b localhost --backup -Djboss.home.dir=/apps/jbossEap/6.2.0 --process-restarted

            jbossusr    3929  874 99 11:32 ?        00:00:26 /apps/java/jdk1.7.0_25/bin/java -D[Server:project2_node1] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms512m -Xmx512m -Djboss.bind.address=localhost -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=lonsl1101432 -DLOG_DIR=/data/logs -Djava.awt.headless=true -Djboss.node.name=DevDomain2HostController1 -Djboss.modules.system.pkgs=org.jboss.byteman -Dspring.profiles.active=jndi-mq,stpmock,jboss -Djava.net.preferIPv4Stack=true -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 -Djboss.server.log.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/log -Djboss.server.temp.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/tmp -Djboss.server.data.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/data -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/servers/project2_node1/data/logging.properties -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server

            • 3. Re: Re: Domain behavior - Server Nodes down when Host Controller dies
              emuckenhuber

              In general killing the host-controller process with SIGKILL should not affect running servers. The process-controller will restart the host-controller and reconnect the servers. If you just use kill (SIGTERM), it would shutdown the host-controller and as part of that the running servers. Those will start again when the host-controller gets restarted by the process-controller. Which might be what is happening in your case.

              1 of 1 people found this helpful
              • 4. Re: Re: Re: Domain behavior - Server Nodes down when Host Controller dies
                vandamo

                Thanks Emanuel, indeed this explains my case regarding the Host Controller.

                Testing the same with Process Controller does kill all children processes still, so do you know if this structure is intended to be kept?

                As explained in original post, this dependency would push us to multiply the host controllers (and related process controllers) on each of our linux boxes, to ensure the projects cannot impact each other.

                • 5. Re: Domain behavior - Server Nodes down when Host Controller dies
                  wdfink

                  Strict speaking, yes.

                  But the host and process controller are very small processes which are mostly not have a high risk.

                  If you maintain two different applications with the same domain/host with server-groups you are able to start/stop the applications independent.

                  Only the small risk for situations where the host controller must be shutdown ro the process controller is dead - but I did not saw that - only by accident. And this might also happen if you have two similar HC on the same box.

                   

                  So it's on you what you prefer

                  1 of 1 people found this helpful
                  • 6. Re: Domain behavior - Server Nodes down when Host Controller dies
                    emuckenhuber

                    In addition to that - the main idea behind the process-controller was to have a parent process keeping other child processes (host-controller and servers) alive, even if one of them crash for some reason. Terminating the process-controller will shutdown all processes. We are investigating some new possibilities with java7 to remove that restriction, but that is more a long term idea.

                    • 7. Re: Domain behavior - Server Nodes down when Host Controller dies
                      vandamo

                      Thank you, this covers my question completely.