-
1. Re: Domain behavior - Server Nodes down when Host Controller dies
wdfink May 23, 2014 11:53 AM (in response to vandamo)Not sure what you do.
Normally the HC will be restarted from the ProcessController.
The server instances are keep running and handle application requests.
Which version do you use exact and how the HC die, did you kill it?
-
2. Re: Re: Domain behavior - Server Nodes down when Host Controller dies
vandamo May 26, 2014 7:43 AM (in response to wdfink)I'm not the one who performed the installation of JBoss nor its domain configuration, so it is possible that something in the setup could cause this,
although I find it unlikely (as the server groups and server nodes are created through the admin interface)
OS: Red Hat Enterprise Linux Server release 6.5 (Santiago)
JBoss:
product-version=6.2.1.GA
release-codename=Janus
management-major-version=1
release-version=7.3.1.Final-redhat-3
I indeed did a kill of the process (see below)
Sorry for the fairly verbose information, here the original running processes, the kill performed, and the new resulting processes, key for me being the node that is also a new process:
Test Server Node - Host Controller Dependency jbossusr 16477 1 0 May16 ? 00:00:00 /bin/sh /apps/jbossEap/6.2.0/bin/domain.sh -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2/domain -Djboss.domain.config.dir=/data/DevDomain2/domain/configuration --domain-config=domain.xml --host-config=host-master.xml -bmanagement linux1 -Djboss.management.native.port=15013 -Djboss.management.http.port=15011
jbossusr 16609 16477 0 May16 ? 00:10:04 /apps/java/jdk1.7.0_25/bin/java -D[Process Controller] -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.boot.log.file=/data/DevDomain2/domain/log/process-controller.log -Dlogging.configuration=file:/data/DevDomain2/domain/configuration/logging.properties -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules org.jboss.as.process-controller -jboss-home /apps/jbossEap/6.2.0 -jvm /apps/java/jdk1.7.0_25//bin/java -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -- -Dorg.jboss.boot.log.file=/data/DevDomain2/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -- -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2/domain -Djboss.domain.config.dir=/data/DevDomain2/domain/configuration --domain-config=domain.xml --host-config=host-master.xml -bmanagement linux1 -Djboss.management.native.port=15013 -Djboss.management.http.port=15011
jbossusr 16634 16609 0 May16 ? 00:15:51 /apps/java/jdk1.7.0_25/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/data/DevDomain2/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules --pc-address 127.0.0.1 --pc-port 36612 -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2/domain -Djboss.domain.config.dir=/data/DevDomain2/domain/configuration --domain-config=domain.xml --host-config=host-master.xml -bmanagement linux1 -Djboss.management.native.port=15013 -Djboss.management.http.port=15011 -Djboss.home.dir=/apps/jbossEap/6.2.0
jbossusr 727 1 0 May21 ? 00:00:00 /bin/sh /apps/jbossEap/6.2.0/bin/domain.sh -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=linux1 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 --host-config=host-slave.xml -Djboss.node.name=DevDomain2HostController1 -b localhost --backup
jbossusr 874 727 0 May21 ? 00:04:48 /apps/java/jdk1.7.0_25/bin/java -D[Process Controller] -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.boot.log.file=/data/DevDomain2HostController1/domain/log/process-controller.log -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/configuration/logging.properties -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules org.jboss.as.process-controller -jboss-home /apps/jbossEap/6.2.0 -jvm /apps/java/jdk1.7.0_25//bin/java -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -- -Dorg.jboss.boot.log.file=/data/DevDomain2HostController1/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -- -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=linux1 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 --host-config=host-slave.xml -Djboss.node.name=DevDomain2HostController1 -b localhost --backup
jbossusr 928 874 0 May21 ? 00:06:27 /apps/java/jdk1.7.0_25/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/data/DevDomain2HostController1/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules --pc-address 127.0.0.1 --pc-port 52370 -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=linux1 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 --host-config=host-slave.xml -Djboss.node.name=DevDomain2HostController1 -b localhost --backup -Djboss.home.dir=/apps/jbossEap/6.2.0
jbossusr 18328 874 2 10:34 ? 00:00:53 /apps/java/jdk1.7.0_25/bin/java -D[Server:project2_node1] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms512m -Xmx512m -Djboss.bind.address=localhost -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=linux1 -DLOG_DIR=/data/logs -Djava.awt.headless=true -Djboss.node.name=DevDomain2HostController1 -Djboss.modules.system.pkgs=org.jboss.byteman -Dspring.profiles.active=jndi-mq,stpmock,jboss -Djava.net.preferIPv4Stack=true -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 -Djboss.server.log.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/log -Djboss.server.temp.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/tmp -Djboss.server.data.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/data -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/servers/project2_node1/data/logging.properties -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server
--KILL Host Controller--------
kill 928
-------------------------------
jbossusr 3882 874 41 11:32 ? 00:00:07 /apps/java/jdk1.7.0_25//bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/data/DevDomain2HostController1/domain/log/host-controller.log -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/configuration/logging.properties -server -Xms64m -Xmx256m -XX:MaxPermSize=128m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules --pc-address 127.0.0.1 --pc-port 52370 -default-jvm /apps/java/jdk1.7.0_25//bin/java -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=lonsl1101432 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 --host-config=host-slave.xml -Djboss.node.name=DevDomain2HostController1 -b localhost --backup -Djboss.home.dir=/apps/jbossEap/6.2.0 --process-restarted
jbossusr 3929 874 99 11:32 ? 00:00:26 /apps/java/jdk1.7.0_25/bin/java -D[Server:project2_node1] -XX:PermSize=256m -XX:MaxPermSize=256m -Xms512m -Xmx512m -Djboss.bind.address=localhost -Djboss.domain.config.dir=/data/DevDomain2HostController1/domain/configuration -Djboss.domain.master.address=lonsl1101432 -DLOG_DIR=/data/logs -Djava.awt.headless=true -Djboss.node.name=DevDomain2HostController1 -Djboss.modules.system.pkgs=org.jboss.byteman -Dspring.profiles.active=jndi-mq,stpmock,jboss -Djava.net.preferIPv4Stack=true -Djboss.domain.base.dir=/data/DevDomain2HostController1/domain -Djboss.home.dir=/apps/jbossEap/6.2.0 -Djboss.domain.master.port=15013 -Djboss.socket.binding.port-offset=+000 -Djboss.server.log.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/log -Djboss.server.temp.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/tmp -Djboss.server.data.dir=/data/DevDomain2HostController1/domain/servers/project2_node1/data -Dlogging.configuration=file:/data/DevDomain2HostController1/domain/servers/project2_node1/data/logging.properties -jar /apps/jbossEap/6.2.0/jboss-modules.jar -mp /apps/jbossEap/6.2.0/modules:/data/project1/modules:/data/project2/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.server
-
3. Re: Re: Domain behavior - Server Nodes down when Host Controller dies
emuckenhuber May 26, 2014 8:02 AM (in response to vandamo)1 of 1 people found this helpfulIn general killing the host-controller process with SIGKILL should not affect running servers. The process-controller will restart the host-controller and reconnect the servers. If you just use kill (SIGTERM), it would shutdown the host-controller and as part of that the running servers. Those will start again when the host-controller gets restarted by the process-controller. Which might be what is happening in your case.
-
4. Re: Re: Re: Domain behavior - Server Nodes down when Host Controller dies
vandamo May 26, 2014 8:40 AM (in response to emuckenhuber)Thanks Emanuel, indeed this explains my case regarding the Host Controller.
Testing the same with Process Controller does kill all children processes still, so do you know if this structure is intended to be kept?
As explained in original post, this dependency would push us to multiply the host controllers (and related process controllers) on each of our linux boxes, to ensure the projects cannot impact each other.
-
5. Re: Domain behavior - Server Nodes down when Host Controller dies
wdfink May 26, 2014 9:15 AM (in response to vandamo)1 of 1 people found this helpfulStrict speaking, yes.
But the host and process controller are very small processes which are mostly not have a high risk.
If you maintain two different applications with the same domain/host with server-groups you are able to start/stop the applications independent.
Only the small risk for situations where the host controller must be shutdown ro the process controller is dead - but I did not saw that - only by accident. And this might also happen if you have two similar HC on the same box.
So it's on you what you prefer
-
6. Re: Domain behavior - Server Nodes down when Host Controller dies
emuckenhuber May 26, 2014 9:24 AM (in response to wdfink)In addition to that - the main idea behind the process-controller was to have a parent process keeping other child processes (host-controller and servers) alive, even if one of them crash for some reason. Terminating the process-controller will shutdown all processes. We are investigating some new possibilities with java7 to remove that restriction, but that is more a long term idea.
-
7. Re: Domain behavior - Server Nodes down when Host Controller dies
vandamo May 26, 2014 9:32 AM (in response to emuckenhuber)Thank you, this covers my question completely.