Tutorial domain clustering in JBoss 7 / EAP 6
willemnoorduin Feb 15, 2013 5:48 AMA while ago I wrote https://community.jboss.org/thread/162859, which described how to setup clusters, mod_cluster, Apache 2.2+ and JBoss 5.1. The same works more-or-less when you want to do such a thing with JBoss 7 standalone nodes, so I think you can figure this one out for yourself. Clustering goes better (in terms that it is much easier to maintain) when you install a domain controller and a couple of slaves which listen to that domain controller.
In this document (typos excluded), I give a howto of such a setup. Maybe it is useful, bits and pieces of this stuff are "in the cloud called the internet" of course.
===========================
CHAPTER 1: General architecture
===========================
1.1. Setup of JBoss and purpose of the document
In this chapter we work with three different kind of servers, of which the IP-addresses and the
canonical names are the following:
192.168.0.1 webserver ws (Apache 2.2.x)
192.168.10.0 domain controller dc (JBoss 7 / EAP6)
192.168.20.0 applicationserver slave1 of dc (JBoss 7 / EAP6)
192.168.30.0 applicationserver slave2 of dc (JBoss 7 / EAP6)
1.2. JBoss Clusters, Instances and applications:
In this document we configure two apache virtual hosts (www.application1.nl and www.application2.nl).
Each virtual host shall be connected via mod_cluster with it's own jboss cluster. Clusters are:
cluster1 consists of {node11@as1 and node12@as2)
cluster2 consists of {node21@as1 and node22@as2}
application1.war is deployed to cluster1 and reacheable via http://www.application1.nl/application1
application2.war is deployed to cluster2 and reacheable via http://www.application2.nl/application2
In this situation we have two Apache Virtual hosts which sent requests to two different JBoss-AS clusters, each JBoss-AS cluster consists two instances, each one of them runs on a different (physical) Linux Server (or whatever).
In the DNS, both URLs www.application1.nl and www.application2.nl resolve to the webserver, and we are using mod_cluster in combination with ajp (Apache JServ Protocol) to pass application calls to the JBoss-AS cluster.
This document describes the configuration of the domain controller and application servers, and does not concentrate on the mod_cluster aspect so much. When you have a clustername to which you can send your application request, things are very much transparent (but see the CHAPTER 8 in https://community.jboss.org/thread/162859). The goal is to describe the cluster seperation completely from the standpoint of the domain controller / slave relationship. This means that, with exeption of creating a new profile, which has to do by "hand", much of the stuff we do with the JBoss client (jboss-cli) on the domain crontroller.
1.3. Brief contents of this document
CHAPTER 2: Installation and configurating the domain crontroller
CHAPTER 3: Installation of the slaves
CHAPTER 4: Creating a new profile
CHAPTER 5: Configuring the architecture for running a cluster
CHAPTER 6: Deploying the applications
CHAPTER 7: What we have not discussed
=================================
CHAPTER 2: The domain crontroller
=================================
2.1. A brief history of the installation of JBoss 7
After downloading JBoss 7 (mine is called jboss-as-7.1.1.Final.zip) and an Oracle JDK (jdk1.6.0_30.tar.gz) do the following:
root@dc # cd /opt
root@dc # cp /tmp/jdk1.6.0_30.tar.gz /tmp/jboss-as-7.1.1.Final.zip .
root@dc # tar xfz jdk1.6.0_30.tar.gz ; unzip jboss-as-7.1.1.Final.zip
root@dc # ln -s jdk1.6.0_30.tar.gz jdk ; ln -s jboss-as-7.1.1.Final jboss
root@dc # chown -R jboss:jboss jboss-as-7.1.1.Final
2.2. Installing the domain-controller
The easiest way to start the domain controller is just fire up /opt/jboss/bin/domain.sh -b <bind-address> after setting JAVA_HOME to the correct value. Note that this is also the clumsiest way, since it starts in the foreground. Now there are plenty of ways to configure it to start in the background and barf into a console log instead of a console-window, but we do not want to go into that here. Now start:
root@dc # /opt/jboss/bin/domain.sh --domain-config domain.xml --host-config=host-master.xml -P file:///apps/jboss/nlptc27b09/configs/domain.properties
with:
root@192.168.10.0:/opt# cat /apps/jboss/192.168.10.0/configs/domain.properties
jboss.bind.address=192.168.10.0
jboss.bind.address.management=192.168.10.0
jboss.socket.binding.port-offset=0
jboss.config.dir=/apps/jboss/nlptc27b09
jboss.domain.log.dir=/var/log/jboss/nlptc27b09
jboss.domain.base.dir=/apps/jboss/profiles/nlptc27b09
jboss.domain.base.url=file:///apps/jboss/profiles/nlptc27b09
jboss.domain.master.address=192.168.10.0
jboss.domain.name=192.168.10.0
Which is just a way of how to administer things more neatly. Anyway, the above processes should run.
You should end with something like this:
root@192.168.10.0:/opt# ps -ef | grep jboss | grep -v grep
jboss 8339 1 0 08:40 ? 00:00:00 /bin/sh /opt/jboss/bin/domain.sh --domain-config domain.xml --host-config=host.xml -P file:///apps/jboss/192.168.10.0/configs/domain.properties
jboss 8382 8339 0 08:40 ? 00:00:03 /opt/jdk/bin/java -D[Process Controller] -server -Dorg.jboss.boot.log.file=/var/log/jboss/192.168.10.0/process-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/192.168.10.0/configuration/logging.properties -jar /opt/jboss/jboss-modules.jar -mp /opt/jboss/modules org.jboss.as.process-controller -jboss-home /opt/jboss -jvm /opt/jdk/bin/java -mp /opt/jboss/modules -- -Dorg.jboss.boot.log.file=/var/log/jboss/192.168.10.0/host-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/192.168.10.0/configuration/logging.properties -server -- -default-jvm /opt/jdk/bin/java --domain-config domain.xml --host-config=host.xml -P file:///apps/jboss/192.168.10.0/configs/domain.properties
jboss 8396 8382 0 08:40 ? 00:00:11 /opt/jdk/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log/jboss/192.168.10.0/host-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/192.168.10.0/configuration/logging.properties -server -jar /opt/jboss/jboss-modules.jar -mp /opt/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /opt/jboss/modules --pc-address 127.0.0.1 --pc-port 46303 -default-jvm /opt/jdk/bin/java --domain-config domain.xml --host-config=host.xml -P file:///apps/jboss/192.168.10.0/configs/domain.properties -Djboss.home.dir=/opt/jboss
2.3. Adding JBoss users
2.3.1 Adding the admin user
The user admin is used to login to the administration website:
root@dc : /opt/jboss/bin # ./add-user.sh
What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a):
Enter the details of the new user to add.
Realm (ManagementRealm) :
Username : admin
Password : <welkom>
Re-enter Password : <welkom>
Updated user 'admin' to file '/opt/jboss-eap-6.0.0/standalone/configuration/mgmt-users.properties'
Updated user 'admin' to file '/opt/jboss-eap-6.0.0/domain/configuration/mgmt-users.properties'
Is this new user going to be used for one AS process to connect to another AS process e.g. slave domain controller?
yes/no? no
You need this user when you want to administer the domain controller with jboss-cli.sh from somewhere else.
2.3.2 Adding the slave user
The user slaveuser is used by external node to login to the Domain Controller:
root@dc : /opt/jboss/bin # ./add-user.sh
What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a):
Enter the details of the new user to add.
Realm (ManagementRealm) :
Username : slaveuser
Password : <welkom>
Re-enter Password : <welkom>
About to add user 'slaveuser' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'slaveuser' to file '/opt/jboss/standalone/configuration/mgmt-users.properties'
Added user 'slaveuser' to file '/opt/jboss/domain/configuration/mgmt-users.properties'
Is this new user going to be used for one AS process to connect to another AS process e.g. slave domain controller?
yes/no? yes
To represent the user add the following to the server-identities definition <secret value="d2Vsa29t" />
=====================================
CHAPTER 3: Installation of the slaves
=====================================
Come up with a unique name (here slave1) for a slave and repeat this chapter for as much slaves as you want to have in your architecture. Note that slaves can be anywhere (that is, on production servers, somewhere else on the internet, and even on your laptop). The slavenames have to be unique (you can use for instance a DNS-like name for bigger architectures).
3.1. A brief history of the installation of JBoss 7 for slaves.
This is easy, just follow 2.1
3.2. Installing a slave server of the dc
Since this is the same thing over and over again, we tell you how to do it on the applicationserver slave1 (192.168.20.0):
3.2.1. Adjusting host-slave.xml
<?xml version='1.0' encoding='UTF-8'?>
<host name="slave1" xmlns="urn:jboss:domain:1.3">
<management>
<security-realms>
<security-realm name="ManagementRealm">
<server-identities>
<secret value="${jboss.domain.login.password}"/>
</server-identities>
<authentication>
<local default-user="$local"/>
<properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
</authentication>
</security-realm>
...
There is some story here to tell, when you define it like this, you must offer a jboss.domain.login.password property during startup. Upshot is, this must be a cleartext property. If you are paranoid, you can work with:
...
<security-realm name="ManagementRealm">
<server-identities>
<secret value="d2Vsa29t"/>
</server-identities>
<authentication>
<local default-user="$local"/>
<properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
</authentication>
</security-realm>
...
The secret value is the one which was given to you in 2.3.2. Now to make this really a slave of the dc, adjust as follows:
...
<domain-controller>
<remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm" username="${jboss.domain.login.user}"/>
</domain-controller>
...
We use variables here, because we use a property file to feed our jboss process. If you don't want that, you can use the literal versions of that:
3.2.2. Starting the slave.
The easiest way to start the domain controller is just fire up /opt/jboss/bin/domain.sh -b <bind-address> after setting JAVA_HOME to the correct value. Note that this is also the clumsiest way, since it starts in the foreground. Now there are plenty of ways to configure it to start in the background and barf into a console log instead of a console-window. Now start:
root@slave1 # /opt/jboss/bin/domain.sh --domain-config domain.xml --host-config=host-slave.xml -P file:///apps/jboss/slave1/configs/domain.properties
with:
root@slave1:/opt# cat /apps/jboss/alave1/configs/domain.properties
jboss.bind.address=192.168.20.0
jboss.bind.address.management=192.168.20.0
jboss.socket.binding.port-offset=0
jboss.domain.log.dir=/var/log/jboss/slave1
jboss.domain.master.address=192.168.10.0
jboss.domain.name=slave1
jboss.domain.login.user=slaveuser
jboss.domain.login.password=welkom
We end up with
root@slave1:/opt# ps -ef | grep jboss | grep -v grep
jboss 6670 1 0 11:44 ? 00:00:00 /bin/sh /opt/jboss/bin/domain.sh --domain-config domain.xml --host-config=host-slave.xml -P file:///apps/jboss/fbewslave/configs/domain.properties
jboss 6713 6670 0 11:44 ? 00:00:00 /opt/jdk/bin/java -D[Process Controller] -server -Dorg.jboss.boot.log.file=/var/log/jboss/fbewslave/process-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/fbewslave/configuration/logging.properties -jar /opt/jboss/jboss-modules.jar -mp /opt/jboss/modules org.jboss.as.process-controller -jboss-home /opt/jboss -jvm /opt/jdk/bin/java -mp /opt/jboss/modules -- -Dorg.jboss.boot.log.file=/var/log/jboss/fbewslave/host-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/fbewslave/configuration/logging.properties -server -- -default-jvm /opt/jdk/bin/java --domain-config domain.xml --host-config=host-slave.xml -P file:///apps/jboss/fbewslave/configs/domain.properties
jboss 6728 6713 3 11:44 ? 00:00:04 /opt/jdk/bin/java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log/jboss/fbewslave/host-controller.log -Dlogging.configuration=file:/apps/jboss/profiles/fbewslave/configuration/logging.properties -server -jar /opt/jboss/jboss-modules.jar -mp /opt/jboss/modules -jaxpmodule javax.xml.jaxp-provider org.jboss.as.host-controller -mp /opt/jboss/modules --pc-address 127.0.0.1 --pc-port 37932 -default-jvm /opt/jdk/bin/java --domain-config domain.xml --host-config=host-slave.xml -P file:///apps/jboss/fbewslave/configs/domain.properties -Djboss.home.dir=/opt/jboss
3.2.3. Controlling if the slave is connected to the domain controller
The slave is connected to the domain controller when you can see slave1 as a host in the jboss-cli as follows:
root@dc:/opt/jboss/bin# ./jboss-cli.sh --controller=192.168.10.0 --connect
[domain@192.168.10.0:9999 /] ls -l /host
slave1
master
=================================
CHAPTER 4: Creating a new profile
=================================
4.1. Add a profile definition to the domain.xml
Since it is not (yet) possible to copy a profile with the JBoss client, we have to copy a profile by hand:
1. Open /opt/jboss/domain/configuration/domain.xml on the domain crontroller.
2. When you want a ha-typed profile, copy all the entries between:
<profile name="ha">
...
</profile>
to a new entry between the <profiles> ... </profiles> tags, for example:
<profile="cluster1">
...
</profile>
Remark:
The following properties of our architecture must be set on the profile level:
* The balancer-name, this is essentially what you need to put in the Apache Virtual Host in the <Proxy> ... </Proxy> tags.
* The jdbc-driver entries (that is, when you need one of-course, but an application without its data is like ...).
* The datasource entries.
* Access logging of the web system.
4.2. Control if the profile is there
After hacking (eh adjusting) the domain.xml of the domain controller, restart the domain crontroller and check whether the profile is there:
[domain@145.78.121.33:9999 /] ls -l /profile
cluster1
cluster2
default
full
full-ha
ha
We see that we have repeat 4.1 with a cluster-name cluster2, since we are going to serve two (seperated clusters).
=============================================================
CHAPTER 5: Configuring the architecture for running a cluster
=============================================================
5.1. A script to configure to configure the profile and the server-group
Remember: allthough it is not a 1-1 map, you can see the combination profile + server-group as a simple definition of what we had in JBoss 5 for a cluster.
Safe the following to a text file (say /tmp/cluster1.cli).
#
# Setting the balancer name in the profile
#
/profile=cluster1/subsystem=modcluster/mod-cluster-config=configuration:write-attribute(name=balancer, value="cluster1")
#
# Creating the server group
#
/server-group=cluster1-server-group:add(profile=cluster1, socket-binding-group=ha-sockets, socket-binding-port-offset=0)
#
# Creating JDBC Driver entries and datasource entry for cluster1
# Datasource definition shown for Oracle
#
/profile=cluster1/subsystem=datasources/jdbc-driver=mysql:add(driver-name="mysql",driver-module-name="com.mysql")
/profile=cluster1/subsystem=datasources/jdbc-driver=oracle:add(driver-name="oracle",driver-module-name="com.oracle.ojdbc6")
data-source --name=cluster1_DS \
--connection-url="jdbc:oracle:thin:@databaseserver:port:database" \
--jndi-name="java:jboss/cluster1_DS" \
--driver-name="oracle" \
--user-name="cluster1_appl" --password="password" \
--use-java-context="true" \
--min-pool-size="0" \
--max-pool-size="64" \
--idle-timeout-minutes="30" \
--background-validation="false" \
--background-validation-millis="1" \
--validate-on-match="true" \
--allocation-retry="0" \
--share-prepared-statements="false" \
--set-tx-query-timeout="false" \
--query-timeout="0" \
--use-try-lock="0" \
--url-delimiter="|" \
--profile=cluster1 add
#
# Creating application properties entry (if needed of course)
#
/server-group=cluster1-server-group/system-property=PROPERTY_FILE:add(value="/home/jboss/data/cluster1.properties")
#
# Creating access log entry
#
/profile=cluster1/subsystem=web/virtual-server=default-host/access-log=configuration:add(extended=false, pattern="%a %t %H %p %U %s ", prefix=false, resolve-hosts=false, rotate=true
Of course there could be more or less entries in this file (just put there what you want), after this:
root@dc # cd /opt/jboss/bin
root@dc # /jboss-cli.sh --controller=192.168.10.0 --connect --file=/tmp/cluster1.cli
5.2. Creating and configuration server-configs
A server-group contains serveral server-configs, and the last ones are synonym with what we had as clusternodes in the good ol' dagys. These server-configs can be running everywhere where there is a JBoss slave is installed. If you have not done so, follow 3.2.3 and check if the slave is there. Then (again in the jboss-cli):
[domain@192.168.10.0:9999 /] /host=slave1/server-config=node11:add(auto-start=true, group=cluster1-server-group, socket-binding-group=ha-sockets, socket-binding-port-offset=0)
{
"outcome" => "success",
"result" => undefined
}
5.3. Starting the nodes and where are my files
You can start, stop or restart the server-configs as follows:
[domain@192.168.10.0:9999 /] /host=slave1/server-config=node11:start
{
"outcome" => "success",
"result" => "STARTING"
}
On slave1 (we can now see something like this:
root@slave1:/home/wnoordui# tree /opt/jboss/domain/servers/node11/
/opt/jboss/domain/servers/node11/
|-- data
|-- log
| |-- accesslog
| | `-- access_log.2013-02-13
| |-- boot.log
| |-- server.log
| `-- server.log.2013-02-11
`-- tmp
5.4. On enabling datasources and testing them
Enable a datasource and test if it's there:
[domain@192.168.10.0:9999 /] /profile=cluster1/subsystem=datasources/data-source=cluster1_DS:enable
{
"outcome" => "success",
"result" => undefined,
"server-groups" => {"cluster1-server-group" => {"host" => {"slave1" => {"node11" => {"response" => {
"outcome" => "success",
"result" => undefined
}}}}}}
}
[domain@192.168.10.0:9999 /] /host=slave1/server=node11/subsystem=datasources/data-source=cluster1_DS:test-connection-in-pool
{
"outcome" => "success",
"result" => [true]
}
=================================================
CHAPTER 6: Deploying and testing the applications
=================================================
6.1. Deploy an application
This is simple, suppose we have a /tmp/testapp.war which contains the application code. Then:
[domain@192.168.10.0:9999 /] deploy /tmp/testapp.war --server-groups=cluster1-server-group
and follow the logfiles on one of the nodes of cluster1-server-group (node11 would be a great choice)/
6.2. Test the application
For me: lynx http://slave1:8080/testapp works fine, but if you are gui-impaired you can you iceweazel, firefox or (even) ie.
=====================================
CHAPTER 7: What we have not discussed
=====================================
In this document only the simplest way of getting clusters or clusterlike architectures running on a domain cluster. I haven't delved into things like hibernation, messaging, etc. Also you can read somewhere else (for example https://community.jboss.org/thread/162859 in CHAPTER 8 there), how to setup an Apache / mod cluster for this thing