1 2 Previous Next 23 Replies Latest reply on Oct 21, 2016 3:49 PM by Sandy Bonneville

    mod_cluster and multiple virtual hosts

    Willem Noorduin Newbie

      I have been on and out on this topic, and the followijng setup for mod_cluster, Apache 2.2+ and JBoss 5.1+ is working in our test environment. Maybe someone can review this setup, so that we might have a little howto on our hands for future usage:

       

      ===============================
      CHAPTER 1: General architecture
      ===============================

       

      1.0.    Setup of JBoss and purpose of the document

       

      In this chapter we work with three different kind of servers, of which the IP-addresses and the
      canonical names are the following:

       

      192.168.0.1     webserver               ws (Apache 2.2.x)
      192.168.10.0    applicationserver       as1 (JBoss 5 +)
      192.168.20.0    applicationserver       as2 (JBoss 5 +)

       

      1.1.    JBoss Clusters, Instances and applications:

       

      In this document we configure two apache virtual hosts (www.application1.nl and www.application2.nl).
      Each virtual host shall be connected via mod_cluster with it's own jboss cluster. Clusters are:

       

      cluster1 consists of {node11@as1 and node12@as2)
      cluster2 consists of {node21@as1 and node22@as2}

       

      application1.war is deployed to cluster1 and reacheable via http://www.application1.nl/application1
      application2.war is deployed to cluster2 and reacheable via http://www.application2.nl/application2

       

      In this situation we have two Apache Virtual hosts which sent requests to two different JBoss-AS clusters, each JBoss-AS cluster consists two instances, each one of them runs on a different (physical) Linux Server (or whatever).

       

      In the DNS, both URLs www.application1.nl and www.application2.nl resolve to the webserver, and we are using mod_cluster in combination with ajp (Apache JServ Protocol) to pass application calls to the JBoss-AS cluster.

       

      You can solve this architecture by mod_jk.so and one humongous, very dirty workers.properties. But a much more elegant way is to solve this with mod_cluster. The last stable community version is mod_cluster 1,1.1, which solves a lot of bugs and problems surrounding the 1.0 versions. Alas, upon writing this document, version 1.0.1 was still the only version supported by Redhat JBoss-AS 5 (Enterprise edition). So in this document we have the following:


      CHAPTER 2: General Setup JBoss instances.

      In this chapter, we describe the general setup of each and every node. In this document, we use the all/ instance template.


      CHAPTER 3: mod_cluster 1.0.1

      This describes the installation and configuration of the JBoss-AS instance.


      CHAPTER 4: mod_cluster 1.1.1

      This describes the installation and configuration of the JBoss-AS instance.


      CHAPTER 5: Setting up the cluster

      In this chapter we use the template created in Chapter 2 to make two cluster (cluster1 anf cluster2), each consisting of two nodes.


      CHAPTER 6: Deploying applications to the cluster

      In this chaoter we nodel the applications application1.war and application2.war after a little application known as clusterjsp.war (see attachment).


      CHAPTER 7: Apache mod_cluster 1.0.x

      How to configure the virtual hosts to run the two applications www.application1.nl/application1 and www.application2.nl/application2 when working with mod_cluster 1.0.x.

       

      CHAPTER 8: Apache mod_cluster 1.1.1

      How to configure the virtual hosts to run the two applications www.application1.nl/application1 and www.application2.nl/application2 when working with mod_cluster 1.1.1.


      CHAPTER 9: Starting up and loggings

      Starting up and what you see in the logs.


      We make use of the following variables, describing the various locations:

       

      o   ${JBOSS_HOME} (the place where JBoss-AS is installed)
      o   ${INSTANCE} (general variable denoting the toplevel directory of s JBoss instance, such that
          for instance ${INSTANCE}/deploy is the deploy/ directory of an instance.


      =======================================
      CHAPTER 2: General Setup JBoss instance
      =======================================`

       

      2.1.    Create one cluster instance

       

      as1 # cd ${JBOSS_HOME}/server
      as1 # cp -r -p all node11
      as1 # chown -R jboss:jboss node11

       

      When you have JBoss-AS Enterprise Edition of Redhat, you can take the production/ template. This template has the same cluster capabilities like all/, but is performance-wise more enhanced (see the discussion  http://community.jboss.org/message/575859)


      2.2.    Tweaking the message service

       

      Out of the box, the Enterprise Edition gives a suckerPassword error when you do nothing about this (I didn't find the problem in the Community Releases 5 and/or 6).

       

      Edit ${INSTANCE}/deploy/messaging/messaging-jboss-beans.xml such that the following is  visible:

       

            <property name="suckerPassword">admin</property>
            <property name="securityDomain">messaging</property>

       

      Edit ${INSTANCE}/deploy/messaging/messaging-service.xml such that the following is visible:

       

            <!-- The password used by the message sucker connections to create connections.
                 THIS SHOULD ALWAYS BE CHANGED AT INSTALL TIME TO SECURE SYSTEM -->
            <attribute name="SuckerPassword">admin</attribute>
            <!-- The name of the server aspects configuration resource

       

      Remarks:

      o  Of course "admin" can be any (hopefully stronger) password, the tweak is that they have to be the same to get quickly rid of the error during startup. Of course you have to look into this a little bit more in a production environment (i.e. this howto is not about how to setup messaging properly).

      o  We shall make the rest of the nodes after we have configured and installed mod_cluster. In this way you have to do the tedious configuration once, instead of a number of times.

      o  A handyman's tip from yours truly is to use vi with syntax coloring for xml files, otherwise you keep searching on that one little mistake...


      2.3. Start JBoss and look for errors

       

      I do this on a regular basis during the process to look of JBoss itself agrees with our configuration. At this time, you should be able the start JBoss-AS without trouble:

       

      as1 # /${JBOSS_HOME}/bin/run.sh -c node11 -g cluster1 -Djboss.messaging.ServerPeerID=1 -Djboss.service.binding.set=ports-default
      -Djboss.server.log.dir=/apps/data/node1/logs/ -Djboss.jvmRoute="node11" -Djboss.Domain=domain1 -Djboss.bind.address="192.168.10.0"

       

      Remark:
      We pass as many variables as possible as "-Djboss. .." constructs. In this way, we can make a template instance which we can use without thinking. This line does start JBoss in the foreground. Note also that we have another log directory than the JBoss default (feel free to remove this one, of course, but when you do use jboss.server.log.dir, it must exist and must be writeable for the JBoss user).


      ============================
      CHAPTER 3: MOD_CLUSTER 1.0.1
      ============================

       

      3.1.    Downloading mod_cluster 1.0.1 Java Bundle:

       

      In most cases this can be found on http://www.jboss.org/mod_cluster. In case you isist in using the Redhat Enterprise Edition, the Java Bundle can be found in jboss-ewp-5.1.0.zip. Let ${MOD_CLUSTER_SRC} be the toplevel directiry of the Java bundle.


      3.2.    Deploying mod_cluster.sar

       

      Make sure the JBoss-AS instance isn't running. This isn't really necessary, but provides us with a more
      clean situation at the end.

       

      as1 # cd ${MOD_CLUSTER_SRC}
      as1 # chown -R jboss:jboss mod-cluster.sar/
      as1 # cp -r -p mod-cluster.sar ${INSTANCE}/deploy/


      3.3.   Configuring the server.xml:

       

      Go to ${INSTANCE{/deploy/jbossweb.sar Open server.xml and add the following listeners (activate the one that applies:

       

      <!-- <Listener className="org.jboss.web.tomcat.service.deployers.MicrocontainerIntegrationLifecycleListener" delegateBeanName="ModClusterService"/> --><!-- Non-clustered mode -->

      <Listener
      className="org.jboss.web.tomcat.service.deployers.MicrocontainerIntegrationLifecycleListener"
      delegateBeanName="HAModClusterService"/--><!-- Clustered mode -->

       

      Open server.xml and add (change into) the following to the Engine tag:

       

      <Engine name="jboss.web" defaultHost="localhost" jvmRoute="${jboss.jvmRoute}">

       

      Add the following Alias to the host-section in server.xml:

       

      <Alias>www.application2.nl</Alias>


      Remarks:

      The url in the in <Alias> ... </Alias> is the URL on the webserver that serves the application and it the only element that isn't dynamic. You can give something like:

       

      <Alias>${jboss.Alias}</Alias>

       

      a try (don't forget to add the appropriate name to the startup line). Because mod_cluster 1.1.1 really don't need it, I haven't given this much thought.


      3.4. Configuring jboss.beans:

       

      Update the WebServer bean in ${INSTANCE}/deploy/jbossweb.sar/META-INF/jboss-beans.xml such that it reads:

       

      <!-- <depends>ModClusterService</depends> --><!-- Non-clustered mode -->
      <depends>HAModClusterService</depends><!-- Clustered mode -->


      3.5. Configuring mod_cluster.sar

       

      Go to ${INSTANCE}/deploy/mod-cluster.sar/META-INF. Open the file mod-cluster-jboss-beans.xml and add the following to the HAModClusterService (proxyList, domain and balancer):

       

      <!-- Configure this node's communication with the load balancer -->
        <bean name="HAModClusterConfig" class="org.jboss.modcluster.config.ha.HAModClusterConfig" mode="On Demand">

          <!-- Comma separated list of address:port listing the httpd servers
               where mod_cluster is running. -->
          <property name="proxyList">${jboss.modcluster.proxyList:localhost:80}</property>
          <property name="domain">${jboss.Domain:DefaultDomain}</property>
      ...
                  <!-- Configuration values for the load balancer itself (must be the
               same on all nodes in the cluster). These will be passed to the
               load balancer. -->

          <property name="stickySession">true</property>
          <property name="stickySessionForce">true</property>
          <property name="stickySessionRemove">false</property>
          <property name="maxAttempts">1</property>
          <property name="workerTimeout">-1</property>
          <property name="balancer">${jboss.modcluster.balancer}</property>
        </bean>


      3.6. Starting JBoss-AS and monitoring for errors:

       

      ${JBOSS_HOME}/bin/run.sh -c node1 -g cluster1 -Djboss.messaging.ServerPeerID=1 -Djboss.service.binding.set=ports-default
      -Djboss.server.log.dir=/apps/data/node1/logs/ -Djboss.jvmRoute="node1" -Djboss.Domain=domain1
      -Djboss.mod_cluster.proxyList="192.168.0.1:7000" -Djboss.modcluster.balancer="cluster1"
      -Djboss.bind.address="192.168.10.0"

       

      Remarks:

      o   proxyList is the IP-address or the FQDN of the webserver.
      o   We come back to the port 7000
      o   bind.address is the IP-address or the FQDN of the application server.

       

      You can exoect a stack trace because JBoss-AS will try to connect to the Webserver. Since we aren't there yet,
      this is okay for now. If you getting these kind of connection errors when the Webserver is started, check if
      there is a firewall in the way.


      ============================
      CHAPTER 4: MOD_CLUSTER 1.1.1
      ============================

       

      4.1.    Downloading mod_cluster 1.1.1 Java Bundle:

       

      In most cases this can be found on http://www.jboss.org/mod_cluster. Let ${MOD_CLUSTER_SRC} be the toplevel directiry of the Java bundle.


      4.2.    Deploying mod_cluster.sar

       

      Make sure the JBoss-AS instance isn't running. This isn't really necessary, but provides us with a more clean situation at the end.

       

      as1 # cd ${MOD_CLUSTER_SRC}
      as1 # chown -R jboss:jboss mod-cluster.sar/
      as1 # cp -r -p mod-cluster.sar ${INSTANCE}/deploy/


      4.3. Configuring the server.xml:

       

      No additional configuration needed:

       

      o  The Listener is not needed anymore.
      o  The jvmRoute is generated by mod_cluster and not needed anymore.
      o  The Alias entry is not needed anymore.


      4.4. Configuring jboss.beans:

       

      The somewhat strange (almost circular) dependency in the Webserver Bean is not needed anymore. No need to update the jboss-beans.xml of the jbossweb.sar/.


      4.5 Configuring mod_cluster.sar

       

      Go to ${INSTANCE}/deploy/mod-cluster.sar/META-INF. Open the file mod-cluster-jboss-beans.xml and add the following to the HAModClusterService (proxyList, domain and balancer):

       

      <!-- Configure this node's communication with the load balancer -->
        <bean name="HAModClusterConfig" class="org.jboss.modcluster.config.ha.HAModClusterConfig" mode="On Demand">

          <!-- Comma separated list of address:port listing the httpd servers
               where mod_cluster is running. -->
          <property name="proxyList">${jboss.modcluster.proxyList:localhost:80}</property>
          <property name="domain">${jboss.Domain:DefaultDomain}</property>
      ...
                  <!-- Configuration values for the load balancer itself (must be the
               same on all nodes in the cluster). These will be passed to the
               load balancer. -->

          <property name="stickySession">true</property>
          <property name="stickySessionForce">true</property>
          <property name="stickySessionRemove">false</property>
          <property name="maxAttempts">1</property>
          <property name="workerTimeout">-1</property>
          <property name="balancer">${jboss.modcluster.balancer}</property>
        </bean>


      4.6. Starting JBoss-AS and monitoring for errors:

       

      ${JBOSS_HOME}/bin/run.sh -c node1 -g cluster1 -Djboss.messaging.ServerPeerID=1 -Djboss.service.binding.set=ports-default
      -Djboss.server.log.dir=/apps/data/node1/logs/ -Djboss.jvmRoute="node1" -Djboss.Domain=domain1
      -Djboss.mod_cluster.proxyList="192.168.0.1:80" -Djboss.modcluster.balancer="cluster1"  -Djboss.bind.address="192.168.10.0"


      Remarks:

      o   proxyList is the IP-address or the FQDN of the webserver.
      o   bind.address is the IP-address or the FQDN of the application server.

       

      You can exoect a stack trace because JBoss-AS will try to connect to the Webserver. Since we aren't there yet,
      this is okay for now. If you getting these kind of connection errors when the Webserver is started, check if
      there is a firewall in the way.


      4.7. JBoss 6 and the mod_cluster 1.1.0 bug

       

      JBoss 6 is shipped with a mod_cluster 1.1.0 version, which has a little bug in that it renamed some of the properties in the mod-cluster-jboss-beans.xml to mod_cluster instead of modcluster. So, in JBoss 6 you must use:

       

          <property name="balancer">${jboss.modcluster.balancer}</property>

       

      instead of mod_cluster. As of mod_cluster 1.1.1, both are possible. I have looked a little at JBoss 6 and (except from this little tweak) configuration goes the same as mentioned in chapter 4. Note that you edit the mod-cluster-jboss-beans.xml here (there is also a mod_cluster-ha-jboss-beans.xml, but this reads in the rest of the configuration).


      =================================
      CHAPTER 5: Setting up the cluster
      =================================

       

      5.1. Setting up the nodes for our two clusters.

       

      In this chapter we use the template created in Chapter 2 to make two cluster (cluster1 anf cluster2), each
      consisting of two nodes:

       

      On as1:

       

      as1 # cd ${JBOSS_HOME}/server
      as1 # cp -r -p node11 node21

       

      So at the end we have:

       

      as1 # ls -l ${JBOSS_HOME}/server

      drwxr-xr-x  8 jboss jboss 4096 Jan 20 14:43 all
      drwxr-xr-x  6 jboss jboss 4096 Jan 20 14:43 default
      drwxr-xr-x  6 jboss jboss 4096 Jan 20 14:43 minimal
      drwxr-xr-x 11 jboss jboss 4096 Feb  3 09:16 node11
      drwxr-xr-x 12 jboss jboss 4096 Jan 31 10:40 node21
      drwxr-xr-x  8 jboss jboss 4096 Jan 20 14:43 production
      drwxr-xr-x  6 jboss jboss 4096 Jan 20 14:43 standard
      drwxr-xr-x  6 jboss jboss 4096 Jan 20 14:43 web


      i.e. node11 belongs to cluster1 and node21 belongs to cluster2. Copy the node11 directory to (say /tmp) of the second JBoss-AS as2. There:

       

      as2 # cd ${JBOSS_HOME}/server
      as2 # mv /tmp/node11 node12
      as2 # cp -r -p node12 node22

       

      So at the end we have:

       

      as1 # ls -l ${JBOSS_HOME}/server

      drwxr-xr-x  8 jboss jboss 4096 Jan 20 14:43 all
      drwxr-xr-x  6 jboss jboss 4096 Jan 20 14:43 default
      drwxr-xr-x  6 jboss jboss 4096 Jan 20 14:43 minimal
      drwxr-xr-x 11 jboss jboss 4096 Feb  3 09:16 node12
      drwxr-xr-x 12 jboss jboss 4096 Jan 31 10:40 node22
      drwxr-xr-x  8 jboss jboss 4096 Jan 20 14:43 production
      drwxr-xr-x  6 jboss jboss 4096 Jan 20 14:43 standard
      drwxr-xr-x  6 jboss jboss 4096 Jan 20 14:43 web


      Remark (if you like math or are a bookkeeper):

       

      o  The first index is the index of the cluster, i.e. cluster1 = {node11, node12 }.
      o  The second index is the index of the JBoss Application Server.


      5.2. Starting the clusters

       

      First make a few loggings locations (when you don't fill jboss.server.log.dir, the logs should be in the ${INSTANCE}/logs directory):

       

      as1 # mkdir -p /apps/data/node11/logs/ /apps/data/node21/logs/
      as1 # chown -R jboss:jboss /apps/data/node11/logs/ /apps/data/node21/logs/

      as2 # mkdir -p /apps/data/node12/logs/ /apps/data/node22/logs/
      as2 # chown -R jbossLjboss /apps/data/node12/logs/ /apps/data/node22/logs/

       

      The cluster can be started by issueing the following start commands:

       

      On as1:

       

      as1 # su - jboss

      as1 $ ${JBOSS_HOME}/bin/run.sh -c node11 -g cluster1 -Djboss.messaging.ServerPeerID=1
      -Djboss.service.binding.set=ports-default -Djboss.server.log.dir=/apps/data/node11/logs/
      -Djboss.jvmRoute="node11" -Djboss.Domain=domain1 -Djboss.mod_cluster.proxyList="192.168.0.1:80"
      -Djboss.modcluster.balancer="cluster1" -Djboss.bind.address="192.168.10.0"

       

      as1 $ ${JBOSS_HOME}/bin/run.sh -c node21 -g cluster2 -Djboss.messaging.ServerPeerID=1
      -Djboss.service.binding.set=ports-01 -Djboss.server.log.dir=/apps/data/node21/logs/
      -Djboss.jvmRoute="node21" -Djboss.Domain=domain2 -Djboss.mod_cluster.proxyList="192.168.0.1:80"
      -Djboss.modcluster.balancer="cluster2" -Djboss.bind.address="192.168.10.0"

       

      On as2:

       

      as2 # su - jboss

      as2 $ ${JBOSS_HOME}/bin/run.sh -c node12 -g cluster1 -Djboss.messaging.ServerPeerID=1
      -Djboss.service.binding.set=ports-default -Djboss.server.log.dir=/apps/data/node12/logs/
      -Djboss.jvmRoute="node11" -Djboss.Domain=domain1 -Djboss.mod_cluster.proxyList="192.168.0.1:80"
      -Djboss.modcluster.balancer="cluster1" -Djboss.bind.address="192.168.10.0"

       

      as2 $ ${JBOSS_HOME}/bin/run.sh -c node22 -g cluster2 -Djboss.messaging.ServerPeerID=1
      -Djboss.service.binding.set=ports-01 -Djboss.server.log.dir=/apps/data/node22/logs/
      -Djboss.jvmRoute="node21" -Djboss.Domain=domain2 -Djboss.mod_cluster.proxyList="192.168.0.1:80"
      -Djboss.modcluster.balancer="cluster2" -Djboss.bind.address="192.168.10.0"


      Remark:

      o  This setup leaves us with four open terminal windows, which is not very practical. Convenient
         ways to startup a cluster and many instances is another story.
      o  This setup starts two clusters which are divided over two physical application servers.


      =================================
      CHAPTER 6: deploying applications
      =================================

       

      6.1. A sample application:

       

      You can just take any application which can be used for testing a cluster. Ours is the application clusterjsp.war, which comes from the ee-samples which can be found all over the web. For the onces that haven't this see the attached clusterjsp.war file. You can unwar this clusterjsp.war file:

       

      as1 # ${JAVA_HOME}/bin/jar xvf clusterjsp.war
        created: META-INF/
      inflated: META-INF/MANIFEST.MF
        created: WEB-INF/
      inflated: HaJsp.jsp
      inflated: ClearSession.jsp
      inflated: WEB-INF/sun-web.xml
      inflated: WEB-INF/web.xml

       

      We can create two separate applications out of this by editting HaJsp.jsp such that it say Application1 or Application2. When this is done, remove the clusterjsp.war file and create two war files:

       

      as1 # ${JAVA_HOME}/bin/jar cvf application1.war *
      added manifest
      adding: ClearSession.jsp(in = 527) (out= 339)(deflated 35%)
      adding: HaJsp.jsp(in = 2623) (out= 1004)(deflated 61%)
      ignoring entry META-INF/
      ignoring entry META-INF/MANIFEST.MF
      adding: WEB-INF/(in = 0) (out= 0)(stored 0%)
      adding: WEB-INF/sun-web.xml(in = 1134) (out= 526)(deflated 53%)
      adding: WEB-INF/web.xml(in = 716) (out= 314)(deflated 56%)

       

      Do the same for application2, leading to application2.war.


      6.2. Deploying application to the cluster

       

      Because the hdscanner is running (look in the ${INSTANCE}/deploy directory for the xml file), deploying to the clusters is easy:

       

      as1 # chown jboss:jboss application1.war application2.war
      as1 # cp -p application1.war ${JBOSS_HOME}/server/node11/farm
      as1 # cp -p application2.war ${JBOSS_HOME}/server/node21/farm

       

      If all goes well, application1.war (for instance) should be copied to the farm/ directory of node12 on the second JBoss-AS as2. Same pattern for application2.


      6.3. Testing of the application

       

      You can always test if the applications are running by using the following commands:

       

      lynx http://192.168.10.0:8080/application1
      lynx http://192.168.10.0:8180/application2

       

      and:

       

      lynx http://192.168.20.0:8080/application1
      lynx http://192.168.20.0:8180/application2

       

      Or use a browser to do just that. Note that you don't use clustering in this test, you just surf to the application at each JBoss-AS instance.


      6.4. Making the document roots of the application

       

      The following task lays ahead for this chapter: Configure virtual hosts for www.application1.nl going to /application1 on JBoss-AS and www.application2.nl going to /application2 on JBoss-AS, where the traffic to (for example) /application1 is send to the JBoss-AS cluster serving application1. First the documentroot:

       

      ws # mkdir -p /apps/www.application1.nl/data
      ws # mkdir -p /apps/www.application2.nl/data
      ws # touch /apps/www.application1.nl/data/index.html
      ws # touch /apps/www.application2.nl/data/index.html
      ws # chown -R jboss:jboss /apps/www.application1.nl/data
      ws # chown -R jboss:jboss /apps/www.application2.nl/data


      ===================================
      CHAPTER 7: Apache mod_cluster 1.0.x
      ===================================

       

      7.1. Downloading the binary mod_cluster release (Redhat):

       

      Go to the redhat support site and download the httpd components for modcluster (these are contained in a zip file with roughly the following name: jboss-ep-native-5.1.0-RHEL5-x86_64.zip. Then:

       

      ws # unzip /tmp/jboss-ep-native-5.1.0-RHEL5-x86_64.zip
      ws # cd /tmp/jboss-ep-5.1/native/lib64
      ws # cp lib* /distr/jboss-ews/httpd/lib
      ws # cp httpd/modules/* /distr/jboss-ews/httpd/modules


      7.2. Making a cluster.conf file

       

      Our httpd.conf reads in other conf files via an "Include conf.d/*.conf" statement in the http.conf file. It is hightly possible that you have an Apache which don't have a conf.d/ directory, but this shouldn't distract you of making one (or just put the statements directly into the httpd.conf. So add, when you don't have it yet, the following to the httpd.conf:

       

      Include conf.d/*.conf

       

      Then make the file ${APACHE_HOME}/conf.d/cluster.conf

       

      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
      LoadModule slotmem_module modules/mod_slotmem.so
      LoadModule manager_module modules/mod_manager.so
      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
      LoadModule advertise_module modules/mod_advertise.so
      LoadModule rewrite_module modules/mod_rewrite.so

       

      UseAlias 1

       

      and put # before these statements when they were already present in the httpd.conf (when you don't do that,
      you'll get warings and errors about doubly loaded modules.

       

      Remark:

       

      The UseAlias 1 is using the <Alias> .. </Alias> tag in the server.xml of the JBoss-AS instance to route the
      treffic back. All these awkward configurations aren't necessary anymore in mod_cluster 1.1.1.


      7.3. Loading some virtual hosts:

       

      We include for each virtual host an include statement for the virtual hosts configurations (which reseides in our system at ${APACHE_HOME}/conf/virtual_hosts/. There are loads of ways to this, each with it's own drawbacks.


      The end of ${APACHE_HOME}/conf/httpd.conf looks like:

       

      NameVirtualHost *:80

      #
      # Include various virtual host configurations
      #

      Include "conf/virtual_hosts/jboss.conf"
      Include "conf/virtual_hosts/application1.conf"
      Include "conf/virtual_hosts/application2.conf


      7.4.  jboss.conf, the first virtual host:

       

      Listen 192.168.0.1:7000

       

      #
      # This virtual host is hit by all the JBoss instances, the port is the same as the
      # port in the proxyList
      #

       

      <VirtualHost 192.168.0.1:7000>
          <Directory />
              Order deny,allow
              Deny from all
              Allow from 192.168
          </Directory>

       

          KeepAliveTimeout 60
          MaxKeepAliveRequests 0

          ManagerBalancerName mycluster
          AdvertiseFrequency 5
          ServerAdvertise Off
          Createbalancers 1
      </VirtualHost>

       

      #
      # The fisrt port 80 host
      #

       

      <VirtualHost *:80>
              ServerName www.zeus.nl
              DocumentRoot /apps/www.zeus.nl/data
              ErrorLog logs/nlptc27b09-error_log
              CustomLog logs/nlptc27b09-access_log common
              LogLevel debug

              <Location /mod_cluster-manager>
                  SetHandler mod_cluster-manager
              </Location>

      </VirtualHost>


      Remark:

      In mod_cluster 1.0.x, the traffic needs a specific port to get the traffix back from the JBoss-AS
      cluster to the JBoss Webserver. This ties closely together with the proxyList and Alias entries on the
      JBoss-AS instances,


      7.5. The Virtual hosts for application1 and application2:

       

      application1.conf:
      =================

       

      <VirtualHost *:80>
          ServerAdmin unix@tntpost.nl
          DocumentRoot /apps/www.application1.nl/data
          ServerName www.application1.nl:80
          ErrorLog logs/www.application1.nl-error_log
          CustomLog logs/www.application1.nl-access_log combined
          LogLevel debug

          ProxyPass / balancer://cluster1/application1 stickysession=JSESSIONID|jsessionid nofailover=On
          ProxyPassReverse / balancer://cluster1/application1

      </VirtualHost>

       

      application2.conf:
      =================

       

      <VirtualHost *:80>
          ServerAdmin unix@tntpost.nl
          DocumentRoot /apps/www.application2.nl/data
          ServerName www.application2.nl:80
          ErrorLog logs/www.application2.nl-error_log
          CustomLog logs/www.application2.nl-access_log combined
          LogLevel debug

          ProxyPass / balancer://cluster2/application2 stickysession=JSESSIONID|jsessionid nofailover=On
          ProxyPassReverse / balancer://cluster2/

      </VirtualHost>

       

      ===================================
      CHAPTER 8: Apache mod_cluster 1.1.1
      ===================================


      8.1. Downloading the binary mod_cluster 1.1.1 release:

       

      Go to http://www.jboss.org/mod_cluster/downloads/1-1-1 and download the appropraite tar archive. The jboss.org package contains an httpd version. We can choose to use this one or copy the needed shared objects to the appropriate directory. We choose the latter. Then:

       

      ws # unzip /tmp/mod_cluster-1.1.1.Final-linux2-x64-ssl.tar.gz
      ws # cd /tmp/opt/jboss/httpd/
      ws # cp lib* /distr/jboss-ews/httpd/lib
      ws # cd httpd/modules
      ws # cp -p mod_advertise.so mod_jk.so mod_manager.so /distr/jboss-ews/httpd/modules
      ws # cp -p mod_proxy_cluster.so mod_slotmem.so /distr/jboss-ews/httpd/modules


      8.2. Making a cluster.conf file

       

      Our httpd.conf reads in other conf files via an "Include conf.d/*.conf" statement in the http.conf file. It is hightly possible that you have an Apache which don't have a conf.d/ directory, but this shouldn't distract you of making one (or just put the statements directly into the httpd.conf. So add, when you don't have it yet, the following to the httpd.conf:

       

      Include conf.d/*.conf

       

      Then make the file ${APACHE_HOME}/conf.d/cluster.conf


      LoadModule proxy_module modules/mod_proxy.so
      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
      LoadModule slotmem_module modules/mod_slotmem.so
      LoadModule manager_module modules/mod_manager.so
      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
      LoadModule advertise_module modules/mod_advertise.so
      LoadModule rewrite_module modules/mod_rewrite.so


      and put # before these statements when they were already present in the httpd.conf (when you don't do that, you'll get warings and errors about doubly loaded modules).


      8.3. Loading some virtual hosts:

       

      We include for each virtual host an include statement for the virtual hosts configurations (which reseides in our system at ${APACHE_HOME}/conf/virtual_hosts/. There are loads of ways to this, each with it's own drawbacks.

       

      The end of ${APACHE_HOME}/conf/httpd.conf looks like:

       

      NameVirtualHost *:80

      #
      # Include various virtual host configurations
      #

      Include "conf/virtual_hosts/jboss.conf"
      Include "conf/virtual_hosts/application1.conf"
      Include "conf/virtual_hosts/application2.conf


      8.4.  jboss.conf, the first virtual host

       

      <VirtualHost *:80>
          <Directory />
              Order deny,allow
              Allow from all
           </Directory>

           <Location /mcm>
               SetHandler mod_cluster-manager
               Order deny,allow
               Allow from 192.168
           </Location>

           KeepAliveTimeout 60
           MaxKeepAliveRequests 0

           ManagerBalancerName testcluster
           AdvertiseFrequency 5
      </VirtualHost>


      Remarks:

       

      o  testcluster is a dummy name, I haven't test the architecture when it's not there, but I think you
         don't have to put it in here.

      o  The mod cluster manager can be reaches under http://192.168.0.1/mcm .


      8.5. The Virtual hosts for application1 and application2:

       

      application1.conf:
      =================

       

      <VirtualHost *:80>
          ServerAdmin unix@tntpost.nl
          DocumentRoot /apps/www.application1.nl/data
          ServerName www.application1.nl:80
          ErrorLog logs/www.application1.nl-error_log
          CustomLog logs/www.application1.nl-access_log combined
          LogLevel debug

          ProxyPass / balancer://cluster1/application1 stickysession=JSESSIONID|jsessionid nofailover=On
          ProxyPassReverse / balancer://cluster1/application1

      </VirtualHost>

       

      application2.conf:
      =================

       

      <VirtualHost *:80>
          ServerAdmin unix@tntpost.nl
          DocumentRoot /apps/www.application2.nl/data
          ServerName www.application2.nl:80
          ErrorLog logs/www.application2.nl-error_log
          CustomLog logs/www.application2.nl-access_log combined
          LogLevel debug

          ProxyPass / balancer://cluster2/application2 stickysession=JSESSIONID|jsessionid nofailover=On
          ProxyPassReverse / balancer://cluster2/

      </VirtualHost>


      ===================================
      CHAPTER 9: Starting up and loggings
      ===================================

       

      Notice we have LogLevel debug though the whole configuration. This enables us to see the MPING and MCPM commands which are used for communication thoughout mod_cluster.


      9.1. Startup sequence:

       

      When you are using JBoss 5 and mod_cluster 1.0.x, it might be needed to first start your webserver (such that the proxyList entry on the JBoss-instances can deliver it's first message) before statring the JBoss-AS instances. I have seen that, when the order is reversed, that the error in the JBoss-AS instance is not recovereable.

       

      When you are using JBoss 5 and mod_cluster 1.1.1., or Jboss 6 (which has mod_cluster 1.1.0), the order to start. JBoss doesn't matter. Do expect to see a connection error when the JBoss-AS backend is started without a webserver.


      9.2. After starting apache:

      ...
      [Mon Jan 24 11:39:27 2011] [debug] mod_proxy_cluster.c(623): update_workers_node done
      [Mon Jan 24 11:39:27 2011] [debug] mod_proxy_cluster.c(604): update_workers_node starting
      [Mon Jan 24 11:39:27 2011] [debug] mod_proxy_cluster.c(623): update_workers_node done
      [Mon Jan 24 11:39:27 2011] [debug] mod_proxy_cluster.c(604): update_workers_node starting
      [Mon Jan 24 11:39:27 2011] [debug] mod_proxy_cluster.c(623): update_workers_node done
      [Mon Jan 24 11:39:34 2011] [debug] mod_proxy_cluster.c(228): Created: worker for ajp://192.168.0.10:8009
      [Mon Jan 24 11:39:34 2011] [debug] mod_proxy_cluster.c(340): proxy: initialized single connection worker 2 in child 17787 for (192.168.0.10)
      [Mon Jan 24 11:39:34 2011] [debug] mod_proxy_cluster.c(381): Created: worker for ajp://192.168.0.10:8009 2 (status): 1
      [Mon Jan 24 11:39:34 2011] [debug] mod_proxy_cluster.c(604): update_workers_node starting
      [Mon Jan 24 11:39:34 2011] [debug] mod_proxy_cluster.c(623): update_workers_node done
      ....

      9.3. After going to an application:

      ...
      [Mon Jan 24 11:42:26 2011] [debug] mod_proxy_cluster.c(1628): cluster: Found value pqMwwCnQX3wP0cRIpX0e1g__.node1 for stickysession JSESSIONID|jsessionid
      [Mon Jan 24 11:42:26 2011] [debug] mod_proxy_cluster.c(1956): cluster: Using route node1
      [Mon Jan 24 11:42:26 2011] [debug] mod_proxy_cluster.c(943): get_balancer_by_node found context /application1
      [Mon Jan 24 11:42:26 2011] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)
      [Mon Jan 24 11:42:26 2011] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://192.168.0.10:8009/application1/
      ...
      [Mon Jan 24 11:42:26 2011] [debug] ajp_header.c(609): ajp_unmarshal_response: ap_set_content_type done
      [Mon Jan 24 11:42:26 2011] [debug] ajp_header.c(599): ajp_unmarshal_response: Header[3] [Content-Length] = [1584]
      [Mon Jan 24 11:42:26 2011] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 03
      [Mon Jan 24 11:42:26 2011] [debug] ajp_header.c(697): ajp_parse_type: got 03
      [Mon Jan 24 11:42:26 2011] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 05
      [Mon Jan 24 11:42:26 2011] [debug] ajp_header.c(697): ajp_parse_type: got 05
      [Mon Jan 24 11:42:26 2011] [debug] mod_proxy_ajp.c(562): proxy: got response from (null) (192.168.0.10)
      [Mon Jan 24 11:42:26 2011] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (192.168.0.10)
      ...


      Of course you also see messages about 192.168.0.20, the second application server.

       

      =========

      ADDENDUM

      =========

       

      There are far more ways to configure mod_cluster than the way you see in this discussion. Anyway, I hope that this little experimentation serves its purpose. Like we said in the discussions, apart from a few minimal examples, there are not so many working examples yet in the mod_cluster documentation.

       

      Message was edited by: Willem Noorduin

        • 1. mod_cluster and multiple virtual hosts
          Willem Noorduin Newbie

          Found a little mistake, when you are starting things up, the reel on as2 must be:

           

          On as2:

           

          as2 # su - jboss

          as2 $ ${JBOSS_HOME}/bin/run.sh -c node12 -g cluster1 -Djboss.messaging.ServerPeerID=1
          -Djboss.service.binding.set=ports-default -Djboss.server.log.dir=/apps/data/node12/logs/
          -Djboss.jvmRoute="node11" -Djboss.Domain=domain1 -Djboss.mod_cluster.proxyList="192.168.0.1:80"
          -Djboss.modcluster.balancer="cluster1" -Djboss.bind.address="192.168.20.0"

           

          as2 $ ${JBOSS_HOME}/bin/run.sh -c node22 -g cluster2 -Djboss.messaging.ServerPeerID=1
          -Djboss.service.binding.set=ports-01 -Djboss.server.log.dir=/apps/data/node22/logs/
          -Djboss.jvmRoute="node21" -Djboss.Domain=domain2 -Djboss.mod_cluster.proxyList="192.168.0.1:80"
          -Djboss.modcluster.balancer="cluster2" -Djboss.bind.address="192.168.20.0"

           

          (Otherwise you have four nodes binding on the same bin address of as1).

          • 2. mod_cluster and multiple virtual hosts
            Jean-Frederic Clere Master

            "============================
            CHAPTER 4: MOD_CLUSTER 1.1.1
            ============================

             

            4.1.    Downloading mod_cluster 1.0.1 Java Bundle:"

             

            I guess you want to say 1.1.1

             

            I like it.

            • 3. mod_cluster and multiple virtual hosts
              Willem Noorduin Newbie

              Thanks, I corrected the mistake just now

              • 4. Re: mod_cluster and multiple virtual hosts
                Andrew Sacamano Newbie

                I've had some issues with the sucker password configurtion as described above - I got NumberFormatExceptions.

                 

                Here's what I had to do to get it to work following (as described at https://bugzilla.redhat.com/show_bug.cgi?id=675016)

                 

                Put the cleartext passowrd in deploy/messaging/messaging-jboss-beans.xml

                 

                Then encode it using

                 

                java -cp jboss-messaging-client.jar org.jboss.messaging.util.SecurityUtil <password>

                 

                and put the encoded text in messaging-service.xml

                 

                I hope this helps others avoid the problems I had.

                • 5. Re: mod_cluster and multiple virtual hosts
                  Dohan Chinthaka Newbie

                  This is a nice guide and thanks for sharing.

                  I followed the guide and jboss configuration is ok and have no problem. I have a problem with starting apache after configuring the "so" files in the httpd.conf.

                   

                  httpd.conf:

                  #LoadModule proxy_module modules/mod_proxy.so

                  #LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

                  LoadModule advertise_module modules/mod_advertise.so

                  LoadModule manager_module modules/mod_manager.so

                  LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

                  LoadModule slotmem_module modules/mod_slotmem.so

                  #LoadModule rewrite_module modules/mod_rewrite.so

                   

                  The error is:

                  httpd: Syntax error on line 116 of /usr/local/apache2/conf/httpd.conf: Cannot load /usr/local/apache2/modules/mod_advertise.so into server: /usr/local/apache2/modules/mod_advertise.so: cannot open shared object file: No such file or directory

                   

                  But the file is there. I am running apache as root.

                   

                  dohanlinux:/usr/local/apache2/bin # ls /usr/local/apache2/modules


                  httpd.exp           mod_authn_dbd.so        mod_authz_owner.so  mod_deflate.so     mod_ident.so         mod_mime_magic.so      mod_proxy_http.so  mod_ssl.so mod_actions.so      mod_authn_dbm.so        mod_authz_user.so   mod_dir.so         mod_imagemap.so      mod_mime.so            mod_proxy_scgi.so  mod_status.so mod_advertise.so    mod_authn_default.so    mod_autoindex.so    mod_dumpio.so      mod_include.so       mod_negotiation.so     mod_proxy.so       mod_substitute.so mod_alias.so        mod_authn_file.so       mod_cern_meta.so    mod_env.so         mod_info.so          mod_proxy_ajp.so       mod_reqtimeout.so  mod_unique_id.so mod_asis.so         mod_authz_dbm.so        mod_cgi.so          mod_expires.so     mod_log_config.so    mod_proxy_balancer.so  mod_rewrite.so     mod_userdir.so mod_auth_basic.so   mod_authz_default.so    mod_dav_fs.so       mod_ext_filter.so  mod_log_forensic.so  mod_proxy_cluster.so   mod_setenvif.so    mod_usertrack.so mod_auth_digest.so  mod_authz_groupfile.so  mod_dav.so          mod_filter.so      mod_logio.so         mod_proxy_connect.so   mod_slotmem.so     mod_version.so mod_authn_anon.so   mod_authz_host.so       mod_dbd.so          mod_headers.so     mod_manager.so       mod_proxy_ftp.so       mod_speling.so     mod_vhost_alias.so

                   

                  Please correct me if I am doing wrong.

                   

                  Message was edited by: Dohan Chinthaka

                  • 6. Re: mod_cluster and multiple virtual hosts
                    Willem Noorduin Newbie

                    Sounds like mod_adverise.so is missing some resources. Try ldd mod_advertise.so for some missing resources, In the mod_cluster package you downloaded, there are probably some libraries that mod_cluster (in your case mod_advertise.so) is needing.

                    • 7. Re: mod_cluster and multiple virtual hosts
                      Dohan Chinthaka Newbie

                      Thanks for your reply. I downloaded the so files from the link you provided and how can I know what resources it is requiring? Is there a documentation or something for it? And further I tried changing the order of so files and still it throws the error with the first so it finds. So isn't there a possibility of another cause?

                      • 8. Re: mod_cluster and multiple virtual hosts
                        Jean-Frederic Clere Master

                        the best guess is that you downloaded .so file that don't correspond to your httpd.

                         

                        try:

                        file mod_authn_dbd.so

                        and

                        file mod_advertise.so

                        • 9. Re: mod_cluster and multiple virtual hosts
                          Dohan Chinthaka Newbie

                          You are correct!!! I have downloadded a wrong version of mod_cluster bundle. I continued with the correct version and now its working smoothly. A big Thank for all. Appreceiate your help.

                           

                          Cheers!!!

                          • 10. Re: mod_cluster and multiple virtual hosts
                            Dohan Chinthaka Newbie

                            Hi, Currently I have secured the communication within Jboss and httpd by adding SSL config parameters in httpd.conf and mod_cluster-jboss-beans.xml.

                             

                            httpd.conf

                            ----------


                            Listen 6666

                            <VirtualHost 10.1.1.12:6666>

                             

                                <Directory />

                                    Order deny,allow

                                    Deny from all

                                    Allow from 10.1.1.

                                </Directory>

                             

                                KeepAliveTimeout 60

                                MaxKeepAliveRequests 0

                             

                                ManagerBalancerName mycluster

                                AdvertiseFrequency 5

                             

                                SSLEngine on

                                SSLCipherSuite AES128-SHA:ALL:!ADH:!LOW:!MD5:!SSLV2:!NULL

                                SSLCertificateFile /usr/local/apache2/ssl/server.crt

                                SSLCertificateKeyFile /usr/local/apache2/ssl/server.key

                                SSLCACertificateFile /usr/local/apache2/ssl/ca.crt

                                SSLVerifyClient require

                                SSLVerifyDepth  10

                             

                            </VirtualHost>

                             

                            mod_cluster-jboss-beans.xml

                            ---------------------------

                            <property name="ssl">true</property>

                            <property name="sslKeyStore">${jboss.server.home.dir}/cert/server.keystore</property>

                            <property name="sslKeyStorePass">xxxxx</property>

                            <property name="sslKeyStoreType">pkcs12</property>

                            <property name="sslTrustStore">${jboss.server.home.dir}/cert/server.keystore</property>

                            <property name="sslTrustStoreType">pkcs12</property>

                            <property name="sslTrustStorePassword">xxxxxx</property>

                             

                            I  want to use https to access my clustered application from the browser too. Can you list  the changes I should do in Apache and Jboss to achieve this? I am using mod_cluster 1.1.1, apache 2.2.19 and jbodd 5.1.0 GA. Thanks in advance.

                            • 11. Re: mod_cluster and multiple virtual hosts
                              Jean-Frederic Clere Master

                              To secure the connection between httpd and the browser look to the mod_ssl docs.

                              Usually there is no need to secure the connection between httpd and JBoss.

                              • 12. Re: mod_cluster and multiple virtual hosts
                                Dohan Chinthaka Newbie

                                Many thanks for the reply. I did it by refering mod_ssl docs

                                 

                                Then I configured my applicaiton for public url (say http://mydomain.com/application1) and facing some problems.

                                When I try to access my application through http://mydomain.com/application1 it redirects me to a local IP (http://10.1.1.200:8080/application1/) and doesn't load the page (checked from outside ip). I am using spring-flex application and I am not using session replication <distributable /> tag. What is wrong with my application. Please help.

                                 

                                Message was edited by: Dohan Chinthaka

                                • 13. Re: mod_cluster and multiple virtual hosts
                                  Willem Noorduin Newbie

                                  If you are using Apache redirection, you are not trusting on mod_cluster to do the job for you. The communication between webserver and application server(s) is done through the mod-cluster module (this has to be configured on both the webserver side (the so's) and the nodes on the application server (the mod-cluster.sar)) over port 8009. So all you have to do is:

                                   

                                  • See if you have the so's (like mod_advertise.so, etc) and the apr's in place (On most servers this is /etc/httpd/modules, but it depends on your OS).
                                  • See if the httpd.conf and the virtual host conf are looking like mentioned above.
                                  • See if you have deployed the mod_cluster,sar (or mod-cluster.sar, the naming depends on the version) in the JBoss profiles under which your application is running.
                                  • See if port 8009 is open between webserver and application server (on your webserver, a simple telnet <application-server> 8009 will suffice). If this does not come back with a prompt, try to do a service iptables stop, or configure iptables in the right way (assuming you are on a **ix box).
                                  • Trust mod_cluster in delivering application calls to the right node. The webserver is trying to match the name after balancer:// (in our case cluster1 with the ${jboss.modcluster.balancer} parameter in the configuration of mod_cluster.sar. At first it does seem like magic, but that's the beauty behind mod_cluster.

                                   

                                   

                                  Also, you can try to debug the architecture by:

                                   

                                  • LogLevel debug (don't forget to turn this of before going into production) in the httpd.conf and virtual host.conf
                                  • See if the application nodes become visible in the mod-cluster-manager.

                                   

                                  Hope these scribles help !

                                  • 14. Re: mod_cluster and multiple virtual hosts
                                    Jean-Frederic Clere Master

                                    you use http instead of ajp, may be.

                                     

                                    Try adding ProxyPreserveHost on

                                     

                                    http://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxypreservehost

                                    1 2 Previous Next