9 Replies Latest reply on Jan 6, 2011 10:56 AM by willemnoorduin

    More JBoss-AS 5 clusters and more Apache Virtual Hosts

    willemnoorduin

      : I have configured two JBoss5-AS clusters (cluster1 and cluster2), which are running on two (Linux) servers. Each cluster contains a number of instances

      which i created on basis of the production instance (not the all instance).

       

      When I setup a seperate Apache webserver with mod_jk, all is working perfectly. I have been reading about mod_cluster and want to implement it because

      it is more versatile than mod_jk. I have successfully implement this for one named virtual host

       

      <VirtualHost *:80>

         ServerAdmin me@host.nl

         DocumentRoot /content/www.test.nl/data

         ServerName www.test.nl

         ErrorLog logs/test-error_log

         CustomLog logs logs/test-access_log combined

       

         <Location /mcm>

               SetHandler mod_cluster_manager

         </Location>

       

          KeepAliveTimeout 60

          MaxKeepAliveRequests 0

       

          ManagerBalancerName testcluster

       

          ProxyPass / balancer://testcluster/

          ProxyPassReverse / balancer://testcluster/

       

         AdvertiseFrequency 10

         AdvertiseGroup 239.255.100.100:60000

       

      </VirtualHost>

       

      (where there can be a type somewhere, because Paste doesn't seem to work here). The AdvertiseGroup address is the multicast address I started

      cluster1 with (so for example ./run.sh -c cluster-node1 -g cluster1 -u 239.255.100.100 ...). If I start the thing up (JBoss nodes and Apache) and I deploy an application, I can see all this in the http://www.test.nl/mcm interface (this is in fact the minimal example named in the documentation of mod_cluster)

       

      The trouble begins when I try to define a second Apache Virtual Host (for example  www.test2.nl and try to attach it to cluster2 (the second cluster). First it

      doesn't seem to be possible to have two AdvertiseGroup alements here (my first hunch was to ser this to the multicast address of the second cluster).

      When I try to throw the mod_cluster configuration in one (default) virtual host, it only sees one cluster (either the one behind 239.255.100.100 or (if I comment

      the AdvertiseGroup command), the first that becomes available

       

      The reason that I want two seperate clusters (and not all applications in one cluster)  is that we want to administer each cluster independently of each other. Question is thus:

       

      How to setup a multiple named virtual host configuration for apache, where each virtual host connects to a different JBoss-AS5 cluster. Can some one give

      a concrete example of such a configuration ?

        • 1. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
          augustsimonelli

          Here's what I wound up doing with our mod_cluster 1.0.4/EAP5.1 setup. I don't use multicast so keep that in mind ...

           

          mod_cluster.conf:

           

          LoadModule slotmem_module modules/mod_slotmem.so
          LoadModule manager_module modules/mod_manager.so
          LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
          LoadModule advertise_module modules/mod_advertise.so

           

          Listen 127.0.0.1:6666
          <VirtualHost 127.0.0.1:6666>

           

              <Directory />
                  Order deny,allow
                  Deny from all
                  Allow from 127.0.0.1
              </Directory>

           

              KeepAliveTimeout 60
              MaxKeepAliveRequests 0

           

              ManagerBalancerName mycluster
              AdvertiseFrequency 5
              ServerAdvertise Off

           

              Createbalancers 1

           


          </VirtualHost>

           

          "Createbalancers 1" means we can don't automatically have the balancer in all virtuals, which i like cause it means i can have non-mod-clustered virtuals for static files/other sites/etc. ALL JBoss instances hit this 6666 port (in this case it's on localhost but obviously in reality you'd need it exposed).

           

          0.conf:

           

          <VirtualHost *:80>
                  ServerName redcloud.local
                  DocumentRoot /var/www/html/redcloud/
                  ErrorLog logs/redcloud.local-error_log
                  CustomLog logs/redcloud.local-access_log common
                  LogLevel debug

           

                  <Location /mod_cluster-manager>
                      SetHandler mod_cluster-manager
                      Order deny,allow
                      Deny from all
                      Allow from 127 192
                  </Location>

           

          </VirtualHost>

           

          a hacky way to ensure the default virtual is this one. i like this cause it allows me to use it as my "working" virtual with my clustermanager and such. i tend to attach my hostname to it cause that's logical in my mind.

           

          virtual.conf

           

          NameVirtualHost *:80

           

          <VirtualHost *:80>

           

                  ServerName modcluster.redcloud.local
                  DocumentRoot /var/www/html/cluster
                  ErrorLog logs/modcluster.redcloud.local-error_log
                  CustomLog logs/modcluster.redcloud.local-access_log common
                  LogLevel debug

           

                  ProxyPass / balancer://ogbalancer/sample stickysession=JSESSIONID|jsessionid
                  ProxyPassReverse / balancer://ogbalancer/sample

           

                  #RewriteEngine on

           

                  #RewriteCond %{REQUEST_URI} !/mod_cluster-manager
                  #RewriteRule ^/(.*)$ balancer://ogbalancer/sample/$1 [P,L]
                  #RewriteRule ^/(.*)$ balancer://ogbalancer/$1 [P,L]
                  #RewriteRule ^/sample/(.*)$ balancer://ogbalancer/sample/$1 [P,L]

           

          </VirtualHost>

           

          because of "createbalancers 1" each virtual needs the balancer created with a proxypass line. So in this example I create ogbalancer for this virtual only. On the jboss side, besides the jbossweb changes needed by 1.0.4, i add

           

          <property name="balancer">ogbalancer</property>

           

          to ./deploy/mod-cluster.sar/META-INF/mod-cluster-jboss-beans.xml

           

          This links the jboss instance to that virtual only. If i have another virtual it won't pass anything to ogbalancer unless i give it a proxypass line for that balancer within the virtual.

           

          Now, i can stand up another jboss instance, connect it to the same 6666 port (in my case with jboss.modcluster.proxyList) as all the others, set ITS balancer name:

           

          <property name="balancer">someotherbalancer</property>

           

          And then call that balancer from another virtual host with its own proxypass line. Now all traffic for this virtual goes to that JBoss instance, and all traffic for my example virtual goes to ogbalancer.

           

          Should work fine with mutlicast; we just have network implications for using that.

           

          August

           

          BTW ... i left the rewrite stuff i was using in as it does work, but i  think the proxypass stuff makes more sense. plus when it comes to  passing querystrings with mod_rewrite i ran into this:  http://community.jboss.org/message/521202 ...

          1 of 1 people found this helpful
          • 2. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
            willemnoorduin

            I can get one virtual host and one JBoss-AS cluster to work. When I do this, I get the mod-cluster-maneger information that there are applications available at ajp://192.168.0.1:8009 and ajp://192.168.0.2:8009 (which is the port on which my first JBoss-AS cluster is listening. The application is also available

             

            When I hookup a second virtual host in Apache, I get information in mod-cluster-manager for ajp://192.168.0.1:8109 and ajp://192.168.0.2:8109 (which is the port of the second JBoss-AS cluster). Now the second application is available. The first application (on port 8009) is not visible in mod-cluster-manager anymore and the application is unavailable.

             

            Any ideas what causes this ?

             

            For your information:

             

            • application1 is running on the cluster consisting of 192.168.0.1:8009 and 192.158.0.2:8009 JBoss-AS
            • application2 is running on the cluster consisting of 192.168.0.1:8109 and 192.158.0.2:8109 JBoss-AS

            Any idea if the IP-addresses of the cluster must be differen too ?

            • 3. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
              augustsimonelli

              When you say you are adding a second virtual host be sure you aren't adding another mod_cluster virtual; you only need one of those (ie only one listening on, say, port 6666). Both JBoss clusters should talk back to that virtual. To be 100% sure the jboss nodes are all hitting the same port start them up with: jboss.modcluster.proxyList=1.2.3.4:6666,1.2.3.5:6666 (replacing witht he correct IPs of course). This will avoid the multicast and let you know what's up. Post your mod_cluster virtual and the two apache virtuals too if you like!

              • 4. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
                willemnoorduin

                Okay, here it goes (sorry for the length of this post ():

                 

                We have three seperate servers in our setup:

                 

                192.168.0.1 webserver
                192.168.0.2 appserver1
                192.168.0.3 appserver2

                 

                We configured the DNS such that www.application1.nl and www.application2.nl are landing on the webserver. The aim is to get application contexts /application1 and /application2 mapped to there own cluster:

                 

                cluster1 contains JBoss instances application1_i1, application1_i2, running on respectively

                appserver1 and appserver2, and reacheable via http://appserver1:8009/application1

                 

                cluster2 contains JBoss instances application2_i1, application2_i2, running on respectively

                appserver1 and appserver2, and reacheable via http://appserver2:8109/application2

                 

                -------------------------------------------------------------------------------------------

                On webserver, we have the following configuration items:

                In the first conf (placed in the conf.d directory) the modules needed for mod_cluster are
                loaded into httpd:

                 

                $ cat mod_cluster.conf

                LoadModule proxy_module modules/mod_proxy.so
                LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
                LoadModule slotmem_module modules/mod_slotmem.so
                LoadModule manager_module modules/mod_manager.so
                LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
                LoadModule advertise_module modules/mod_advertise.so

                At the end of the httpd.conf, we load the conf files of the virtual host. We do this to
                establish an order (naming them 0.conf etc works in the same way):


                $ tail httpd.conf

                Include "conf/virtual_hosts/jboss.conf"
                Include "conf/virtual_hosts/default.conf"
                Include "conf/virtual_hosts/application1conf"
                Include "conf/virtual_hosts/application2.conf"


                The vitrual host confs are the following:

                $ cat jboss.conf

                Listen 192.168.0.1:7777

                <VirtualHost 192.168.0.1:7777>
                    KeepAliveTimeout 60
                    MaxKeepAliveRequests 0

                    ManagerBalancerName mycluster
                    AdvertiseFrequency 5
                    ServerAdvertise Off

                    Createbalancers 1
                </VirtualHost>

                 

                $ cat default.conf

                <VirtualHost *:80>
                    ServerAdmin unix@tntpost.nl
                    DocumentRoot /apps/nlptc27b14/data
                    ServerName nlptc27b14.nlptc27.post.tnt:80
                    ErrorLog logs/nlptc27b14-error_log
                    CustomLog logs/nlptc27b14-access_log combined
                    LogLevel debug

                    # mod_cluster defaults

                    <Location /mcm>
                       SetHandler mod_cluster-manager
                    </Location>

                </VirtualHost>

                 

                $ cat application1.nl.post.tnt.conf

                <VirtualHost *:80>
                    ServerAdmin unix@tntpost.nl
                    DocumentRoot /apps/www.application1.nl/data
                    ServerName www.application1.nl:80
                    ErrorLog logs/www.application1.nl-error_log
                    CustomLog logs/www.application1.nl-access_log combined
                    LogLevel debug

                    ProxyPass / balancer://cluster1/application1 stickysession=JSESSIONID|jsessionid

                nofailover=On
                    ProxyPassReverse / balancer://cluster1/application1

                </VirtualHost>


                $ cat application2.nl.post.tnt.conf

                 

                <VirtualHost *:80>
                    # Begin rewriting TRACK/TRACE
                    RewriteEngine on
                    RewriteCond %{REQUEST_METHOD} ^TRACK
                    RewriteRule .* - [F]
                    RewriteCond %{REQUEST_METHOD} ^TRACE
                    RewriteRule .* - [F]
                    # End rewriting TRACK/TRACE

                    ServerAdmin unix@tntpost.nl
                    DocumentRoot /apps/www.application2.nl/data
                    ServerName www.application2.nl:80
                    ErrorLog logs/www.application2.nl-error_log
                    CustomLog logs/www.application2.nl-access_log combined

                    ProxyPass / balancer://cluster2/application2 stickysession=JSESSIONID|jsessionid

                nofailover=On
                    ProxyPassReverse / balancer://cluster2/

                </VirtualHost>

                 

                -----------------------------------------------------------------------------------------

                 

                On appserver1. we have (for example) instance application1_i1, which is primarily a clone of the production instance (on Redhat JBoss 5, this is the same as a clone of the all instance). The following items are changed:

                 

                1. mod-cluster.sar is deployed in application1_i1.

                 

                2. application1 is deployed in application1_i1.

                 

                3. In the mod-cluster.sar/META-INF/mod-cluster-jboss-beans.xml we added the following
                propeties to the "HAModClusterConfig" and the "ModClusterService":

                    <property name="balancer">cluster1</property>
                    <property name="proxyList">${jboss.modcluster.proxyList:}</property>

                The variable ${jboss.modcluster.proxyList:} is given during startup via a -D option.

                 

                4. In the jbossweb.sar/server.xml we added the following things:

                 

                a. Added the following listener:

                   <Listener

                className="org.jboss.web.tomcat.service.deployers.MicrocontainerIntegrationLifecycleListener" delegateBeanName="HAModClusterService"/>

                 

                b. added jvmRoute="${jboss.jvmRoute}" to the jboss.web Engine entry

                The variable ${jboss.jvmRoute} is given during startup via a -D option.

                --------------------------------------------------------------------------------------------

                On appserver1, we have instance application2_i1, which is the same as application1_i1. The only difference with application1_i2 is that we deploy application2 in this instance.

                --------------------------------------------------------------------------------------------

                On appserver2 we have the same configuration, only now that instances are called application1_i2 and application2_i2

                --------------------------------------------------------------------------------------------

                On appserver1, we can now start (for example) instance application1_i1 as follows:

                 

                cd /opt/jboss/jboss-as/bin; /opt/jboss/jboss-as/bin/run.sh -c application1_i1 -b appserver1 -g

                cluster1 -u 239.255.100.100 -m 60100 -Djboss.messaging.ServerPeerID=0

                -Djboss.service.binding.set=ports-default -Djboss.server.log.dir=/apps/data/test/fbewbrpp/logs/

                -Djboss.jvmRoute="appserver1" -Djboss.Domain=cluster1

                -Djboss.modcluster.proxyList="192.168.0.1:7777"

                 

                On appserver2, we can now start (for example) instance application1_i2 as follows:

                 

                cd /opt/jboss/jboss-as/bin; /opt/jboss/jboss-as/bin/run.sh -c application1_i2 -b appserver2 -g

                cluster1 -u 239.255.100.100 -m 60100 -Djboss.messaging.ServerPeerID=1

                -Djboss.service.binding.set=ports-default -Djboss.server.log.dir=/apps/data/test/fbewbrpp/logs/

                -Djboss.jvmRoute="appserver1" -Djboss.Domain=cluster1

                -Djboss.modcluster.proxyList="192.168.0.1:7777"

                 

                Now these form one cluster. We can do the same with cluster2 = (application2_i1,

                application2_i2).

                -------------------------------------------------------------------------------------------

                Problem is still: When I run the configuration with 1 apache virtual host (www.application1.nl), then www.application1.nl/application1 is available and the cluster manager at http://webserver/mcm give two active nodes.

                 

                When I run the configuration with 2 apache virtual hosts (www.application1.nl and www.application2.nl), then www.application1.nl/application1 is unavailable (Temporary unavailable while lynx http://appserver1:8080/application1 from the webserver gives the application1), and
                www.application2.nl/application2 is available. In this case I see two nodes in the cluster manager (I expect 4) and not all the information there can I call reliable.


                • 5. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
                  augustsimonelli

                  Still working through your configs ... but one thing i noticed, which may just be a typo in your post ... is that you have the same jvmroute:

                   

                  Djboss.jvmRoute="appserver1"

                   

                  for both sides of the cluster ... that should be different.

                   

                  Aplogies if that's just a typo in the post - just trying to shake out any obvious funnies.

                  • 6. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
                    augustsimonelli

                    also, when you start up the "2_" other nodes on appserver1 and appserver2 (2_i1 and 2_i2) i assume they are on ports-01 or somethign? does that show properly in the mod_cluster-manager (ie you see ajp 8109)? I've not done it with ports stuff as i find using IPs easier (we just create virtual nics on each host, one for each jboss instance).

                    • 7. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
                      willemnoorduin
                      • The jvmRoute is set via an environment variable jboss.jvmRoute, which is set in our startscript to be the hostename of the server. Since these are different, the appserver1 entry in jvmRoute is a typo (I had to edit company stuff out). In our production test case, the jvmRoutes are different
                      • We are indeed using ports-default and ports-01 (ports-02, ports-03, etc) settings when starting up the JBoss-AS instances, so that you have http://appserver1:8080 and http://appserver2:8180 available

                       

                      The resaon we cannot use virtual nics is that we don't have that many IP's available (the 192.68.x.y is of course an example here, in real life we have to use routable IP-addresses). Questions are thus:

                       

                      1. Have you tried to make the configuration work (or not work) with the ports-?? options from JBoss, or just with each cluster it's own IP-address ?
                      2. Is it possible to make a configuration we are using to work ? I find it strange if it's not, bacause the whole mod_cluster / JBoss ports mechanism is designed in the first place to make this easier.
                      3. Is there any other documentation available (other than getting the minimal-configuratiun to work) ?
                      • 8. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
                        augustsimonelli

                        Nope - i never did it with the ports configs buti agree it should work ...

                        What version of apache are you using? And what version of mod_cluster? Is it the RHN one?

                        • 9. Re: More JBoss-AS 5 clusters and more Apache Virtual Hosts
                          willemnoorduin

                          We have downloaded JBoss-EWS (Enterprise Webserver from Redhat). From what I could see it is just a bundle of Apache 2.2.17 and Tomcat 5.5.x ot Tomcat 6.0.y. The version of mod_cluster I think is 1.0, because we still have the bug that needs ServerName www.application1.nl:80 instead of www.application1.nl The mod_cluster is the RHN one, yeah.