11 Replies Latest reply: Aug 4, 2009 11:13 AM by Chris Lowe RSS

    Node applications not registering correctly on initial launc

    Chris Lowe Apprentice

      Hello,

      I've seen some unexpected behaviour a few times with mod_cluster. I have 2 nodes both of which have an application deployed using all/farm. Both nodes have been shut down along with httpd. So from a standing start, I launch httpd quickly followed by the nodes and I eventually get the following output from mod_cluster-manager:

      Auto Refresh
       Node JvmRoute1 (ajp://123.234.123.100:8009): Enable Contexts Disable Contexts
      
      
      Balancer: mycluster,Domain: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 77
      Node JvmRoute2 (ajp://123.234.123.101:8009): Enable Contexts Disable Contexts
      
      Balancer: mycluster,Domain: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 78


      The nodes have registered with mod_cluster but the applications are missing. At this point both nodes have completely initialised and I have even left a few minutes for good measure. If I then restart httpd, I then almost instantly get what I was expecting in the first place:

      Auto Refresh
       Node JvmRoute1 (ajp://123.234.123.100:8009): Enable Contexts Disable Contexts
      
      
      Balancer: mycluster,Domain: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 76
       Virtual Host 1:
      
      Contexts:
      
      /, Status: ENABLED Disable
      Aliases:
      
      cluster.domain.com
      
       Node JvmRoute2 (ajp://123.234.123.101:8009): Enable Contexts Disable Contexts
      
      
      Balancer: mycluster,Domain: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 78
       Virtual Host 1:
      
      Contexts:
      
      /, Status: ENABLED Disable
      Aliases:
      
      cluster.domain.com


      I'm using mod_cluster 1.0.1.GA in unicast mode.

      Any ideas on what might be causing this?

      Cheers,

      Chris.

        • 1. Re: Node applications not registering correctly on initial l
          Chris Lowe Apprentice

          Another slight variation on this is that I gracefully shutdown my nodes and restart them. I get the same as above: mod_cluster-manager lists the nodes but not the applications.

          Without restarting httpd or the nodes, I then deploy an application to all/farm and soon the applications are listed again in mod_cluster-manager (and the application functions fine).

          • 2. Re: Node applications not registering correctly on initial l
            Jean-Frederic Clere Master

            Look to http://www.jboss.org/mod_cluster/java/properties.html#proxy
            try to set excludedContexts (the default value contains ROOT).

            • 3. Re: Node applications not registering correctly on initial l
              Chris Lowe Apprentice

              Hi Jean-Frederic,

              Hmm, that doesn't quite work:

              Node JvmRoute2 (ajp://123.234.123.101:8009): Enable Contexts Disable Contexts
              
              
              Balancer: mycluster,Domain: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 76
               Virtual Host 1:
              
              Contexts:
              
              /, Status: ENABLED Disable
              Aliases:
              
              cluster.domain.com
              
               Node JvmRoute1 (ajp://123.234.123.100:8009): Enable Contexts Disable Contexts
              
              
              Balancer: mycluster,Domain: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 1,Ttl: 60000000,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: -1
               Virtual Host 1:
              
              Contexts:
              
              /, Status: ENABLED Disable
              Aliases:
              
              localhost


              Notice that the alias is localhost and not cluster.domain.com.

              If the excludeContexts value is really at play here, wouldn't my context have always been blocked?

              Cheers,

              Chris.


              PS - I notice in the documentation that the default excludeLists doesn't list admin-console in 5.1; I suspect it is excluded since I'm not seeing it and the docs need a tweak?


              • 4. Re: Node applications not registering correctly on initial l
                Jean-Frederic Clere Master

                "Hmm, that doesn't quite work: ", well in the output you have / in both nodes, that looks ok for me.
                I can't reproduce it which version of AS and mod_cluster are you using?

                "Notice that the alias is localhost and not cluster.domain.com." well you have probably localhost in server.xml.

                "admin-console" that is MODCLUSTER-87. (fixed)

                • 5. Re: Node applications not registering correctly on initial l
                  Chris Lowe Apprentice

                  Hi Jean-Frederic,

                  I'm using the following:

                  mod_cluster 1.0.1.GA, linux2 x64 build;
                  JBoss AS 5.1.0.GA-jdk6.

                  Behind mod_cluster I have 2 nodes configured as a cluster via a TCP JGroups stack.

                  Yes, my server.xml does define my host name as "localhost". My application is bound to the root context via the following jboss-web config:

                  <jboss-web>
                  ...
                   <context-root>/</context-root>
                   <virtual-host>cluster.domain.com</virtual-host>
                  </jboss-web>



                  You're right with the excludeLists overridden to allow ROOT, mod_cluster-manager does show an application whose context is root. However, given that the Alias is now "localhost", which application is mod_cluster picking up events for - my application or ROOT.war? If I restart httpd, then aliases are listed for both cluster.domain.com and localhost, so that would suggest that ROOT.war?

                  I tried this experiment: with ROOT removed from the excludeLists, I undeployed my application from the cluster and the ROOT.war application is served. If I then deploy my application, then that is served from the same URL.

                  During this experiment, mod_cluster-manager reports only the localhost alias for each node. After my application is redeployed, I restarted httpd and I given Aliases for both localhost and cluster.domain.com.

                  Isn't this behaviour is undesirable? What if I add another application bound to root and with a different virtual-host, which application would win? My expectation is that several applications should be able to exist, each bound to a root context but with different virtual-host names. As applications are deployed and undeployed, mod_cluster should detect events from each of the aliases and handle requests accordingly. Have I got that wrong? Or is that beyond the scope of mod_cluster?

                  Although I intend to lock down the JBoss instance by removing a good number of these services (and restricting the ones I want to keep to localhost) it would be good to prevent requests ever getting through from Apache.

                  Basically, I'd like mod_cluster to only proxy for those contexts and aliases listed by mod_cluster-manager. The UseAlias option sounds like what I need here, so I tried setting UseAlias to 1 as shown in the Apache HTTPD conf section of the docs, but I get the following error from Apache:

                  Invalid command 'UseAlias', perhaps misspelled or defined by a module not included in the server configuration


                  Even if this option was to work, the fact that certain events from my aliased application are not getting through would cause problems?

                  Finally, regardless of the whether excludeLists has been overridden or not, if I bypass VirtualHosts by accessing my Apache server directly using the IP, I can get to all of the JBoss internals: jmx-admin, admin-console - everything. I've specified the "CreateBalancers 1" option in my httpd.conf, shouldn't that mean that proxying only occurs where I have explictly specified a ProxyPass directive for a VirtualHost? Or have I misunderstood that?

                  Cheers,

                  Chris.

                  • 6. Re: Node applications not registering correctly on initial l
                    Jean-Frederic Clere Master

                    I have created a JIRA MODCLUSTER-89

                    • 7. Re: Node applications not registering correctly on initial l
                      Chris Lowe Apprentice

                      Hi Jean-Frederic, ,

                      That's good, it'll be great to get one fixed. Thanks for investigating this, your time is greatly appreciated!

                      Can you comment on the error message I'm getting for specifying the UseAlias parameter and that all traffic is proxied if accessing the server directly by IP?

                      Let me know if I can be of any assistance with testing etc.

                      Cheers,

                      Chris.

                      • 8. Re: Node applications not registering correctly on initial l
                        Jean-Frederic Clere Master

                        UseAlias is working for me. Are you really using 1.0.1.GA?

                        I don't understand what you do with IP, it looks like you are accessing jboss directly.

                        • 9. Re: Node applications not registering correctly on initial l
                          Chris Lowe Apprentice

                          Hi Jean-Frederic,

                          I am using 1.0.1.GA. I've elaborated on the UseAlias and direct IP access issues below.

                          Cheers,

                          Chris.


                          UseAlias

                          From the docs:

                          UseAlias: UseAlias - Check that the Alias corresponds to the ServerName (See Host Name Aliases)
                          0: Don't (ignore Aliases)
                          1: Check it
                          Default: 0 Ignore the Alias information from the nodes.


                          In my httpd.conf, I have the following line to tell mod_cluster to check match Alias with my virtual hosts ServerName:

                          UseAlias 1


                          But when I try and start HTTPD, I get the following error:

                          Invalid command 'UseAlias', perhaps misspelled or defined by a module not included in the server configuration



                          Direct IP

                          In my http.conf I have defined one virtual host:

                          <VirtualHost *:80>
                           ServerName cluster.domain.com
                          
                           ProxyPass / balancer://mycluster/
                          </VirtualHost>
                          
                          CreateBalancers 1


                          From the docs:
                          CreateBalancers: Define how the balancer are created in the httpd VirtualHosts, this is to allow directives like:

                          ProxyPass / balancer://mycluster1/
                          0: Create in all VirtualHosts defined in httpd.
                          1: Don't create balancers (requires at least one ProxyPass/ProxyPassMatch to define the balancer names).
                          2: Create only the main server.
                          Default: 2


                          I was expecting the "CreateBalancers 1" directive to prevent proxying unless I specify ProxyPass explicitly for a virtual host.

                          If I access my server using the host name, then requests for http://cluster.domain.com/ result in my application being served. So far so good.

                          However, if I access my web server directly using the IP - http:// 234.123.234.99/ (i.e. missing HTTP host header) then the JBoss status page is served and even though I have not overridden the excludeList to allow ROOT. In fact I can access all JBoss applications - jmx-console etc.

                          Is that the expected behaviour for mod_proxy? If so, how can I lock down my server to only allow traffic via the appropriate virtual hosts setting?

                          Cheers,

                          Chris.



                          • 10. Re: Node applications not registering correctly on initial l
                            Chris Lowe Apprentice

                            BTW - is there a way that I can tell that I'm actually using 1.0.1.GA?

                            • 11. Re: Node applications not registering correctly on initial l
                              Chris Lowe Apprentice

                              Hi Jean-Frederic,

                              My apologies - after running an md5sum comparison I found that one of the .so objects had not been copied when I upgraded to 1.0.1.GA.

                              The UseAlias directive is no longer causing an error and this prevents the issue of accessing direct IP.

                              Cheers,

                              Chris.