6 Replies Latest reply on Aug 11, 2011 9:50 AM by lexsoto

    Sticky sessions not working

    lexsoto

      Hello,

       

      We are running into a problem where sticky sessions are not being honored by mod-cluster, or so we think.

      Our environment:

       

      •   Apache/2.2.16
      • JBoss 5.1.0.GA
      • ModCluster: 1.1.2.Final

       

      Our web app is deployed in 2 JBoss servers:

       

      1. jvmRoute: "gsuat01" with IP 10.151.4.210 
      2. jvmRoute: "gsuat02" with IP 10.151.4.211

       

      Enabling header loggins on the JBoss servers show

      2011-08-08 15:00:55,027 TRACE [org.jboss.web.tomcat.service.session.JvmRouteValve] (ajp-0.0.0.0-8009-34) handleJvmRoute(): We have detected a failover with different jvmRoute. old one: gsuat02 new one: gsuat01. Will reset the session id.

       

       

      and intermittently:

       

      2011-08-08 15:00:55,850 WARN  [org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.CacheListener] (Incoming-7,10.151.4.210:46709) Possible concurrency problem: Replicated version id 86 is less than or equal to in-memory version for session USER01

       

       

      The logs from httpd show that the wrong server is selected:

       

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy_cluster.c(1839): cluster: Found value Gy1807FDGdCfgW34aGU2ow__.gsuat02 for stickysession JSESSIONID|jsessionid

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy_cluster.c(1839): cluster: Found value Gy1807FDGdCfgW34aGU2ow__.gsuat02 for stickysession JSESSIONID|jsessionid

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy_cluster.c(1441): proxy: byrequests balancer DONE (ajp://10.151.4.210:8009)

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://10.151.4.210:8009/

      [Tue Aug 09 10:25:20 2011] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (10.151.4.210)

      [Tue Aug 09 10:25:20 2011] [debug] proxy_util.c(2055): proxy: connecting ajp://10.151.4.210:8009/ to 10.151.4.210:8009

      [Tue Aug 09 10:25:20 2011] [debug] proxy_util.c(2153): proxy: connected / to 10.151.4.210:8009

      [Tue Aug 09 10:25:20 2011] [debug] proxy_util.c(2308): proxy: AJP: fam 2 socket created to connect to 10.151.4.210

       

       

      Notice the session id refers to "gsuat02" but the request is being sent (wrongly) to 10.151.4.210 which corresponds to "gsuat01"

       

      Our httpd configuration is:

       

      LoadModule proxy_module modules/mod_proxy.so

      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

      LoadModule slotmem_module modules/mod_slotmem.so

      LoadModule manager_module modules/mod_manager.so

      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

       

      <VirtualHost *:80>

          ProxyPass / balancer://internalCluster/

       

           <Location /mod_cluster-manager>

              SetHandler mod_cluster-manager

              Order deny,allow

              Deny from all

              Allow from 10.151.4.

          </Location>

      </VirtualHost>

       

      Maxsessionid 250

      Maxnode 75

      Maxhost 75

      UseAlias 1

       

      Listen 10.151.4.160:6666

      <VirtualHost 10.151.4.160:6666>

          ErrorLog logs/mod_cluster-error_log

          CustomLog logs/mod_cluster-access_log combined

       

          <Directory />

              Order deny,allow

              Deny from all

              Allow from 10.151.4.

          </Directory>

       

          ManagerBalancerName internalCluster

       

          KeepAliveTimeout 60

          MaxKeepAliveRequests 0

          LogLevel debug

      </VirtualHost>

       

       

      and in our mod-cluster configuration at the JBoss side we have:

       

      <bean name="ModClusterConfig" class="org.jboss.modcluster.config.ha.HAModClusterConfig" mode="On Demand">

       

          <property name="proxyList">${jboss.mod_cluster.proxyList:}</property>

          <property name="advertise">false</property>

          <property name="advertiseGroupAddress">${jboss.mod_cluster.advertise.address,jboss.partition.udpGroup:224.0.1.105}</property>

          <property name="advertisePort">${jboss.mod_cluster.advertise.port:23364}</property>

          <property name="excludedContexts">${jboss.mod_cluster.excludedContexts:JBOSS,localhost:ROOT,admin-console,invoker,jbossws,jmx-console,juddi,web-console}</property>

          <property name="autoEnableContexts">${jboss.mod_cluster.autoEnableContexts:true}</property>

          <property name="ssl">false</property>

          <property name="stickySession">true</property>

          <property name="stickySessionForce">false</property>

          <property name="stickySessionRemove">false</property>

          <property name="maxAttempts">1</property>

          <property name="workerTimeout">-1</property>

          <property name="nodeTimeout">-1</property>

        </bean>

       

       

      Mod cluster manager DUMP output page shows both JBoss servers:

       

      Node: [6],Name: gsuat02,Balancer: internalCluster,LBGroup: ,Host: 10.151.4.211,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 1,Ttl: 60,Elected: 253,Read: 4828840,Transfered: 547741,Connected: 0,Load: 82

       

      Node: [13],Name: gsuat01,Balancer: internalCluster,LBGroup: ,Host: 10.151.4.210,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 1,Ttl: 60,Elected: 264,Read: 7219247,Transfered: 800564,Connected: 0,Load: 78

       

       

      Any help will be appreciated.  Are we doing anything wrong? Anybody running into the same issue out there?

       

      Thanks in advance,

      Alex Soto

        • 1. Re: Sticky sessions not working
          jfclere

          All looks OK.

          Could you look to the CONFIG messages in the debug of error_log?

          • 2. Re: Sticky sessions not working
            lexsoto

            Thanks for the reply.  Here is what I grepped from the error log:

             

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:14:06 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:14:06 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:14:06 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

             

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

             

             

            Let me know if you need to see anything else, we have spent a lot of time trying to figure out what's wrong.

            • 3. Re: Sticky sessions not working
              lexsoto

              Jean-Frederic:

               

              We managed to recompile the mod-cluster with HAVE_CLUSTER_EX_DEBUG setting.    Colored in red is where I think it's getting confused.  First, it finds the route "gsuat01," later it says "No route found," then it sends the request to the wrong server.  I hope it helps.

               

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1982): proxy_cluster_trans for 0 (null) (null) uri: / args: (null) unparsed_uri: /

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1839): cluster: Found value npK2tiLnn64BPKQ2xgwLpA__.gsuat01 for stickysession JSESSIONID|jsessionid

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1847): cluster: Found route gsuat01

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1794): find_nodedomain: finding node for gsuat01: internalcluster

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1807): find_nodedomain: finding domain for gsuat01: internalcluster

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1839): cluster: Found value npK2tiLnn64BPKQ2xgwLpA__.gsuat01 for stickysession JSESSIONID|jsessionid

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1847): cluster: Found route gsuat01

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1794): find_nodedomain: finding node for gsuat01: internalcluster

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1807): find_nodedomain: finding domain for gsuat01: internalcluster

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2040): proxy_cluster_trans using internalCluster uri: proxy:balancer://internalCluster/

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2073): proxy_cluster_canon url: //internalCluster/

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2424): proxy_cluster_pre_request: url balancer://internalcluster/

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2212): cluster:No route found

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1359): proxy: Entering byrequests for CLUSTER (balancer://internalcluster)

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1441): proxy: byrequests balancer DONE (ajp://10.151.4.211:8009)

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2587): proxy_cluster_pre_request: balancer (balancer://internalcluster) worker (ajp://10.151.4.211:8009) rewritten to ajp://10.151.4.211:8009/

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://10.151.4.211:8009/

              [Wed Aug 10 11:30:57 2011] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (10.151.4.211)

              [Wed Aug 10 11:30:57 2011] [debug] proxy_util.c(2055): proxy: connecting ajp://10.151.4.211:8009/ to 10.151.4.211:8009

              [Wed Aug 10 11:30:57 2011] [debug] proxy_util.c(2153): proxy: connected / to 10.151.4.211:8009

              [Wed Aug 10 11:30:57 2011] [debug] proxy_util.c(2308): proxy: AJP: fam 2 socket created to connect to 10.151.4.211

              [Wed Aug 10 11:30:57 2011] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong

              [Wed Aug 10 11:30:57 2011] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done

               

               

               

               

              Thanks for your help so far,

              Alex Soto

              • 4. Re: Sticky sessions not working
                lexsoto

                Based on the previous log output and going over the code, it looks like the route value is being overwritten. 

                In line 1859 of "mod_proxy_cluster.c" this value is set in "r->notes"

                                    apr_table_setn(r->notes, "session-route", route);

                Later on, line 2205 tries to retrieve the value:

                 

                    /* We already should have the route in the notes for the trans() */

                    *route = apr_table_get(r->notes, "session-route");

                 

                but it's no there anymore. Does this seem plausible? 

                • 5. Re: Sticky sessions not working
                  jfclere

                  In fact it doesn't find the balancer "internalcluster" .. . Because it probably has "internalCluster" (Note the c/C).

                  It doesn't set the route because find_nodedomain returns APR_NOTFOUND

                  See the message "find_nodedomain: finding domain for gsuat01: internalcluster".

                   

                  Try to use internalcluster instead internalcluste

                  • 6. Re: Sticky sessions not working
                    lexsoto

                    I am very happy to inform that it is working now.  However, it seems odd that it would not work when the balancer name used camel casing. 

                    All I did was to change "internalCluster" to "internalcluster" and voilà!, it worked.

                     

                    The documentation does not mention any kind of restriction on this value:

                     

                    http://docs.jboss.org/mod_cluster/1.1.0/html/native.config.html#d0e596

                     

                    I have also verified that we used the exact same casing in all places we had this name.  Almost looks like a bug to me.

                     

                    In any case, thanks a lot for the help.