6 Replies Latest reply on Aug 11, 2011 9:50 AM by alex soto

    Sticky sessions not working

    alex soto Newbie

      Hello,

       

      We are running into a problem where sticky sessions are not being honored by mod-cluster, or so we think.

      Our environment:

       

      •   Apache/2.2.16
      • JBoss 5.1.0.GA
      • ModCluster: 1.1.2.Final

       

      Our web app is deployed in 2 JBoss servers:

       

      1. jvmRoute: "gsuat01" with IP 10.151.4.210 
      2. jvmRoute: "gsuat02" with IP 10.151.4.211

       

      Enabling header loggins on the JBoss servers show

      2011-08-08 15:00:55,027 TRACE [org.jboss.web.tomcat.service.session.JvmRouteValve] (ajp-0.0.0.0-8009-34) handleJvmRoute(): We have detected a failover with different jvmRoute. old one: gsuat02 new one: gsuat01. Will reset the session id.

       

       

      and intermittently:

       

      2011-08-08 15:00:55,850 WARN  [org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.CacheListener] (Incoming-7,10.151.4.210:46709) Possible concurrency problem: Replicated version id 86 is less than or equal to in-memory version for session USER01

       

       

      The logs from httpd show that the wrong server is selected:

       

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy_cluster.c(1839): cluster: Found value Gy1807FDGdCfgW34aGU2ow__.gsuat02 for stickysession JSESSIONID|jsessionid

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy_cluster.c(1839): cluster: Found value Gy1807FDGdCfgW34aGU2ow__.gsuat02 for stickysession JSESSIONID|jsessionid

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy_cluster.c(1441): proxy: byrequests balancer DONE (ajp://10.151.4.210:8009)

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)

      [Tue Aug 09 10:25:20 2011] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://10.151.4.210:8009/

      [Tue Aug 09 10:25:20 2011] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (10.151.4.210)

      [Tue Aug 09 10:25:20 2011] [debug] proxy_util.c(2055): proxy: connecting ajp://10.151.4.210:8009/ to 10.151.4.210:8009

      [Tue Aug 09 10:25:20 2011] [debug] proxy_util.c(2153): proxy: connected / to 10.151.4.210:8009

      [Tue Aug 09 10:25:20 2011] [debug] proxy_util.c(2308): proxy: AJP: fam 2 socket created to connect to 10.151.4.210

       

       

      Notice the session id refers to "gsuat02" but the request is being sent (wrongly) to 10.151.4.210 which corresponds to "gsuat01"

       

      Our httpd configuration is:

       

      LoadModule proxy_module modules/mod_proxy.so

      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

      LoadModule slotmem_module modules/mod_slotmem.so

      LoadModule manager_module modules/mod_manager.so

      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

       

      <VirtualHost *:80>

          ProxyPass / balancer://internalCluster/

       

           <Location /mod_cluster-manager>

              SetHandler mod_cluster-manager

              Order deny,allow

              Deny from all

              Allow from 10.151.4.

          </Location>

      </VirtualHost>

       

      Maxsessionid 250

      Maxnode 75

      Maxhost 75

      UseAlias 1

       

      Listen 10.151.4.160:6666

      <VirtualHost 10.151.4.160:6666>

          ErrorLog logs/mod_cluster-error_log

          CustomLog logs/mod_cluster-access_log combined

       

          <Directory />

              Order deny,allow

              Deny from all

              Allow from 10.151.4.

          </Directory>

       

          ManagerBalancerName internalCluster

       

          KeepAliveTimeout 60

          MaxKeepAliveRequests 0

          LogLevel debug

      </VirtualHost>

       

       

      and in our mod-cluster configuration at the JBoss side we have:

       

      <bean name="ModClusterConfig" class="org.jboss.modcluster.config.ha.HAModClusterConfig" mode="On Demand">

       

          <property name="proxyList">${jboss.mod_cluster.proxyList:}</property>

          <property name="advertise">false</property>

          <property name="advertiseGroupAddress">${jboss.mod_cluster.advertise.address,jboss.partition.udpGroup:224.0.1.105}</property>

          <property name="advertisePort">${jboss.mod_cluster.advertise.port:23364}</property>

          <property name="excludedContexts">${jboss.mod_cluster.excludedContexts:JBOSS,localhost:ROOT,admin-console,invoker,jbossws,jmx-console,juddi,web-console}</property>

          <property name="autoEnableContexts">${jboss.mod_cluster.autoEnableContexts:true}</property>

          <property name="ssl">false</property>

          <property name="stickySession">true</property>

          <property name="stickySessionForce">false</property>

          <property name="stickySessionRemove">false</property>

          <property name="maxAttempts">1</property>

          <property name="workerTimeout">-1</property>

          <property name="nodeTimeout">-1</property>

        </bean>

       

       

      Mod cluster manager DUMP output page shows both JBoss servers:

       

      Node: [6],Name: gsuat02,Balancer: internalCluster,LBGroup: ,Host: 10.151.4.211,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 1,Ttl: 60,Elected: 253,Read: 4828840,Transfered: 547741,Connected: 0,Load: 82

       

      Node: [13],Name: gsuat01,Balancer: internalCluster,LBGroup: ,Host: 10.151.4.210,Port: 8009,Type: ajp,Flushpackets: Off,Flushwait: 10,Ping: 10,Smax: 1,Ttl: 60,Elected: 264,Read: 7219247,Transfered: 800564,Connected: 0,Load: 78

       

       

      Any help will be appreciated.  Are we doing anything wrong? Anybody running into the same issue out there?

       

      Thanks in advance,

      Alex Soto

        • 1. Re: Sticky sessions not working
          Jean-Frederic Clere Master

          All looks OK.

          Could you look to the CONFIG messages in the debug of error_log?

          • 2. Re: Sticky sessions not working
            alex soto Newbie

            Thanks for the reply.  Here is what I grepped from the error log:

             

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:32:06 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:36:07 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:45:08 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:47:36 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:48:07 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 12:53:54 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:07:15 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:11:11 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:14:06 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:14:06 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:14:06 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:14:39 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:18:13 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

             

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat02&Host=10.151.4.211&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:35:09 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(1653): manager_trans CONFIG (/)

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(2296): manager_handler CONFIG (/) processing: "JVMRoute=gsuat01&Host=10.151.4.210&Maxattempts=1&Port=8009&StickySessionForce=No&Type=ajp"

            [Tue Aug 09 13:36:19 2011] [debug] mod_manager.c(2339): manager_handler CONFIG  OK

             

             

            Let me know if you need to see anything else, we have spent a lot of time trying to figure out what's wrong.

            • 3. Re: Sticky sessions not working
              alex soto Newbie

              Jean-Frederic:

               

              We managed to recompile the mod-cluster with HAVE_CLUSTER_EX_DEBUG setting.    Colored in red is where I think it's getting confused.  First, it finds the route "gsuat01," later it says "No route found," then it sends the request to the wrong server.  I hope it helps.

               

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1982): proxy_cluster_trans for 0 (null) (null) uri: / args: (null) unparsed_uri: /

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1839): cluster: Found value npK2tiLnn64BPKQ2xgwLpA__.gsuat01 for stickysession JSESSIONID|jsessionid

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1847): cluster: Found route gsuat01

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1794): find_nodedomain: finding node for gsuat01: internalcluster

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1807): find_nodedomain: finding domain for gsuat01: internalcluster

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1839): cluster: Found value npK2tiLnn64BPKQ2xgwLpA__.gsuat01 for stickysession JSESSIONID|jsessionid

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1847): cluster: Found route gsuat01

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1794): find_nodedomain: finding node for gsuat01: internalcluster

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1807): find_nodedomain: finding domain for gsuat01: internalcluster

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2040): proxy_cluster_trans using internalCluster uri: proxy:balancer://internalCluster/

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2073): proxy_cluster_canon url: //internalCluster/

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2424): proxy_cluster_pre_request: url balancer://internalcluster/

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2212): cluster:No route found

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1359): proxy: Entering byrequests for CLUSTER (balancer://internalcluster)

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(1441): proxy: byrequests balancer DONE (ajp://10.151.4.211:8009)

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_cluster.c(2587): proxy_cluster_pre_request: balancer (balancer://internalcluster) worker (ajp://10.151.4.211:8009) rewritten to ajp://10.151.4.211:8009/

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)

              [Wed Aug 10 11:30:57 2011] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://10.151.4.211:8009/

              [Wed Aug 10 11:30:57 2011] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (10.151.4.211)

              [Wed Aug 10 11:30:57 2011] [debug] proxy_util.c(2055): proxy: connecting ajp://10.151.4.211:8009/ to 10.151.4.211:8009

              [Wed Aug 10 11:30:57 2011] [debug] proxy_util.c(2153): proxy: connected / to 10.151.4.211:8009

              [Wed Aug 10 11:30:57 2011] [debug] proxy_util.c(2308): proxy: AJP: fam 2 socket created to connect to 10.151.4.211

              [Wed Aug 10 11:30:57 2011] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong

              [Wed Aug 10 11:30:57 2011] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done

               

               

               

               

              Thanks for your help so far,

              Alex Soto

              • 4. Re: Sticky sessions not working
                alex soto Newbie

                Based on the previous log output and going over the code, it looks like the route value is being overwritten. 

                In line 1859 of "mod_proxy_cluster.c" this value is set in "r->notes"

                                    apr_table_setn(r->notes, "session-route", route);

                Later on, line 2205 tries to retrieve the value:

                 

                    /* We already should have the route in the notes for the trans() */

                    *route = apr_table_get(r->notes, "session-route");

                 

                but it's no there anymore. Does this seem plausible? 

                • 5. Re: Sticky sessions not working
                  Jean-Frederic Clere Master

                  In fact it doesn't find the balancer "internalcluster" .. . Because it probably has "internalCluster" (Note the c/C).

                  It doesn't set the route because find_nodedomain returns APR_NOTFOUND

                  See the message "find_nodedomain: finding domain for gsuat01: internalcluster".

                   

                  Try to use internalcluster instead internalcluste

                  • 6. Re: Sticky sessions not working
                    alex soto Newbie

                    I am very happy to inform that it is working now.  However, it seems odd that it would not work when the balancer name used camel casing. 

                    All I did was to change "internalCluster" to "internalcluster" and voilà!, it worked.

                     

                    The documentation does not mention any kind of restriction on this value:

                     

                    http://docs.jboss.org/mod_cluster/1.1.0/html/native.config.html#d0e596

                     

                    I have also verified that we used the exact same casing in all places we had this name.  Almost looks like a bug to me.

                     

                    In any case, thanks a lot for the help.