Load Balancing Cluster not working with Apache HTTP Server 2.4.6 and JBoss EAP 7

[adityan]

I am certifying my application on JBoss EAP 7. My application works on standalone mode but in cluster mode, my application gets deployed but I am unable to login. I am again re-routed to login url.
I have setup cluster using mod_cluster.

There is no error log in Load Balancer Server logs neither on individual nodes of the cluster.
In my test environment, my load balancer and 2 server nodes are on same machine. I have given JBoss nodes different ports and instance-ids to differentiate the nodes.

Are there any session replication and/or sticky session issues in JBoss EAP 7.x?

I am sharing my mod_cluster.conf and JBoss standalone-ha.xml extracts from one of the nodes in cluster. Thanks in advance. Please ignore commas in XML snippets, having style issues.

Following are the snippets in standalone-ha.xml file:

Modcluster settings are:

<subsystem xmlns="urn:jboss:domain:modcluster:2.0">
     <mod-cluster-config advertise-socket="modcluster" proxies="proxy1" balancer="testcluster" advertise="true" connector="ajp">
          <dynamic-load-provider>
               <load-metric type="busyness"/>
          </dynamic-load-provider>
     </mod-cluster-config>
</subsystem>

Undertow settings are:

<subsystem xmlns="urn:jboss:domain:undertow:3.1" instance-id="node1">
   <buffer-cache name="default"/>
   <server name="default-server">
   <ajp-listener name="ajp" socket-binding="ajp"/>
   <http-listener name="default" socket-binding="http" redirect-socket="https"/>
   <host name="default-host" alias="localhost">
   <location name="/" handler="welcome-content"/>
   <filter-ref name="server-header"/>
   <filter-ref name="x-powered-by-header"/>
   </host>
   </server>
   <servlet-container name="default">
   <jsp-config/>
   <websockets/>
   </servlet-container>
   <handlers>
   <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
   </handlers>
   <filters>
   <response-header name="server-header" header-name="Server" header-value="JBoss-EAP/7"/>
   <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
   </filters>
</subsystem>

Contents of Socket Binding Group are:

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
   <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
   <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
   <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
   <socket-binding name="http" port="${jboss.http.port:8080}"/>
   <socket-binding name="https" port="${jboss.https.port:8443}"/>
   <socket-binding name="jgroups-mping" interface="private" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
   <socket-binding name="jgroups-tcp" interface="private" port="7600"/>
   <socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
   <socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
   <socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
   <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
   <socket-binding name="txn-recovery-environment" port="4712"/>
   <socket-binding name="txn-status-manager" port="4713"/>
   <outbound-socket-binding name="mail-smtp">
   <remote-destination host="localhost" port="25"/>
   </outbound-socket-binding>
   <outbound-socket-binding name="proxy1">
   <!-- host and port number of the load-balancer. -->
   <remote-destination host="x.x.x.x" port="81"/>
   </outbound-socket-binding>
</socket-binding-group>

Contents of mod_cluster.conf are as follows:

LoadModule proxy_cluster_module modules/mod_proxy_cluster.so
LoadModule cluster_slotmem_module modules/mod_cluster_slotmem.so
LoadModule manager_module modules/mod_manager.so
LoadModule advertise_module modules/mod_advertise.so 
MemManagerFile "//httpd2.4.6Home/var/cache/mod_cluster" 
<IfModule manager_module>
     Listen 81
     <VirtualHost *:81>
     <Directory />
          Require all granted
     </Directory>
     <Location />
          Require all granted
     </Location>
     <Location /mod_cluster_manager>
          SetHandler mod_cluster-manager
          Require all granted
     </Location>
     KeepAliveTimeout 60
     MaxKeepAliveRequests 0
     ManagerBalancerName testcluster
     AdvertiseFrequency 5
     ServerAdvertise on
     EnableMCPMReceive
     </VirtualHost>
</IfModule>

[pferraro]

There shouldn't be any issues with session replication or session stickiness in EAP 7.x.  When you say "standalone mode" vs "clustered mode", what do you mean exactly?  These are orthogonal terms.  Standalone refers to how the server is managed (vs domain mode), whereas clustered mode refers to the profile the server uses, e.g. default or full vs ha or full-ha.

Also, can you paste the configuration of your "private" interface?  This defaults to the loopback interface, thus typically needs to explicitly configured in order to cluster nodes running on different hosts.

[adityan]

Thanks for your response.

My setup is as follows:

I have 2 JBoss instances, both are running using standalone-ha.xml profile.

I am using external load balancer, Apache HTTPD 2.4.6.

The interface configuration in standalone-ha.xml of both servers where x.x.x.x is IP of respective hosts, is as follows:

<interfaces>

        <interface name="management">

            <inet-address value="${jboss.bind.address.management:x.x.x.x}"/>

        </interface>

        <interface name="public">

            <inet-address value="${jboss.bind.address:x.x.x.x}"/>

        </interface>

        <interface name="private">

            <inet-address value="${jboss.bind.address.private:x.x.x.x}"/>

        </interface>

</interfaces>

The default interface is standard socket binding group is "public". Can you guide me where I can make further changes to make this cluster work?

My start up command is as follows for starting the JBoss instances as follows:

standalone.bat -c standalone-ha.xml -Djboss.node.name=node1 -b x.x.x.x

[pferraro]

A few observations:

• The jboss.node.name will default to the host name, if undefined.  Consequently, if your server instance a running on the same host, they must each define a distinct "jboss.node.name" system property.
• Likewise, if your server instance a running on the same host (i.e. they share the same bind address) then you must differentiate them via a port-offset. This can be defined via configuration, or via the "jboss.socket.binding.port-offset" system property.
• The -b switch only sets the "jboss.bind.address" system property. The bind address used by JGroups for cluster formation must also be set, either via configuration, or the "jboss.bind.address.private" system property.
• If there are issues with cluster formation, or with port conflicts, these would be visible in your standalone/log/server.log

[adityan]

Thanks for your response.

"jboss.node.name", "jboss.socket.binding.port-offset", "jboss.bind.address.private" are properly set. I have no error logs related to cluster formation or port conflicts in standalone/log/server.log.

As of now JGroup use private interface as mentioned in standard socket binding.

It is just that session is not getting maintained. I had no issues in deploying same application on JBoss 6.x EAP servers and Apache HTTPD 2.2.x/2.4.6 server as load balancer.

Where can I look to find possible cause for this issue?

I made one change: I set sticky-session as "true" in modcluster subsystem in standalone-ha.xml. But still issue persists.

[adityan]

Is this issue related to following issue:

[JBEAP-6078] [GSS](7.1.0) Using Infinispan in Distribution Mode for JBossWeb breaks session stickiness - JBoss Issue Tra…

I have not used <distributable/> tag in my application.

[pferraro]

Is this issue related to following issue:

[JBEAP-6078] [GSS](7.1.0) Using Infinispan in Distribution Mode for JBossWeb breaks session stickiness - JBoss Issue Tra…

No - this has nothing to do with that.

I have not used <distributable/> tag in my application.

There's your problem.  Only <distributable/> web applications will replicate sessions.  Without <distributable/> web sessions will only exist on the node that created them.

[adityan]

Is this tag only specific to deploying application on JBoss EAP 7? My application without this tag is deployed on JBoss 6.x EAP cluster environment.

Secondly, After adding this tag I am getting 500 Internal server error instead of redirection.

[pferraro]

Is this tag only specific to deploying application on JBoss EAP 7? My application without this tag is deployed on JBoss 6.x EAP cluster environment.

This is part of the JEE specification, specifically, §7.7.2 & §10.1 of the servlet specification.

Secondly, After adding this tag I am getting 500 Internal server error instead of redirection.

There should be an ERROR message in your server.log indicating the cause of this 500 error.

[adityan]

I checked in the server log file. I did not get any error loag but a warning as follows:

This server.log of node 1.

WARN  [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t2) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node2

Is this the issue affecting the cluster?  What is its resolution?

I also went through cluster deployment guide for JBoss 6.4. It is also mentioned to add <distributable/> in the web.xml for deploying on cluster mode.  But on that platform my application gets deployed and works even without adding this tag.

So for JBoss EAP 7.0 onwards, is it mandatory to have that in deployment descriptor?

One more thing, each JBoss EAP 7.0 instance has its own WAR file deployed. This setup worked in JBoss 6.x EAP.

[pferraro]

Aditya Neurgaonkar wrote:

I checked in the server log file. I did not get any error loag but a warning as follows:

This server.log of node 1.

WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (transport-thread--p16-t2) ISPN000197: Error updating cluster member list: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node2

Is this the issue affecting the cluster? What is its resolution?

This was fixed in EAP 7.0.2.  See:

[JBEAP-5227] [GSS](7.0.z) RPCs to non-existant FORK channel are dropped - JBoss Issue Tracker

I also went through cluster deployment guide for JBoss 6.4. It is also mentioned to add <distributable/> in the web.xml for deploying on cluster mode. But on that platform my application gets deployed and works even without adding this tag.

So for JBoss EAP 7.0 onwards, is it mandatory to have that in deployment descriptor?

You only need to use <distributable/> if you require your web sessions to be distributable (i.e. available from a node that did not create it).  If you omit this, your application will still deploy - but your web sessions will not be accessible from other nodes.  Again, this is part of the the JEE standard.

One more thing, each JBoss EAP 7.0 instance has its own WAR file deployed. This setup worked in JBoss 6.x EAP.

I don't understand.  What is your question?

[adityan]

Thanks for your response.

I read about distributable sessions, I am not maintaining it in my application so I removed it from my application.

In my application I have implemented spring security for user authentication. This works properly in standalone mode but fails in cluster mode. Is there any issue with sticky session and spring security for JBoss EAP 7?

Are there any changes in modcluster module for JBoss EAP 7? Is it different from JBoss EAP 6.x?

[pferraro]

I don't know what you mean by "standalone" mode vs "cluster" mode.  "Standalone" indicates how a server is managed (as opposed to "domain" mode).  Clustered configurations are provided for both standalone servers and domain-managed servers.

Sticky session configuration has not changed in mod_cluster - however, it is imperative that your nodes identify themselves to the load balancer using unique names.  This identifier is configured via the "instance-id" attribute in the undertow subsystem, which defaults to the "jboss.node.name" property if undefined.  This attribute has not changed between EAP6 and EAP7.

[adityan]

Ok. Are there any known issues with respect to spring security, jasig CAS and sticky session maintenance on JBoss EAP 7 when deployed in clustered configurations for standalone servers? "Instance-ids" are properly maintained for JBoss EAP 7.

This combination works ok JBoss EAP 6 in cluster configuration for standalone servers. There are no exceptions thrown nor logged in server logs.

Finally, I want to apologize for the confusion caused by "standalone mode vs cluster mode". What I really meant to say is "clustered configuration for standalone servers of JBoss EAP 7 using Apache HTTP Server."

Please let me know if there are any issues for either spring security or jasig CAS which are breaking sticky session.

Reference URL for jasig CAS

Home - Central Authentication Service - Apereo Wiki

GitHub - apereo/cas: Apereo CAS - Enterprise Single Sign On for all earthlings and beyond.

https://www.apereo.org/

[pferraro]

I suspect the underlying issue is the interaction between Spring Security and EAP's session manager.  Typically, following login, the session ID is changed to prevent session fixation attacks - and it seems Spring Security does indeed do this.  I assume that Spring Security relies on the HttpServletRequest.changeSessionId() method?.  Can you register a javax.servlet.http.HttpSessionIdListener to verify that this is indeed the case?