11 Replies Latest reply on Jul 24, 2013 2:32 PM by tzman

    Session replication not occurring across different hosts

    tzman

      I am running into an issue with domain mode configuration. I have Apache 2.2 with mod_cluster, 1 domain controller and 2 host controllers running on a single pc for testing. That is 3 separate instances of JBoss Eap 6.1

      I can see both host controllers join the domain. And I can see both nodes recognize mod_cluster with the log output that they are using jvm-route. The number of nodes in the cluster logged appears correct.

      I have tried a number of tutorials and have come across an issue getting session replication to occur across different hosts. Using the different clustering demo apps, i can session replication will work if failover occurs to a server within the same server group on the same host, but if failover occurs to a server in a second host the session is invalid.

      I am not getting any errors in the server logs or the http log.

      The following is my configuration, compiled from different tutorials.

      Any clue as to what would cause this behavior? Is there additional information I can provide?

      Any insight is greatly appreciated.

       

      Httpd.conf

      LoadModule proxy_module modules/mod_proxy.so

       

      LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

      LoadModule proxy_http_module modules/mod_proxy_http.so

       

      LoadModule proxy_cluster_module modules/mod_proxy_cluster.so

       

      LoadModule manager_module modules/mod_manager.so

      LoadModule slotmem_module modules/mod_slotmem.so

      LoadModule advertise_module modules/mod_advertise.so

       

      # MOD_CLUSTER_ADDS

      # Adjust to you hostname and subnet.

      <IfModule manager_module>

        Listen 172.16.4.34:6666

        ManagerBalancerName main-server-group

        <VirtualHost 172.16.4.34:6666>

          <Location />

           Order deny,allow

           Deny from all

           Allow from all

          </Location>

       

          ErrorLog "logs/chatter_log"

          LogLevel debug

          KeepAliveTimeout 300

          MaxKeepAliveRequests 0

          #ServerAdvertise on http://127.0.0.1:6666

          AdvertiseFrequency 30

          #AdvertiseSecurityKey secret

          #AdvertiseGroup 224.0.1.105:23364

          EnableMCPMReceive

       

      #    <Location /mod_cluster_manager>

      #       SetHandler mod_cluster-manager

      #       Order deny,allow

      #       Deny from all

      #       Allow from all

      #    </Location>

       

        </VirtualHost>

      </IfModule>

      NameVirtualHost 172.16.4.34:80

       

      <VirtualHost 172.16.4.34:80>

          ServerName 172.16.4.34

          ErrorLog "logs/cluster_log"

          LogLevel debug

      #    ServerAlias 172.16.4.34

       

          ProxyPassReverse / balancer://main-server-group

          ProxyPreserveHost On

       

          <Location />

              Order deny,allow

              Allow from All

          </Location>

       

          <Location /mod_cluster_manager>

              SetHandler mod_cluster-manager

              Order deny,allow

              Deny from all

              Allow from all

          </Location>

      </VirtualHost>

       

      Domain.xml (using full-ha profile and full-ha-sockets)

               <subsystem xmlns="urn:jboss:domain:web:1.4" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

                     <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>

                     <connector name="ajp" protocol="AJP/1.3" scheme="http" socket-binding="ajp"/>

                     <virtual-server name="default-host" enable-welcome-root="true">

                         <alias name="localhost"/>

                         <alias name="example.com"/>

                     </virtual-server>

                 </subsystem>

             <subsystem xmlns="urn:jboss:domain:modcluster:1.1">

                     <mod-cluster-config advertise-socket="modcluster" proxy-list="172.16.4.34:6666" connector="ajp">

                         <dynamic-load-provider>

                             <load-metric type="busyness"/>

                         </dynamic-load-provider>

                     </mod-cluster-config>

                 </subsystem>

        <server-groups>

            <server-group name="main-server-group" profile="full-ha">

                <jvm name="default">

                    <heap size="1303m" max-size="1303m"/>

                    <permgen max-size="256m"/>

                </jvm>

                <socket-binding-group ref="full-ha-sockets"/>

                <deployments>

                    <deployment name="cluster-example.war" runtime-name="cluster-example.war"/>

                </deployments>

            </server-group>

            <server-group name="other-server-group" profile="full-ha">

                <jvm name="default">

                    <heap size="1303m" max-size="1303m"/>

                    <permgen max-size="256m"/>

                </jvm>

                <socket-binding-group ref="full-ha-sockets"/>

            </server-group>

        </server-groups>


      Domain host.xml

      <domain-controller>

            <local/>

         </domain-controller>

       

         <interfaces>

             <interface name="management">

                 <inet-address value="${jboss.bind.address.management:172.16.4.34}"/>

             </interface>

         </interfaces>

       

       

      Slave 1 host.xml

      <domain-controller>

       

            <remote host="${jboss.domain.master.address:172.16.4.34}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>

         </domain-controller>

       

         <interfaces>

             <interface name="management">

                 <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

             </interface>

             <interface name="public">

                <inet-address value="${jboss.bind.address:172.16.4.34}"/>

             </interface>

             <interface name="unsecure">

                 <!-- Used for IIOP sockets in the standard configuration.

                      To secure JacORB you need to setup SSL -->

                 <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>

             </interface>

         </interfaces>

         <servers>

             <server name="server-one" group="main-server-group">

                 <jvm name="default">

                   <jvm-options>

                     <option value="-agentlib:jdwp=transport=dt_socket,address=8787,server=y,suspend=n"/>

                   </jvm-options>

                </jvm>

                 <socket-bindings port-offset="100"/>

             </server>

             <server name="server-two" group="main-server-group"  auto-start="false">

                 <socket-bindings port-offset="200"/>

             </server>

         </servers>

       

       

      Slave 2 host.xml

      <domain-controller>

       

            <remote host="${jboss.domain.master.address:172.16.4.34}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>

         </domain-controller>

       

         <interfaces>

             <interface name="management">

                 <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>

             </interface>

             <interface name="public">

                <inet-address value="${jboss.bind.address:172.16.4.34}"/>

             </interface>

             <interface name="unsecure">

                 <!-- Used for IIOP sockets in the standard configuration.

                      To secure JacORB you need to setup SSL -->

                 <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>

             </interface>

         </interfaces>

         <servers>

             <server name="server-three" group="main-server-group">

                 <socket-bindings port-offset="300"/>

             </server>

             <server name="server-four" group="main-server-group"  auto-start="false">

                 <socket-bindings port-offset="400"/>

             </server>

         </servers>

        • 1. Re: Session replication not occurring across different hosts
          rhusar

          I don't see anything obvious yet.

           

          I would guess there is some misconfiguration. Why are there 2 server groups with the same configuration? Also make sure that really all the servers form a cluster for a cache named "web" to cluster your web sessions.

          • 2. Re: Session replication not occurring across different hosts
            tzman

            Radoslav Husar wrote

            I would guess there is some misconfiguration. Why are there 2 server groups with the same configuration?

             

            Do you mean profile and sockets? I planned to not replicate across the entire but only within a server group. What are the implications of having the same configuration?

             

             

            Also make sure that really all the servers form a cluster for a cache named "web" to cluster your web sessions.

            Where is this configured?

            • 3. Re: Session replication not occurring across different hosts
              tzman

              I removed the second server group. And ensured there was an infinispan cluster configuration named web. This is in the domain.xml file so all hosts should share this.

               

              I am still seeing the same problem. Any suggestions would be greatly appreciated.

              • 4. Re: Session replication not occurring across different hosts
                rhusar

                Do you mean profile and sockets? I planned to not replicate across the entire but only within a server group. What are the implications of having the same configuration?

                Yes, that is fine. For that you obviously cannot have exactly the "same" configuration. You would want to:

                • set different multicast addresses or ports so that you create 2 separate groups and replicate in between them
                • set 2 different load-balancing-groups in mod_cluster so that failover always happens within the group

                 

                Also make sure that really all the servers form a cluster for a cache named "web" to cluster your web sessions.

                Where is this configured?

                I meant for you to look in the log if you see the cluster size is what you expected.

                 

                I am still seeing the same problem. Any suggestions would be greatly appreciated.

                So how are you testing it works?

                • 5. Re: Session replication not occurring across different hosts
                  tzman

                  Thanks for the help!

                   

                   

                  Radoslav Husar wrote:

                   

                  Do you mean profile and sockets? I planned to not replicate across the entire but only within a server group. What are the implications of having the same configuration?

                  Yes, that is fine. For that you obviously cannot have exactly the "same" configuration. You would want to:

                  • set different multicast addresses or ports so that you create 2 separate groups and replicate in between them
                  • set 2 different load-balancing-groups in mod_cluster so that failover always happens within the group

                  Ok, will give that a try once i have one group working.

                   

                   

                  Also make sure that really all the servers form a cluster for a cache named "web" to cluster your web sessions.

                  Where is this configured?

                  I meant for you to look in the log if you see the cluster size is what you expected.

                   

                  I am still seeing the same problem. Any suggestions would be greatly appreciated.

                  So how are you testing it works?

                  Yes, the log shows the correct cluster size and I can see nodes communication with other nodes when i start or stop one of the hosts.

                  I went through several different tutorials to get as far as I have, 2 of them have demo apps to test.

                  http://blog.akquinet.de/2012/06/21/clustering-in-jboss-as7eap-6/

                  https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto

                   

                  The first link has a test application that keeps a count. I have been using this to test with. The count will go up each time the page is refreshed. Stopping the current node in the same host, will show the count to continue to increment. If I stop the entire host and refresh the page, the count goes to zero and starts incrementing from there. I can start and stop nodes and have the counts increment independently from there. I get the same sort of behavior from the other test app on the second link.

                   

                  I have also tried to test our application with the same configuration. This is a Seam 2.2 application with JSF 1.2. The problem i run into there is that when i stop the current host, and then hit the page again, I am getting a marshalling problem with a Seam interceptor when it tries to read the existing session from the original host. I am not sure if this is related, but it is odd that the session replication there seems to be trying to work.

                  • 6. Re: Session replication not occurring across different hosts
                    tzman

                    Here is the end of the log from the start up of the second host.

                     

                    08:00:24,939 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000078: Starting JGroups Channel

                    08:00:24,948 INFO  [stdout] (ServerService Thread Pool -- 63)

                    08:00:24,948 INFO  [stdout] (ServerService Thread Pool -- 63) -------------------------------------------------------------------

                    08:00:24,948 INFO  [stdout] (ServerService Thread Pool -- 63) GMS: address=slave2:server-three/ejb, cluster=ejb, physical address=172.16.4.34:55500

                    08:00:24,948 INFO  [stdout] (ServerService Thread Pool -- 63) -------------------------------------------------------------------

                     

                     

                    08:00:24,948 INFO  [stdout] (ServerService Thread Pool -- 64)

                    08:00:24,948 INFO  [stdout] (ServerService Thread Pool -- 64) -------------------------------------------------------------------

                    08:00:24,948 INFO  [stdout] (ServerService Thread Pool -- 64) GMS: address=slave2:server-three/web, cluster=web, physical address=172.16.4.34:55500

                    08:00:24,948 INFO  [stdout] (ServerService Thread Pool -- 64) -------------------------------------------------------------------

                     

                     

                    08:00:25,023 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 63) ISPN000094: Received new cluster view: [tzman:server-one/ejb|1] [tzman:server-one/ejb, slave2:server-three/ejb]

                    08:00:25,023 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000094: Received new cluster view: [tzman:server-one/web|1] [tzman:server-one/web, slave2:server-three/web]

                    08:00:25,045 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 63) ISPN000079: Cache local address is slave2:server-three/ejb, physical addresses are [172.16.4.34:55500]

                    08:00:25,048 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 64) ISPN000079: Cache local address is slave2:server-three/web, physical addresses are [172.16.4.34:55500]

                    08:00:25,048 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 63) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.6.Final

                    08:00:25,073 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 63) ISPN000161: Using a batchMode transaction manager

                    08:00:25,073 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 62) ISPN000161: Using a batchMode transaction manager

                    08:00:25,073 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 64) ISPN000161: Using a batchMode transaction manager

                    08:00:25,073 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 65) ISPN000161: Using a batchMode transaction manager

                    08:00:25,152 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 63) ISPN000031: MBeans were successfully registered to the platform MBean server.

                    08:00:25,154 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 65) ISPN000031: MBeans were successfully registered to the platform MBean server.

                    08:00:25,154 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 64) ISPN000031: MBeans were successfully registered to the platform MBean server.

                    08:00:25,155 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 62) ISPN000031: MBeans were successfully registered to the platform MBean server.

                    08:00:25,206 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 62) JBAS010281: Started repl cache from ejb container

                    08:00:25,206 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) JBAS010281: Started default-host/cluster cache from web container

                    08:00:25,215 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 63) JBAS010281: Started remote-connector-client-mappings cache from ejb container

                    08:00:25,215 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) JBAS010281: Started repl cache from web container

                    08:00:25,215 INFO  [org.jboss.as.clustering] (MSC service thread 1-13) JBAS010238: Number of cluster members: 2

                    08:00:25,216 INFO  [org.jboss.as.clustering] (MSC service thread 1-8) JBAS010238: Number of cluster members: 2

                    08:00:25,241 INFO  [org.infinispan.factories.TransactionManagerFactory] (MSC service thread 1-4) ISPN000161: Using a batchMode transaction manager

                    08:00:25,246 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-4) ISPN000031: MBeans were successfully registered to the platform MBean server.

                    08:00:25,253 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010281: Started de.akquinet.jbosscc.cluster.ClusteredStatefulBean cache from ejb container

                    08:00:25,360 INFO  [org.jboss.web] (ServerService Thread Pool -- 64) JBAS018210: Register web context: /cluster

                    08:00:25,416 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 64) Initializing Mojarra 2.1.19-jbossorg-1 20130304-0420 for context '/cluster'

                    08:00:25,533 INFO  [org.hibernate.validator.internal.util.Version] (ServerService Thread Pool -- 64) HV000001: Hibernate Validator 4.3.1.Final-redhat-1

                    08:00:25,650 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "cluster-example.war" (runtime-name : "cluster-example.war")

                    08:00:25,653 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.1.0.GA (AS 7.2.0.Final-redhat-8) started in 3444ms - Started 233 of 358 services (124 services are passive or on-demand)

                    • 7. Re: Session replication not occurring across different hosts
                      rhusar

                      I also find using a counter very convenient (here is a small test app that does that and few other things: https://github.com/clusterbench/clusterbench)

                       

                      You seem to be running at different hosts, you need to make sure you are sending the same session cookies to both servers (cookies identify the session right...). The easiest way is to use a loadbalancer, that will do that for you (the browser will always send the same cookies to the LB).

                       

                      Here is a quick tip using curl if you don't have an LB up yet (query the URL, generate new session, spoof cookies for host1 to host2, shutdown server on host1, query host2 with spoofied cookies, voila!)

                       

                      $ curl http://host1:8080/clusterbench/session -b cookies1.txt -c cookies1.txt
                      0
                      $ cat cookies1.txt
                      # Netscape HTTP Cookie File
                      # http://curl.haxx.se/docs/http-cookies.html
                      # This file was generated by libcurl! Edit at your own risk.
                      
                      host1          FALSE          /clusterbench          FALSE          0          JSESSIONID          TjZxiaP22Qkqw5zBo87m+8XA
                      $ sed "s/host1/host2/" cookies1.txt > cookies2.txt
                      $ curl http://host2:8080/clusterbench/session -b cookies2.txt -c cookies2.txt
                      1
                      

                       

                      Right?

                      • 8. Re: Session replication not occurring across different hosts
                        tzman

                        I believe I have that downloaded.. I can try it as well.

                         

                        I have Apache with mod_cluster set up as a load balancer, so im not accessing the hosts directly. However, all of this is running on a single pc. I believe apache and mod cluster should be handling the session id correctly, but i will check and make sure.

                        • 9. Re: Session replication not occurring across different hosts
                          tzman

                          I was able to test using the clusterbench app. The session test clusterbench/session works as expected for both ee5 and ee6. However, the ejb version gives me the same results as the other demo apps. It just gives me a count of 0 when I shutdown the current node and refresh the page. This makes sense more with the problem I am having with our application. In that case there is not an ehb involved, it is just session data.

                           

                          What would cause an ejb to not replicate when the http session replicates correctly?

                           

                          Thanks again for the help

                          • 10. Re: Session replication not occurring across different hosts
                            tzman

                            I have gone over the tutorial again and cannot see what I may have missed here. I am attaching my configuration files as well as log files from the test.

                             

                            Running through Apache http://localhost/clusterbench/ejbservlet

                             

                            In this test, Host-1 Sever-One was hit for the first request. I refreshed to a count of 3 and killed the host (Ctrl-C). After refreshing the page the count went to 0.

                             

                            Anyone have any ideas here??

                            • 11. Re: Session replication not occurring across different hosts
                              tzman

                              Radoslav, can you provide any more info on this?

                               

                              Any help is greatly appreciated.