Session replication is not working as expected in Red Hat JBoss Enterprise Application Platform - Version 6.3.0.GA

[joysagar]

Hi All,

I'm bit new to Jboss, could you please help me out in finding the root cause in the below mentioned issue.Thanks and much appreciated,

Issue : Session replication is not happening between master(domain controller) and slave(host). Following are the logs,

From master:

[Server:server-three] 07:53:38,246 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000078: Starting JGroups Channel

[Server:server-three] 07:53:38,260 INFO  [stdout] (ServerService Thread Pool -- 65)

[Server:server-three] 07:53:38,260 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

[Server:server-three] 07:53:38,261 INFO  [stdout] (ServerService Thread Pool -- 65) GMS: address=master:server-three/web, cluster=web, physical address=10.78.216.145:7850

[Server:server-three] 07:53:38,261 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

[Host Controller] 07:53:40,511 INFO  [org.jboss.as.domain] (Host Controller Service Threads - 25) JBAS010918: Registered remote slave host "slave", JBoss EAP 6.3.0.GA (AS 7.4.0.Final-redhat-19)

[Server:server-three] 07:53:41,273 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000094: Received new cluster view: [master:server-three/web|0] [master:server-three/web]

[Server:server-three] 07:53:41,275 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000079: Cache local address is master:server-three/web, physical addresses are [10.78.216.145:7850]

[Server:server-three] 07:53:41,279 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 65) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.10.Final

[Server:server-three] 07:53:41,288 INFO  [org.jboss.as.clustering] (MSC service thread 1-6) JBAS010238: Number of cluster members: 1

From slave:

[Server:server-three-slave] 07:53:36,274 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 67) ISPN000078: Starting JGroups Channel

[Server:server-three-slave] 07:53:36,287 INFO  [stdout] (ServerService Thread Pool -- 67)

[Server:server-three-slave] 07:53:36,287 INFO  [stdout] (ServerService Thread Pool -- 67) -------------------------------------------------------------------

[Server:server-three-slave] 07:53:36,288 INFO  [stdout] (ServerService Thread Pool -- 67) GMS: address=slave:server-three-slave/web, cluster=web, physical address=10.78.216.36:7850

[Server:server-three-slave] 07:53:36,288 INFO  [stdout] (ServerService Thread Pool -- 67) -------------------------------------------------------------------

[Server:server-three-slave] 07:53:39,301 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 67) ISPN000094: Received new cluster view: [slave:server-three-slave/web|0] [slave:server-three-slave/web]

[Server:server-three-slave] 07:53:39,302 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 67) ISPN000079: Cache local address is slave:server-three-slave/web, physical addresses are [10.78.216.36:7850]

[Server:server-three-slave] 07:53:39,306 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 67) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.10.Final

[Server:server-three-slave] 07:53:39,315 INFO  [org.jboss.as.clustering] (MSC service thread 1-5) JBAS010238: Number of cluster members: 1

I'm advertising my cluster and i can see the both master and slave nodes are present in the UI,

https://10.78.X.X:8445/mod_cluster_manager,

I have disabled firewall and SELinux to avoid network problems.I believe, i have made all the required configurations in domain.xml and ssl.conf of httpd. I have also added  <distributable/> tag and jboss-web.xml changes in my war. It'd be of more help if anyone of you direct me in a different direction as i'm totally lost,

Adding configuration files for easy access,

Thanks,

Joy.

[joysagar]

I believe i'm half way through. I have changed my domain.xml by adding sticky-session="true" at <mod-cluster-config>. First call is able to through successfully but the subsequent calls are not happening as expected, any ideas.

[rhusar]

To begin with, the cluster did not form, the number of members in the cluster is 1, meaning you have two clusters of size 1:

[Server:server-three] 07:53:41,288 INFO  [org.jboss.as.clustering] (MSC service thread 1-6) JBAS010238: Number of cluster members: 1

By default, the configuration uses UDP multicast to discover the cluster nodes (MPING protocol). Make sure that multicast is getting through your network.

[joysagar]

Thanks for the response Radoslav. I'm bit new to the networking stuff, pardon me if i don't make sense. When i try ifconfig eth0 in both the machines, i got the following as output. As per some sites, i believe multicasting is enabled,

eth0      Link encap:Ethernet  HWaddr 00:0C:29:FD:52:99

          inet addr:10.78.216.36  Bcast:10.78.217.255  Mask:255.255.254.0

          inet6 addr: fe80::20c:29ff:fefd:5299/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:2595595 errors:0 dropped:0 overruns:0 frame:0

          TX packets:382322 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:571190838 (544.7 MiB)  TX bytes:51811030 (49.4 MiB)

Sorry for being a pain but please help me out in finding the cause. Thanks and appreciate,

Regards,

Joy.

[joysagar]

Radoslav,

Thanks for helping me out, In my domain.xml i found that my default-stack at jboss-groups is pointed to TCP instead of UDP. After changing this, following are logs from both the machines,

From Master:

[Server:server-three] 09:00:29,961 INFO  [stdout] (ServerService Thread Pool -- 66)

[Server:server-three] 09:00:29,961 INFO  [stdout] (ServerService Thread Pool -- 66) -------------------------------------------------------------------

[Server:server-three] 09:00:29,962 INFO  [stdout] (ServerService Thread Pool -- 66) GMS: address=master:server-three/web, cluster=web, physical address=10.78.216.145:55450

[Server:server-three] 09:00:29,962 INFO  [stdout] (ServerService Thread Pool -- 66) -------------------------------------------------------------------

[Server:server-three] 09:00:30,046 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 66) ISPN000094: Received new cluster view: [slave:server-three-slave/web|3] [slave:server-three-slave/web, master:server-three/web]

[Server:server-three] 09:00:30,048 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 66) ISPN000079: Cache local address is master:server-three/web, physical addresses are [10.78.216.145:55450]

[Server:server-three] 09:00:30,052 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 66) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.10.Final

[Server:server-three] 09:00:30,063 INFO  [org.jboss.as.clustering] (MSC service thread 1-7) JBAS010238: Number of cluster members: 1

[Server:server-three] 09:00:30,089 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 66) ISPN000161: Using a batchMode transaction manager

[Server:server-three] 09:00:30,089 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 72) ISPN000161: Using a batchMode transaction manager

[Server:server-three] 09:00:30,234 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 72) ISPN000031: MBeans were successfully registered to the platform MBean server.

[Server:server-three] 09:00:30,234 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 66) ISPN000031: MBeans were successfully registered to the platform MBean server.

[Server:server-three] 09:00:30,317 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 72) JBAS010281: Started default-host/PeachFe cache from web container

[Server:server-three] 09:00:30,317 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 66) JBAS010281: Started repl cache from web container

[Server:server-three] 09:00:30,333 INFO  [org.jboss.web] (ServerService Thread Pool -- 66) JBAS018210: Register web context: /PeachFe

[Server:server-three] 09:00:30,380 INFO  [org.keycloak.adapters.as7.KeycloakAuthenticatorValve] (ServerService Thread Pool -- 66) **** using org.keycloak.json.adapterConfig

[Server:server-three] 09:00:30,380 INFO  [org.keycloak.adapters.as7.KeycloakAuthenticatorValve] (ServerService Thread Pool -- 66) {"realm" : "PEACH-ECMG", "resource" : "ECMG", "public-client" : true, "realm-public-key" : "MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCyxB4fma0gI2cDpHlsQuHjBrO3WnpeRAm+Kj8MHpv8Xji4+PM9xM+uCCAilbqYntjZZfqBThg21T/IpyKp3sQpmaE3mDUg+NkHZDe+apAbHed44eaLRZl3UyIprnxSnK1wm6Jc31smBrOdCh9P6A8FFkOnbLTLAcI72r73eHUMnQIDAQAB", "auth-server-url" : "https://10.78.217.164:8443/auth", "ssl-not-required" : false, "disable-trust-manager" : true}

[Server:server-three] 09:00:38,717 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS018559: Deployed "PeachFe.war" (runtime-name : "PeachFe.war")

[Server:server-three] 09:00:38,722 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss EAP 6.3.0.GA (AS 7.4.0.Final-redhat-19) started in 14439ms - Started 256 of 335 services (126 services are lazy, passive or on-demand)

From Slave:

[Server:server-three-slave] 08:48:37,778 INFO  [stdout] (ServerService Thread Pool -- 65)

[Server:server-three-slave] 08:48:37,779 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

[Server:server-three-slave] 08:48:37,779 INFO  [stdout] (ServerService Thread Pool -- 65) GMS: address=slave:server-three-slave/web, cluster=web, physical address=10.78.216.36:55450

[Server:server-three-slave] 08:48:37,779 INFO  [stdout] (ServerService Thread Pool -- 65) -------------------------------------------------------------------

[Server:server-three-slave] 08:48:37,880 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000094: Received new cluster view: [master:server-three/web|1] [master:server-three/web, slave:server-three-slave/web]

[Server:server-three-slave] 08:48:37,883 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000079: Cache local address is slave:server-three-slave/web, physical addresses are [10.78.216.36:55450]

[Server:server-three-slave] 08:48:37,886 INFO  [org.infinispan.factories.GlobalComponentRegistry] (ServerService Thread Pool -- 65) ISPN000128: Infinispan version: Infinispan 'Delirium' 5.2.10.Final

[Server:server-three-slave] 08:48:37,898 INFO  [org.jboss.as.clustering] (MSC service thread 1-3) JBAS010238: Number of cluster members: 2

[Server:server-three-slave] 08:48:37,938 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 65) ISPN000161: Using a batchMode transaction manager

[Server:server-three-slave] 08:48:37,938 INFO  [org.infinispan.factories.TransactionManagerFactory] (ServerService Thread Pool -- 72) ISPN000161: Using a batchMode transaction manager

[Server:server-three-slave] 08:48:38,102 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 65) ISPN000031: MBeans were successfully registered to the platform MBean server.

[Server:server-three-slave] 08:48:38,102 INFO  [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 72) ISPN000031: MBeans were successfully registered to the platform MBean server.

[Server:server-three-slave] 08:48:38,190 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 72) JBAS010281: Started default-host/PeachFe cache from web container

[Server:server-three-slave] 08:48:38,203 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 65) JBAS010281: Started repl cache from web container

[Server:server-three-slave] 08:48:38,215 INFO  [org.jboss.web] (ServerService Thread Pool -- 65) JBAS018210: Register web context: /PeachFe

[Server:server-three-slave] 08:48:38,255 INFO  [org.keycloak.adapters.as7.KeycloakAuthenticatorValve] (ServerService Thread Pool -- 65) **** using org.keycloak.json.adapterConfig

[Server:server-three-slave] 08:48:38,256 INFO  [org.keycloak.adapters.as7.KeycloakAuthenticatorValve] (ServerService Thread Pool -- 65) {"realm" : "PEACH-ECMG", "resource" : "ECMG", "public-client" : true, "realm-public-key" : "MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCyxB4fma0gI2cDpHlsQuHjBrO3WnpeRAm+Kj8MHpv8Xji4+PM9xM+uCCAilbqYntjZZfqBThg21T/IpyKp3sQpmaE3mDUg+NkHZDe+apAbHed44eaLRZl3UyIprnxSnK1wm6Jc31smBrOdCh9P6A8FFkOnbLTLAcI72r73eHUMnQIDAQAB", "auth-server-url" : "https://10.78.217.164:8443/auth", "ssl-not-required" : false, "disable-trust-manager" : true}

I'm stating my master first and slave second. In master( Number of cluster members: 1), slave( Number of cluster members: 2). Is this fine ??

Regards,

Joy.

[wdfink]

It's not only the NIC, if you are in different network sections or there is a link/router/firewall in between it might not work.

A simple test which use only JGroups can be found here TestingJBoss

[wdfink]

In that case the master should show a message that the cluster view has changed - and everything is fine from the JGroups cluster perspective.

Did your replication work now?

[rhusar]

Yes, this looks fine.

Received new cluster view: [master:server-three/web|1] [master:server-three/web, slave:server-three-slave/web]

So the cluster has formed with the 2 members being the 2 nodes you started.

[joysagar]

Thanks Radoslav,

By this can we say the cluster is formed and session replication is happening ?

Thanks Wolf-Dieter Fink for the response,

I will try out the test you have provided and let you know the results,

Actually, we are planning to have a physical hardware balancer so i don't require modcluster.

I want to form a simple cluster without any load balancer and the session replication should be happening at the same time.It'd be of great help if anyone of you provide me documentation related to cluster alone. In my case do i need to use httpd ? My web application supports https, if i use httpd, is it manadatory to provide sslcertificate and sslcertificatekey at my ssl.conf ?

Please help me out with the answers,

Regards,

Joy.

[joysagar]

Hi All,

We have decided not to use any (software) load balancer and httpd. So i brought a simple cluster where session replication is expected to happen,

I can see the following lines in my master and slave machines,

Master:

[Server:server-three] 05:17:53,401 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,shared=udp) ISPN000094: Received new cluster view: [master:server-three/web|1] [master:server-three/web, slave:server-three-slave/web]

Slave:

[Server:server-three-slave] 05:17:38,446 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 65) ISPN000094: Received new cluster view: [master:server-three/web|1] [master:server-three/web, slave:server-three-slave/web]

In our application, we have few REST calls,and few of the REST calls have internal redirection calls.

Issue:If a request is triggered and falls on master and gets redirected to another system and comes back with the response and if it falls on slave a new JSessionID is getting created and it is treating it as a new session.Does this mean, session replication is not happening ??. As we are not using mod_cluster, i can't specify sticky sessions anywhere.

FYI: I found a simple war file(https://code.google.com/p/hsr-test/) to test the session replication. I used software(Zen) load balancer just for testing purpose and i found the counter is getting incremented whenever any one of the machine is down. I believe its session replication is happening. Can any one kindly clarify this as well.

Class file :https://code.google.com/p/hsr-test/source/browse/trunk/hsr-test/src/com/mouseoverstudio/test/HSRTest.java

Any help will be much appreciated,

Thanks,

Regards,

Joy.

[joysagar]

Hi Radoslav,

Recently i'm facing a different problem. Please find below log event,

07:14:50,084 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 75) ISPN000094: Received new cluster view: [master:server-three/web|6] [master:server-three/web, slave:server-three-slave/web, master:server-three/web, slave:server-three-slave/web]

[Server:server-three-slave] 07:14:50,090 ERROR [org.jgroups.JChannel] (ServerService Thread Pool -- 75) exception in channelConnected() callback: java.lang.IllegalStateException: JBAS010272: A node named slave:server-three-slave already exists in this cluster. Perhaps there is already a server running on this host? If so, restart this server with a unique node name, via -Djboss.node.name=<node-name>

It is not able to clear the previous cluster formed. We can see it here, [master:server-three/web, slave:server-three-slave/web, master:server-three/web, slave:server-three-slave/web]

Please help me out in clearing this,

I have tried restarting the server even the machines. No use,

Thanks,

Regards,

Joy

[rhusar]

To run 2 nodes on the same physical host, you need to configure a unique "jboss.node.name" property for that host.