7 Replies Latest reply on Jul 25, 2013 6:11 AM by nadirx

Not able to form a cluster of two nodes using Infinispan server

raihan26 Jul 19, 2013 7:03 PM

I am planning to use Infinispan as the remote cache cluster that will sit between my application and database. Now I am planning to make a cluster of few nodes, let's say 2 nodes. I started the Infinispan server using the bin/clustered.sh script on both of my Ubuntu machines without modifying any files. I just started infinispan server using bin/clustered.sh script on both of the machines.

rkost@rj-vm9-14105:~/infinispan/infinispan-server-6.0.0.Alpha1/bin$ uname -a Linux rj-vm9-14105 2.6.35-22-server #33-Ubuntu SMP Sun Sep 19 20:48:58 UTC 2010 x86_64 GNU/Linux

But somehow they are not able to form a cluster and I am not getting any exception as well. I am thinking, it might be because of Multicast thing? I have checked Multicast is enabled as well. So might be I need to make sure of some port configuration? Not sure on this.

I am wondering, how one machine will be aware of other nodes joining the cluster with the use of multicast? Do I need to modify some configuration files to let them know that these nodes are joining or have joined?

Machine 1 console-

root@rj-raptor-vm1-14938:/home/rkost/infinispan/infinispan-server-6.0.0.Alpha1/bin# sh clustered.sh

=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /home/rkost/infinispan/infinispan-server-6.0.0.Alpha1

JAVA: java

JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

=========================================================================

23:16:14,426 INFO [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1

23:16:14,874 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA

23:16:14,934 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015899: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) starting

23:16:15,784 INFO [org.xnio] (MSC service thread 1-5) XNIO Version 3.0.7.GA

23:16:15,788 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)

23:16:15,792 INFO [org.xnio.nio] (MSC service thread 1-5) XNIO NIO Implementation Version 3.0.7.GA

23:16:15,799 INFO [org.jboss.remoting] (MSC service thread 1-5) JBoss Remoting version 3.2.14.GA

23:16:15,865 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.

23:16:15,865 INFO [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem

23:16:15,868 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.

23:16:15,876 INFO [org.jboss.as.security] (MSC service thread 1-4) JBAS013170: Current PicketBox version=4.0.15.Final

23:16:15,879 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem

23:16:15,921 INFO [org.jboss.as.connector.logging] (MSC service thread 1-6) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)

23:16:15,926 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]

23:16:15,957 INFO [org.jboss.as.naming] (MSC service thread 1-9) JBAS011802: Starting Naming Service

23:16:16,073 INFO [org.apache.coyote.http11] (MSC service thread 1-3) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/127.0.0.1:8080

23:16:16,080 INFO [org.apache.coyote.ajp] (MSC service thread 1-16) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/127.0.0.1:8009

23:16:16,087 INFO [org.apache.coyote.http11] (MSC service thread 1-3) JBWEB003000: Coyote HTTP/1.1 starting on: http-/127.0.0.1:8080

23:16:16,106 INFO [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final

23:16:16,135 INFO [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364

23:16:16,253 INFO [org.jboss.as.remoting] (MSC service thread 1-16) JBAS017100: Listening on 127.0.0.1:4447

23:16:16,253 INFO [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 127.0.0.1:9999

23:16:16,407 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-3) ISPN000128: Infinispan version: Infinispan '<TBD>' 6.0.0.Alpha1

23:16:16,416 INFO [org.infinispan.server.endpoint] (MSC service thread 1-6) JDGS010000: WebSocketServer starting

23:16:16,416 INFO [org.infinispan.server.endpoint] (MSC service thread 1-11) JDGS010000: HotRodServer starting

23:16:16,416 INFO [org.infinispan.server.endpoint] (MSC service thread 1-6) JDGS010001: WebSocketServer listening on 127.0.0.1:8181

23:16:16,416 INFO [org.infinispan.server.endpoint] (MSC service thread 1-11) JDGS010001: HotRodServer listening on 127.0.0.1:11222

23:16:16,512 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-15) ISPN000078: Starting JGroups Channel

23:16:16,523 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-15) [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

23:16:16,524 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-15) [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

23:16:16,524 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-15) [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

23:16:16,525 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-15) [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

23:16:16,528 INFO [stdout] (MSC service thread 1-15)

23:16:16,528 INFO [stdout] (MSC service thread 1-15) -------------------------------------------------------------------

23:16:16,528 INFO [stdout] (MSC service thread 1-15) GMS: address=rj-raptor-vm1-14938/clustered, cluster=clustered, physical address=127.0.0.1:55200

23:16:16,528 INFO [stdout] (MSC service thread 1-15) -------------------------------------------------------------------

23:16:16,550 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-7) ISPN000031: MBeans were successfully registered to the platform MBean server.

23:16:16,550 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.

23:16:16,552 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) JBAS010281: Started jboss-web-policy cache from security container

23:16:16,552 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started other cache from security container

23:16:16,561 INFO [org.infinispan.server.endpoint] (MSC service thread 1-8) JDGS010000: REST starting

23:16:16,756 INFO [org.infinispan.server.endpoint] (MSC service thread 1-8) JDGS010002: REST mapped to /rest

23:16:21,546 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-15) ISPN000094: Received new cluster view: [rj-raptor-vm1-14938/clustered|0] [rj-raptor-vm1-14938/clustered]

23:16:21,620 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-15) ISPN000079: Cache local address is rj-raptor-vm1-14938/clustered, physical addresses are [127.0.0.1:55200]

23:16:21,664 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-15) ISPN000031: MBeans were successfully registered to the platform MBean server.

23:16:21,664 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.

23:16:21,665 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-16) ISPN000031: MBeans were successfully registered to the platform MBean server.

23:16:21,675 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started memcachedCache cache from clustered container

23:16:21,675 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-15) JBAS010281: Started default cache from clustered container

23:16:21,675 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-16) JBAS010281: Started namedCache cache from clustered container

23:16:21,676 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: MemcachedServer starting

23:16:21,677 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010001: MemcachedServer listening on 127.0.0.1:11211

23:16:22,216 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-11) ISPN000031: MBeans were successfully registered to the platform MBean server.

23:16:22,220 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-11) JBAS010281: Started ___hotRodTopologyCache cache from clustered container

23:16:22,319 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management

23:16:22,320 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990

23:16:22,320 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) started in 8264ms - Started 93 of 142 services (49 services are passive or on-demand)

Machine 2 console (And then I started Infinispan server on machine 2 after I started on machine 1)

root@rj-raptor-vm2-14939:/home/rkost/infinispan/infinispan-server-6.0.0.Alpha1/bin# sh clustered.sh

=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /home/rkost/infinispan/infinispan-server-6.0.0.Alpha1

JAVA: java

=========================================================================

22:45:59,604 INFO [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1

22:45:59,855 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA

22:45:59,917 INFO [org.jboss.as] (MSC service thread 1-8) JBAS015899: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) starting

22:46:00,731 INFO [org.xnio] (MSC service thread 1-16) XNIO Version 3.0.7.GA

22:46:00,734 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)

22:46:00,740 INFO [org.xnio.nio] (MSC service thread 1-16) XNIO NIO Implementation Version 3.0.7.GA

22:46:00,748 INFO [org.jboss.remoting] (MSC service thread 1-16) JBoss Remoting version 3.2.14.GA

22:46:00,773 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.

22:46:00,797 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.

22:46:00,800 INFO [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem

22:46:00,801 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem

22:46:00,803 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]

22:46:00,809 INFO [org.jboss.as.security] (MSC service thread 1-14) JBAS013170: Current PicketBox version=4.0.15.Final

22:46:00,824 INFO [org.jboss.as.connector.logging] (MSC service thread 1-4) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)

22:46:00,828 INFO [org.jboss.as.naming] (MSC service thread 1-5) JBAS011802: Starting Naming Service

22:46:00,948 INFO [org.apache.coyote.http11] (MSC service thread 1-6) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/127.0.0.1:8080

22:46:00,951 INFO [org.apache.coyote.ajp] (MSC service thread 1-9) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/127.0.0.1:8009

22:46:00,960 INFO [org.apache.coyote.http11] (MSC service thread 1-6) JBWEB003000: Coyote HTTP/1.1 starting on: http-/127.0.0.1:8080

22:46:00,986 INFO [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final

22:46:00,995 INFO [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364

22:46:01,181 INFO [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 127.0.0.1:9999

22:46:01,181 INFO [org.jboss.as.remoting] (MSC service thread 1-7) JBAS017100: Listening on 127.0.0.1:4447

22:46:01,302 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-13) ISPN000128: Infinispan version: Infinispan '<TBD>' 6.0.0.Alpha1

22:46:01,355 INFO [org.infinispan.server.endpoint] (MSC service thread 1-5) JDGS010000: HotRodServer starting

22:46:01,356 INFO [org.infinispan.server.endpoint] (MSC service thread 1-5) JDGS010001: HotRodServer listening on 127.0.0.1:11222

22:46:01,357 INFO [org.infinispan.server.endpoint] (MSC service thread 1-15) JDGS010000: WebSocketServer starting

22:46:01,358 INFO [org.infinispan.server.endpoint] (MSC service thread 1-15) JDGS010001: WebSocketServer listening on 127.0.0.1:8181

22:46:01,458 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000078: Starting JGroups Channel

22:46:01,472 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-12) [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

22:46:01,473 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-12) [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

22:46:01,474 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-12) [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

22:46:01,474 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-12) [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

22:46:01,478 INFO [stdout] (MSC service thread 1-12)

22:46:01,478 INFO [stdout] (MSC service thread 1-12) -------------------------------------------------------------------

22:46:01,478 INFO [stdout] (MSC service thread 1-12) GMS: address=rj-raptor-vm2-14939/clustered, cluster=clustered, physical address=127.0.0.1:55200

22:46:01,478 INFO [stdout] (MSC service thread 1-12) -------------------------------------------------------------------

22:46:01,483 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-8) ISPN000031: MBeans were successfully registered to the platform MBean server.

22:46:01,483 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-13) ISPN000031: MBeans were successfully registered to the platform MBean server.

22:46:01,486 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010281: Started jboss-web-policy cache from security container

22:46:01,486 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-13) JBAS010281: Started other cache from security container

22:46:01,495 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010000: REST starting

22:46:01,707 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010002: REST mapped to /rest

22:46:06,495 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000094: Received new cluster view: [rj-raptor-vm2-14939/clustered|0] [rj-raptor-vm2-14939/clustered]

22:46:06,572 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000079: Cache local address is rj-raptor-vm2-14939/clustered, physical addresses are [127.0.0.1:55200]

22:46:06,618 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-12) ISPN000031: MBeans were successfully registered to the platform MBean server.

22:46:06,618 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.

22:46:06,618 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-6) ISPN000031: MBeans were successfully registered to the platform MBean server.

22:46:06,628 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-12) JBAS010281: Started memcachedCache cache from clustered container

22:46:06,628 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started namedCache cache from clustered container

22:46:06,628 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) JBAS010281: Started default cache from clustered container

22:46:06,628 INFO [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010000: MemcachedServer starting

22:46:06,628 INFO [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010001: MemcachedServer listening on 127.0.0.1:11211

22:46:07,052 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-5) ISPN000031: MBeans were successfully registered to the platform MBean server.

22:46:07,056 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010281: Started ___hotRodTopologyCache cache from clustered containerReply to original post

22:46:07,143 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management

22:46:07,144 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990

22:46:07,144 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) started in 7780ms - Started 93 of 142 services (49 services are passive or on-demand)

As per the above logs, I don't think so they are forming a cluster of two nodes. Something is wrong for sure. Can anyone help me out here?

1. Re: Not able to form a cluster of two nodes using Infinispan server

mircea.markus Jul 23, 2013 9:34 AM (in response to raihan26)

seems like the server binds to the loopback interface (127.0.0.1) instead of the LAN.
Try adding -Djgroups.bind_addr=<IP of the network card used in LAN> into the script that launces the JVM.
Actions
2. Re: Not able to form a cluster of two nodes using Infinispan server

raihan26 Jul 23, 2013 2:03 PM (in response to mircea.markus)

I just did ifconfig on my ubuntu machine and I got below result-

root@rj-raptor-vm2-14939:/home/rjamal/shellapp/shellrapParent# ifconfig
eth0      Link encap:Ethernet HWaddr fa:16:3e:ec:c1:6b
          inet addr:10.9.240.220 Bcast:10.9.240.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:feec:c16b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:472064 errors:0 dropped:0 overruns:0 frame:0
          TX packets:239982 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:665447412 (665.4 MB) TX bytes:30777275 (30.7 MB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:9112 errors:0 dropped:0 overruns:0 frame:0
          TX packets:9112 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1215875 (1.2 MB) TX bytes:1215875 (1.2 MB)

And you want me to use the bold letter IP Address right? If yes, then it should be something like this while launching-

     sh clustered.sh -Djgroups.bind_addr=10.9.240.220

Let me know if this is right? And I need to do the same thing for other nodes as well.. Right?
Actions
3. Re: Not able to form a cluster of two nodes using Infinispan server

nadirx Jul 23, 2013 3:22 PM (in response to raihan26)

Yes.
Actions
4. Re: Not able to form a cluster of two nodes using Infinispan server

raihan26 Jul 23, 2013 3:32 PM (in response to nadirx)

I just did the same thing on both of my nodes. Below is the machine logs-

Machine 1 console logs

root@rj-raptor-vm1-14938:/home/rkost/infinispan/infinispan-server-6.0.0.Alpha1/bin# sh clustered.sh -Djgroups.bind_addr=10.9.251.184
=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /home/rkost/infinispan/infinispan-server-6.0.0.Alpha1

JAVA: java

JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

=========================================================================

19:29:12,163 INFO [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1
19:29:12,413 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA
19:29:12,475 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015899: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) starting
19:29:13,331 INFO [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.7.GA
19:29:13,333 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
19:29:13,338 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.7.GA
19:29:13,346 INFO [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 3.2.14.GA
19:29:13,408 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.
19:29:13,411 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.
19:29:13,413 INFO [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem
19:29:13,417 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem
19:29:13,430 INFO [org.jboss.as.security] (MSC service thread 1-14) JBAS013170: Current PicketBox version=4.0.15.Final
19:29:13,445 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]
19:29:13,458 INFO [org.jboss.as.connector.logging] (MSC service thread 1-6) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)
19:29:13,465 INFO [org.jboss.as.naming] (MSC service thread 1-13) JBAS011802: Starting Naming Service
19:29:13,617 INFO [org.apache.coyote.http11] (MSC service thread 1-8) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/127.0.0.1:8080
19:29:13,624 INFO [org.apache.coyote.ajp] (MSC service thread 1-16) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/127.0.0.1:8009
19:29:13,627 INFO [org.apache.coyote.http11] (MSC service thread 1-8) JBWEB003000: Coyote HTTP/1.1 starting on: http-/127.0.0.1:8080
19:29:13,662 INFO [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final
19:29:13,676 INFO [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364
19:29:13,844 INFO [org.jboss.as.remoting] (MSC service thread 1-10) JBAS017100: Listening on 127.0.0.1:4447
19:29:13,844 INFO [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100: Listening on 127.0.0.1:9999
19:29:13,996 INFO [org.infinispan.server.endpoint] (MSC service thread 1-12) JDGS010000: HotRodServer starting
19:29:13,996 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: WebSocketServer starting
19:29:13,997 INFO [org.infinispan.server.endpoint] (MSC service thread 1-12) JDGS010001: HotRodServer listening on 127.0.0.1:11222
19:29:13,997 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010001: WebSocketServer listening on 127.0.0.1:8181
19:29:13,998 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-7) ISPN000128: Infinispan version: Infinispan '<TBD>' 6.0.0.Alpha1
19:29:14,118 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-9) ISPN000078: Starting JGroups Channel
19:29:14,131 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-9) [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
19:29:14,132 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-9) [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
19:29:14,132 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-9) [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
19:29:14,132 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-9) [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
19:29:14,136 INFO [stdout] (MSC service thread 1-9)
19:29:14,136 INFO [stdout] (MSC service thread 1-9) -------------------------------------------------------------------
19:29:14,136 INFO [stdout] (MSC service thread 1-9) GMS: address=rj-raptor-vm1-14938/clustered, cluster=clustered, physical address=10.9.251.184:55200
19:29:14,136 INFO [stdout] (MSC service thread 1-9) -------------------------------------------------------------------
19:29:14,162 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-7) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:29:14,162 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-15) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:29:14,165 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) JBAS010281: Started jboss-web-policy cache from security container
19:29:14,165 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-15) JBAS010281: Started other cache from security container
19:29:14,176 INFO [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010000: REST starting
19:29:14,367 INFO [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010002: REST mapped to /rest
19:29:19,150 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-9) ISPN000094: Received new cluster view: [rj-raptor-vm1-14938/clustered|0] [rj-raptor-vm1-14938/clustered]
19:29:19,252 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-9) ISPN000079: Cache local address is rj-raptor-vm1-14938/clustered, physical addresses are [10.9.251.184:55200]
19:29:19,293 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-6) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:29:19,293 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-4) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:29:19,294 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-9) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:29:19,307 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010281: Started memcachedCache cache from clustered container
19:29:19,307 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) JBAS010281: Started namedCache cache from clustered container
19:29:19,307 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010281: Started default cache from clustered container
19:29:19,307 INFO [org.infinispan.server.endpoint] (MSC service thread 1-9) JDGS010000: MemcachedServer starting
19:29:19,308 INFO [org.infinispan.server.endpoint] (MSC service thread 1-9) JDGS010001: MemcachedServer listening on 127.0.0.1:11211
19:29:19,755 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-12) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:29:19,759 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-12) JBAS010281: Started ___hotRodTopologyCache cache from clustered container
19:29:19,859 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management
19:29:19,860 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990
19:29:19,860 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) started in 7944ms - Started 93 of 142 services (49 services are passive or on-demand)

Machine 2 console

root@rj-raptor-vm2-14939:/home/rkost/infinispan/infinispan-server-6.0.0.Alpha1/bin# sh clustered.sh -Djgroups.bind_addr=10.9.240.220
=========================================================================

JBoss Bootstrap Environment

JBOSS_HOME: /home/rkost/infinispan/infinispan-server-6.0.0.Alpha1

JAVA: java

JAVA_OPTS: -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

=========================================================================

19:28:08,673 INFO [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1
19:28:08,932 INFO [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA
19:28:08,992 INFO [org.jboss.as] (MSC service thread 1-7) JBAS015899: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) starting
19:28:09,938 INFO [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)
19:28:09,938 INFO [org.xnio] (MSC service thread 1-4) XNIO Version 3.0.7.GA
19:28:09,949 INFO [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.0.7.GA
19:28:09,958 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 3.2.14.GA
19:28:10,021 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.
19:28:10,029 INFO [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.
19:28:10,033 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem
19:28:10,041 INFO [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem
19:28:10,045 INFO [org.jboss.as.security] (MSC service thread 1-2) JBAS013170: Current PicketBox version=4.0.15.Final
19:28:10,051 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]
19:28:10,060 INFO [org.jboss.as.connector.logging] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)
19:28:10,079 INFO [org.jboss.as.naming] (MSC service thread 1-10) JBAS011802: Starting Naming Service
19:28:10,208 INFO [org.apache.coyote.http11] (MSC service thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/127.0.0.1:8080
19:28:10,217 INFO [org.apache.coyote.http11] (MSC service thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/127.0.0.1:8080
19:28:10,221 INFO [org.apache.coyote.ajp] (MSC service thread 1-7) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/127.0.0.1:8009
19:28:10,244 INFO [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final
19:28:10,265 INFO [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364
19:28:10,414 INFO [org.jboss.as.remoting] (MSC service thread 1-15) JBAS017100: Listening on 127.0.0.1:9999
19:28:10,414 INFO [org.jboss.as.remoting] (MSC service thread 1-11) JBAS017100: Listening on 127.0.0.1:4447
19:28:10,569 INFO [org.infinispan.server.endpoint] (MSC service thread 1-8) JDGS010000: WebSocketServer starting
19:28:10,569 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: HotRodServer starting
19:28:10,570 INFO [org.infinispan.server.endpoint] (MSC service thread 1-8) JDGS010001: WebSocketServer listening on 127.0.0.1:8181
19:28:10,570 INFO [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010001: HotRodServer listening on 127.0.0.1:11222
19:28:10,622 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-4) ISPN000128: Infinispan version: Infinispan '<TBD>' 6.0.0.Alpha1
19:28:10,746 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-11) ISPN000078: Starting JGroups Channel
19:28:10,785 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-11) [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
19:28:10,785 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-11) [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
19:28:10,785 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-11) [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
19:28:10,786 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-11) [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
19:28:10,790 INFO [stdout] (MSC service thread 1-11)
19:28:10,790 INFO [stdout] (MSC service thread 1-11) -------------------------------------------------------------------
19:28:10,790 INFO [stdout] (MSC service thread 1-11) GMS: address=rj-raptor-vm2-14939/clustered, cluster=clustered, physical address=10.9.240.220:55200
19:28:10,790 INFO [stdout] (MSC service thread 1-11) -------------------------------------------------------------------
19:28:10,815 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-4) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:28:10,815 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-9) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:28:10,819 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010281: Started other cache from security container
19:28:10,819 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010281: Started jboss-web-policy cache from security container
19:28:10,835 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010000: REST starting
19:28:11,020 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010002: REST mapped to /rest
19:28:15,807 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-11) ISPN000094: Received new cluster view: [rj-raptor-vm2-14939/clustered|0] [rj-raptor-vm2-14939/clustered]
19:28:15,897 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-11) ISPN000079: Cache local address is rj-raptor-vm2-14939/clustered, physical addresses are [10.9.240.220:55200]
19:28:15,934 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-11) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:28:15,934 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:28:15,934 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-10) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:28:15,944 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started memcachedCache cache from clustered container
19:28:15,944 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-10) JBAS010281: Started namedCache cache from clustered container
19:28:15,944 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-11) JBAS010281: Started default cache from clustered container
19:28:15,945 INFO [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010000: MemcachedServer starting
19:28:15,945 INFO [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010001: MemcachedServer listening on 127.0.0.1:11211
19:28:16,603 INFO [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.
19:28:16,608 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started ___hotRodTopologyCache cache from clustered container
19:28:16,703 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management
19:28:16,703 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990
19:28:16,704 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) started in 8283ms - Started 93 of 142 services (49 services are passive or on-demand)

And I don't think so they have formed a cluster of two nodes. Is there any other things missing? Might be multicast issue? How do I make sure multicast is enabled properly and any other settings are also configured properly?

Thanks for the help on this.
Actions
5. Re: Not able to form a cluster of two nodes using Infinispan server

mircea.markus Jul 24, 2013 12:55 PM (in response to raihan26)

what's the ifconfig for the 2nd node?

Also this migth be helpful: http://www.jgroups.org/tutorial/html/ch01.html#d0e130
Actions
6. Re: Not able to form a cluster of two nodes using Infinispan server

raihan26 Jul 24, 2013 2:30 PM (in response to mircea.markus)

This is the ifconfig for the other node-

root@rj-raptor-vm1-14938:/home/rjamal/infinispan/infinispan-server-6.0.0.Alpha1/bin# ifconfig
eth0      Link encap:Ethernet HWaddr fa:16:3e:92:59:12
          inet addr:10.9.251.184 Bcast:10.9.251.255 Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe92:5912/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:604506 errors:0 dropped:0 overruns:0 frame:0
          TX packets:376884 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:838973473 (838.9 MB) TX bytes:67117844 (67.1 MB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:34128 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34128 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5990228 (5.9 MB) TX bytes:5990228 (5.9 MB)

One quick thing how the other node will figure out that any other nodes is joining. Bcoz I am not changing any config file which will tell about the other nodes.
Actions
7. Re: Not able to form a cluster of two nodes using Infinispan server

nadirx Jul 25, 2013 6:11 AM (in response to raihan26)

One quick thing how the other node will figure out that any other nodes is joining. Bcoz I am not changing any config file which will tell about the other nodes.

It can be done in a variety of ways. By default using UDP multicast to broadcast an "I want to join" message to an existing cluster. This generally works for all nodes on the same subnet, which is the problem in your configuration:

the two nodes belong to different subnets:
Node 1 inet addr:10.9.240.220 Bcast:10.9.240.255 Mask:255.255.255.0
Node 2 inet addr:10.9.251.184 Bcast:10.9.251.255 Mask:255.255.255.0

Node 1 belongs to a network 10.9.240.0/24
Node 2 belongs to a network 10.9.251.0/24

Make them belong to the same network and you're set.

Tristan
Actions

Go to original post