7 Replies Latest reply on Jul 25, 2013 6:11 AM by Tristan Tarrant

    Not able to form a cluster of two nodes using Infinispan server

    Raihan Jamal Newbie

      I am planning to use Infinispan as the remote cache cluster that will sit between my application and database. Now I am planning to make a cluster of few nodes, let's say 2 nodes. I started the Infinispan server using the bin/clustered.sh script on both of my Ubuntu machines without modifying any files. I just started infinispan server using bin/clustered.sh script on both of the machines.

       

      rkost@rj-vm9-14105:~/infinispan/infinispan-server-6.0.0.Alpha1/bin$ uname -a

      Linux rj-vm9-14105 2.6.35-22-server #33-Ubuntu SMP Sun Sep 19
      20:48:58 UTC 2010 x86_64 GNU/Linux

       

      But somehow they are not able to form a cluster and I am not getting any exception as well. I am thinking, it might be because of Multicast thing? I have checked Multicast is enabled as well. So might be I need to make sure of some port configuration? Not sure on this.

       

      I am wondering, how one machine will be aware of other nodes joining the cluster with the use of multicast? Do I need to modify some configuration files to let them know that these nodes are joining or have joined?

       

       

      Machine 1 console-

       

       

       

          root@rj-raptor-vm1-14938:/home/rkost/infinispan/infinispan-server-6.0.0.Alpha1/bin# sh clustered.sh

          =========================================================================

          

          

            JBoss Bootstrap Environment

          

          

            JBOSS_HOME: /home/rkost/infinispan/infinispan-server-6.0.0.Alpha1

          

          

            JAVA: java

          

          

            JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

          

          

          =========================================================================

          

          

          23:16:14,426 INFO  [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1

          23:16:14,874 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA

          23:16:14,934 INFO  [org.jboss.as] (MSC service thread 1-7) JBAS015899: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) starting

          23:16:15,784 INFO  [org.xnio] (MSC service thread 1-5) XNIO Version 3.0.7.GA

          23:16:15,788 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)

          23:16:15,792 INFO  [org.xnio.nio] (MSC service thread 1-5) XNIO NIO Implementation Version 3.0.7.GA

          23:16:15,799 INFO  [org.jboss.remoting] (MSC service thread 1-5) JBoss Remoting version 3.2.14.GA

          23:16:15,865 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.

          23:16:15,865 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem

          23:16:15,868 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.

          23:16:15,876 INFO  [org.jboss.as.security] (MSC service thread 1-4) JBAS013170: Current PicketBox version=4.0.15.Final

          23:16:15,879 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem

          23:16:15,921 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-6) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)

          23:16:15,926 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]

          23:16:15,957 INFO  [org.jboss.as.naming] (MSC service thread 1-9) JBAS011802: Starting Naming Service

          23:16:16,073 INFO  [org.apache.coyote.http11] (MSC service thread 1-3) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/127.0.0.1:8080

          23:16:16,080 INFO  [org.apache.coyote.ajp] (MSC service thread 1-16) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/127.0.0.1:8009

          23:16:16,087 INFO  [org.apache.coyote.http11] (MSC service thread 1-3) JBWEB003000: Coyote HTTP/1.1 starting on: http-/127.0.0.1:8080

          23:16:16,106 INFO  [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final

          23:16:16,135 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364

          23:16:16,253 INFO  [org.jboss.as.remoting] (MSC service thread 1-16) JBAS017100: Listening on 127.0.0.1:4447

          23:16:16,253 INFO  [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 127.0.0.1:9999

          23:16:16,407 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-3) ISPN000128: Infinispan version: Infinispan '<TBD>' 6.0.0.Alpha1

          23:16:16,416 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-6) JDGS010000: WebSocketServer starting

          23:16:16,416 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-11) JDGS010000: HotRodServer starting

          23:16:16,416 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-6) JDGS010001: WebSocketServer listening on 127.0.0.1:8181

          23:16:16,416 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-11) JDGS010001: HotRodServer listening on 127.0.0.1:11222

          23:16:16,512 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-15) ISPN000078: Starting JGroups Channel

          23:16:16,523 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-15) [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

          23:16:16,524 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-15) [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

          23:16:16,524 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-15) [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

          23:16:16,525 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-15) [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

          23:16:16,528 INFO  [stdout] (MSC service thread 1-15)

          23:16:16,528 INFO  [stdout] (MSC service thread 1-15) -------------------------------------------------------------------

          23:16:16,528 INFO  [stdout] (MSC service thread 1-15) GMS: address=rj-raptor-vm1-14938/clustered, cluster=clustered, physical address=127.0.0.1:55200

          23:16:16,528 INFO  [stdout] (MSC service thread 1-15) -------------------------------------------------------------------

          23:16:16,550 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-7) ISPN000031: MBeans were successfully registered to the platform MBean server.

          23:16:16,550 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.

          23:16:16,552 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) JBAS010281: Started jboss-web-policy cache from security container

          23:16:16,552 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started other cache from security container

          23:16:16,561 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-8) JDGS010000: REST starting

          23:16:16,756 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-8) JDGS010002: REST mapped to /rest

          23:16:21,546 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-15) ISPN000094: Received new cluster view: [rj-raptor-vm1-14938/clustered|0] [rj-raptor-vm1-14938/clustered]

          23:16:21,620 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-15) ISPN000079: Cache local address is rj-raptor-vm1-14938/clustered, physical addresses are [127.0.0.1:55200]

          23:16:21,664 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-15) ISPN000031: MBeans were successfully registered to the platform MBean server.

          23:16:21,664 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.

          23:16:21,665 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-16) ISPN000031: MBeans were successfully registered to the platform MBean server.

          23:16:21,675 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started memcachedCache cache from clustered container

          23:16:21,675 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-15) JBAS010281: Started default cache from clustered container

          23:16:21,675 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-16) JBAS010281: Started namedCache cache from clustered container

          23:16:21,676 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: MemcachedServer starting

          23:16:21,677 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010001: MemcachedServer listening on 127.0.0.1:11211

          23:16:22,216 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-11) ISPN000031: MBeans were successfully registered to the platform MBean server.

          23:16:22,220 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-11) JBAS010281: Started ___hotRodTopologyCache cache from clustered container

          23:16:22,319 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management

          23:16:22,320 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990

          23:16:22,320 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) started in 8264ms - Started 93 of 142 services (49 services are passive or on-demand)

       

       

       

       

      Machine 2 console (And then I started Infinispan server on machine 2 after I started on machine 1)

       

       

       

          root@rj-raptor-vm2-14939:/home/rkost/infinispan/infinispan-server-6.0.0.Alpha1/bin# sh clustered.sh

          =========================================================================

          

          

            JBoss Bootstrap Environment

          

          

            JBOSS_HOME: /home/rkost/infinispan/infinispan-server-6.0.0.Alpha1

          

          

            JAVA: java

          

          

            JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

          

          

          =========================================================================

          

          

          22:45:59,604 INFO  [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1

          22:45:59,855 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA

          22:45:59,917 INFO  [org.jboss.as] (MSC service thread 1-8) JBAS015899: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) starting

          22:46:00,731 INFO  [org.xnio] (MSC service thread 1-16) XNIO Version 3.0.7.GA

          22:46:00,734 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)

          22:46:00,740 INFO  [org.xnio.nio] (MSC service thread 1-16) XNIO NIO Implementation Version 3.0.7.GA

          22:46:00,748 INFO  [org.jboss.remoting] (MSC service thread 1-16) JBoss Remoting version 3.2.14.GA

          22:46:00,773 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.

          22:46:00,797 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.

          22:46:00,800 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem

          22:46:00,801 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem

          22:46:00,803 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]

          22:46:00,809 INFO  [org.jboss.as.security] (MSC service thread 1-14) JBAS013170: Current PicketBox version=4.0.15.Final

          22:46:00,824 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-4) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)

          22:46:00,828 INFO  [org.jboss.as.naming] (MSC service thread 1-5) JBAS011802: Starting Naming Service

          22:46:00,948 INFO  [org.apache.coyote.http11] (MSC service thread 1-6) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/127.0.0.1:8080

          22:46:00,951 INFO  [org.apache.coyote.ajp] (MSC service thread 1-9) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/127.0.0.1:8009

          22:46:00,960 INFO  [org.apache.coyote.http11] (MSC service thread 1-6) JBWEB003000: Coyote HTTP/1.1 starting on: http-/127.0.0.1:8080

          22:46:00,986 INFO  [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final

          22:46:00,995 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364

          22:46:01,181 INFO  [org.jboss.as.remoting] (MSC service thread 1-4) JBAS017100: Listening on 127.0.0.1:9999

          22:46:01,181 INFO  [org.jboss.as.remoting] (MSC service thread 1-7) JBAS017100: Listening on 127.0.0.1:4447

          22:46:01,302 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-13) ISPN000128: Infinispan version: Infinispan '<TBD>' 6.0.0.Alpha1

          22:46:01,355 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-5) JDGS010000: HotRodServer starting

          22:46:01,356 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-5) JDGS010001: HotRodServer listening on 127.0.0.1:11222

          22:46:01,357 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-15) JDGS010000: WebSocketServer starting

          22:46:01,358 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-15) JDGS010001: WebSocketServer listening on 127.0.0.1:8181

          22:46:01,458 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000078: Starting JGroups Channel

          22:46:01,472 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-12) [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

          22:46:01,473 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-12) [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

          22:46:01,474 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-12) [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

          22:46:01,474 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-12) [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

          22:46:01,478 INFO  [stdout] (MSC service thread 1-12)

          22:46:01,478 INFO  [stdout] (MSC service thread 1-12) -------------------------------------------------------------------

          22:46:01,478 INFO  [stdout] (MSC service thread 1-12) GMS: address=rj-raptor-vm2-14939/clustered, cluster=clustered, physical address=127.0.0.1:55200

          22:46:01,478 INFO  [stdout] (MSC service thread 1-12) -------------------------------------------------------------------

          22:46:01,483 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-8) ISPN000031: MBeans were successfully registered to the platform MBean server.

          22:46:01,483 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-13) ISPN000031: MBeans were successfully registered to the platform MBean server.

          22:46:01,486 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-8) JBAS010281: Started jboss-web-policy cache from security container

          22:46:01,486 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-13) JBAS010281: Started other cache from security container

          22:46:01,495 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010000: REST starting

          22:46:01,707 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010002: REST mapped to /rest

          22:46:06,495 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000094: Received new cluster view: [rj-raptor-vm2-14939/clustered|0] [rj-raptor-vm2-14939/clustered]

          22:46:06,572 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-12) ISPN000079: Cache local address is rj-raptor-vm2-14939/clustered, physical addresses are [127.0.0.1:55200]

          22:46:06,618 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-12) ISPN000031: MBeans were successfully registered to the platform MBean server.

          22:46:06,618 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.

          22:46:06,618 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-6) ISPN000031: MBeans were successfully registered to the platform MBean server.

          22:46:06,628 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-12) JBAS010281: Started memcachedCache cache from clustered container

          22:46:06,628 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started namedCache cache from clustered container

          22:46:06,628 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) JBAS010281: Started default cache from clustered container

          22:46:06,628 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010000: MemcachedServer starting

          22:46:06,628 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010001: MemcachedServer listening on 127.0.0.1:11211

          22:46:07,052 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-5) ISPN000031: MBeans were successfully registered to the platform MBean server.

          22:46:07,056 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010281: Started ___hotRodTopologyCache cache from clustered containerReply to original post

          22:46:07,143 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management

          22:46:07,144 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990

          22:46:07,144 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) started in 7780ms - Started 93 of 142 services (49 services are passive or on-demand)

       

       

      As per the above logs, I don't think so they are forming a cluster of two nodes. Something is wrong for sure. Can anyone help me out here?

        • 1. Re: Not able to form a cluster of two nodes using Infinispan server
          Mircea Markus Master

          seems like the server binds to the loopback interface (127.0.0.1) instead of the LAN.

          Try adding -Djgroups.bind_addr=<IP of the network card used in LAN> into the script that launces the JVM.

          • 2. Re: Not able to form a cluster of two nodes using Infinispan server
            Raihan Jamal Newbie

            I just did ifconfig on my ubuntu machine and I got below result-

             

             

            root@rj-raptor-vm2-14939:/home/rjamal/shellapp/shellrapParent# ifconfig

            eth0      Link encap:Ethernet  HWaddr fa:16:3e:ec:c1:6b

                      inet addr:10.9.240.220  Bcast:10.9.240.255  Mask:255.255.255.0

                      inet6 addr: fe80::f816:3eff:feec:c16b/64 Scope:Link

                      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

                      RX packets:472064 errors:0 dropped:0 overruns:0 frame:0

                      TX packets:239982 errors:0 dropped:0 overruns:0 carrier:0

                      collisions:0 txqueuelen:1000

                      RX bytes:665447412 (665.4 MB)  TX bytes:30777275 (30.7 MB)

             

             

            lo        Link encap:Local Loopback

                      inet addr:127.0.0.1  Mask:255.0.0.0

                      inet6 addr: ::1/128 Scope:Host

                      UP LOOPBACK RUNNING  MTU:16436  Metric:1

                      RX packets:9112 errors:0 dropped:0 overruns:0 frame:0

                      TX packets:9112 errors:0 dropped:0 overruns:0 carrier:0

                      collisions:0 txqueuelen:0

                      RX bytes:1215875 (1.2 MB)  TX bytes:1215875 (1.2 MB)

             

             

            And you want me to use the bold letter IP Address right? If yes, then it should be something like this while launching-

             

                 sh clustered.sh -Djgroups.bind_addr=10.9.240.220

             

            Let me know if this is right? And I need to do the same thing for other nodes as well.. Right?

            • 4. Re: Not able to form a cluster of two nodes using Infinispan server
              Raihan Jamal Newbie

              I just did the same thing on both of my nodes. Below is the machine logs-

               

              Machine 1 console logs

               

              root@rj-raptor-vm1-14938:/home/rkost/infinispan/infinispan-server-6.0.0.Alpha1/bin# sh clustered.sh -Djgroups.bind_addr=10.9.251.184

              =========================================================================

               

               

                JBoss Bootstrap Environment

               

               

                JBOSS_HOME: /home/rkost/infinispan/infinispan-server-6.0.0.Alpha1

               

               

                JAVA: java

               

               

                JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

               

               

              =========================================================================

               

               

              19:29:12,163 INFO  [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1

              19:29:12,413 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA

              19:29:12,475 INFO  [org.jboss.as] (MSC service thread 1-7) JBAS015899: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) starting

              19:29:13,331 INFO  [org.xnio] (MSC service thread 1-2) XNIO Version 3.0.7.GA

              19:29:13,333 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)

              19:29:13,338 INFO  [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.0.7.GA

              19:29:13,346 INFO  [org.jboss.remoting] (MSC service thread 1-2) JBoss Remoting version 3.2.14.GA

              19:29:13,408 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.

              19:29:13,411 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.

              19:29:13,413 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem

              19:29:13,417 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem

              19:29:13,430 INFO  [org.jboss.as.security] (MSC service thread 1-14) JBAS013170: Current PicketBox version=4.0.15.Final

              19:29:13,445 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]

              19:29:13,458 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-6) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)

              19:29:13,465 INFO  [org.jboss.as.naming] (MSC service thread 1-13) JBAS011802: Starting Naming Service

              19:29:13,617 INFO  [org.apache.coyote.http11] (MSC service thread 1-8) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/127.0.0.1:8080

              19:29:13,624 INFO  [org.apache.coyote.ajp] (MSC service thread 1-16) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/127.0.0.1:8009

              19:29:13,627 INFO  [org.apache.coyote.http11] (MSC service thread 1-8) JBWEB003000: Coyote HTTP/1.1 starting on: http-/127.0.0.1:8080

              19:29:13,662 INFO  [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final

              19:29:13,676 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364

              19:29:13,844 INFO  [org.jboss.as.remoting] (MSC service thread 1-10) JBAS017100: Listening on 127.0.0.1:4447

              19:29:13,844 INFO  [org.jboss.as.remoting] (MSC service thread 1-3) JBAS017100: Listening on 127.0.0.1:9999

              19:29:13,996 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-12) JDGS010000: HotRodServer starting

              19:29:13,996 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: WebSocketServer starting

              19:29:13,997 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-12) JDGS010001: HotRodServer listening on 127.0.0.1:11222

              19:29:13,997 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010001: WebSocketServer listening on 127.0.0.1:8181

              19:29:13,998 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-7) ISPN000128: Infinispan version: Infinispan '<TBD>' 6.0.0.Alpha1

              19:29:14,118 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-9) ISPN000078: Starting JGroups Channel

              19:29:14,131 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-9) [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

              19:29:14,132 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-9) [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

              19:29:14,132 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-9) [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

              19:29:14,132 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-9) [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

              19:29:14,136 INFO  [stdout] (MSC service thread 1-9)

              19:29:14,136 INFO  [stdout] (MSC service thread 1-9) -------------------------------------------------------------------

              19:29:14,136 INFO  [stdout] (MSC service thread 1-9) GMS: address=rj-raptor-vm1-14938/clustered, cluster=clustered, physical address=10.9.251.184:55200

              19:29:14,136 INFO  [stdout] (MSC service thread 1-9) -------------------------------------------------------------------

              19:29:14,162 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-7) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:29:14,162 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-15) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:29:14,165 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-7) JBAS010281: Started jboss-web-policy cache from security container

              19:29:14,165 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-15) JBAS010281: Started other cache from security container

              19:29:14,176 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010000: REST starting

              19:29:14,367 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010002: REST mapped to /rest

              19:29:19,150 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-9) ISPN000094: Received new cluster view: [rj-raptor-vm1-14938/clustered|0] [rj-raptor-vm1-14938/clustered]

              19:29:19,252 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-9) ISPN000079: Cache local address is rj-raptor-vm1-14938/clustered, physical addresses are [10.9.251.184:55200]

              19:29:19,293 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-6) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:29:19,293 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-4) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:29:19,294 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-9) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:29:19,307 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010281: Started memcachedCache cache from clustered container

              19:29:19,307 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-6) JBAS010281: Started namedCache cache from clustered container

              19:29:19,307 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010281: Started default cache from clustered container

              19:29:19,307 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-9) JDGS010000: MemcachedServer starting

              19:29:19,308 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-9) JDGS010001: MemcachedServer listening on 127.0.0.1:11211

              19:29:19,755 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-12) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:29:19,759 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-12) JBAS010281: Started ___hotRodTopologyCache cache from clustered container

              19:29:19,859 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management

              19:29:19,860 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990

              19:29:19,860 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) started in 7944ms - Started 93 of 142 services (49 services are passive or on-demand)

               

               

              Machine 2 console

               

              root@rj-raptor-vm2-14939:/home/rkost/infinispan/infinispan-server-6.0.0.Alpha1/bin# sh clustered.sh -Djgroups.bind_addr=10.9.240.220

              =========================================================================

               

               

                JBoss Bootstrap Environment

               

               

                JBOSS_HOME: /home/rkost/infinispan/infinispan-server-6.0.0.Alpha1

               

               

                JAVA: java

               

               

                JAVA_OPTS:  -server -XX:+UseCompressedOops -XX:+TieredCompilation -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Djboss.server.default.config=clustered.xml -Dsun.nio.ch.bugLevel=''

               

               

              =========================================================================

               

               

              19:28:08,673 INFO  [org.jboss.modules] (main) JBoss Modules version 1.2.0.CR1

              19:28:08,932 INFO  [org.jboss.msc] (main) JBoss MSC version 1.0.4.GA

              19:28:08,992 INFO  [org.jboss.as] (MSC service thread 1-7) JBAS015899: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) starting

              19:28:09,938 INFO  [org.jboss.as.server] (Controller Boot Thread) JBAS015888: Creating http management service using socket-binding (management-http)

              19:28:09,938 INFO  [org.xnio] (MSC service thread 1-4) XNIO Version 3.0.7.GA

              19:28:09,949 INFO  [org.xnio.nio] (MSC service thread 1-4) XNIO NIO Implementation Version 3.0.7.GA

              19:28:09,958 INFO  [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 3.2.14.GA

              19:28:10,021 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 19) JBAS010280: Activating Infinispan subsystem.

              19:28:10,029 INFO  [org.jboss.as.clustering.jgroups] (ServerService Thread Pool -- 22) JBAS010260: Activating JGroups subsystem.

              19:28:10,033 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 25) JBAS011800: Activating Naming Subsystem

              19:28:10,041 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 27) JBAS013171: Activating Security Subsystem

              19:28:10,045 INFO  [org.jboss.as.security] (MSC service thread 1-2) JBAS013170: Current PicketBox version=4.0.15.Final

              19:28:10,051 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 31) JBAS012605: Activated the following JSF Implementations: [main, 1.2]

              19:28:10,060 INFO  [org.jboss.as.connector.logging] (MSC service thread 1-1) JBAS010408: Starting JCA Subsystem (JBoss IronJacamar 1.0.15.Final)

              19:28:10,079 INFO  [org.jboss.as.naming] (MSC service thread 1-10) JBAS011802: Starting Naming Service

              19:28:10,208 INFO  [org.apache.coyote.http11] (MSC service thread 1-1) JBWEB003001: Coyote HTTP/1.1 initializing on : http-/127.0.0.1:8080

              19:28:10,217 INFO  [org.apache.coyote.http11] (MSC service thread 1-1) JBWEB003000: Coyote HTTP/1.1 starting on: http-/127.0.0.1:8080

              19:28:10,221 INFO  [org.apache.coyote.ajp] (MSC service thread 1-7) JBWEB003046: Starting Coyote AJP/1.3 on ajp-/127.0.0.1:8009

              19:28:10,244 INFO  [org.jboss.modcluster.ModClusterService] (ServerService Thread Pool -- 32) Initializing mod_cluster 1.2.3.Final

              19:28:10,265 INFO  [org.jboss.modcluster.advertise.impl.AdvertiseListenerImpl] (ServerService Thread Pool -- 32) Listening to proxy advertisements on 224.0.1.115:23,364

              19:28:10,414 INFO  [org.jboss.as.remoting] (MSC service thread 1-15) JBAS017100: Listening on 127.0.0.1:9999

              19:28:10,414 INFO  [org.jboss.as.remoting] (MSC service thread 1-11) JBAS017100: Listening on 127.0.0.1:4447

              19:28:10,569 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-8) JDGS010000: WebSocketServer starting

              19:28:10,569 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010000: HotRodServer starting

              19:28:10,570 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-8) JDGS010001: WebSocketServer listening on 127.0.0.1:8181

              19:28:10,570 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-3) JDGS010001: HotRodServer listening on 127.0.0.1:11222

              19:28:10,622 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-4) ISPN000128: Infinispan version: Infinispan '<TBD>' 6.0.0.Alpha1

              19:28:10,746 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-11) ISPN000078: Starting JGroups Channel

              19:28:10,785 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-11) [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

              19:28:10,785 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-11) [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

              19:28:10,785 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-11) [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

              19:28:10,786 WARN  [org.jgroups.protocols.UDP] (MSC service thread 1-11) [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

              19:28:10,790 INFO  [stdout] (MSC service thread 1-11)

              19:28:10,790 INFO  [stdout] (MSC service thread 1-11) -------------------------------------------------------------------

              19:28:10,790 INFO  [stdout] (MSC service thread 1-11) GMS: address=rj-raptor-vm2-14939/clustered, cluster=clustered, physical address=10.9.240.220:55200

              19:28:10,790 INFO  [stdout] (MSC service thread 1-11) -------------------------------------------------------------------

              19:28:10,815 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-4) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:28:10,815 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-9) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:28:10,819 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-4) JBAS010281: Started other cache from security container

              19:28:10,819 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-9) JBAS010281: Started jboss-web-policy cache from security container

              19:28:10,835 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010000: REST starting

              19:28:11,020 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010002: REST mapped to /rest

              19:28:15,807 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-11) ISPN000094: Received new cluster view: [rj-raptor-vm2-14939/clustered|0] [rj-raptor-vm2-14939/clustered]

              19:28:15,897 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-11) ISPN000079: Cache local address is rj-raptor-vm2-14939/clustered, physical addresses are [10.9.240.220:55200]

              19:28:15,934 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-11) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:28:15,934 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-2) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:28:15,934 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-10) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:28:15,944 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-2) JBAS010281: Started memcachedCache cache from clustered container

              19:28:15,944 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-10) JBAS010281: Started namedCache cache from clustered container

              19:28:15,944 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-11) JBAS010281: Started default cache from clustered container

              19:28:15,945 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010000: MemcachedServer starting

              19:28:15,945 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-16) JDGS010001: MemcachedServer listening on 127.0.0.1:11211

              19:28:16,603 INFO  [org.infinispan.jmx.CacheJmxRegistration] (MSC service thread 1-3) ISPN000031: MBeans were successfully registered to the platform MBean server.

              19:28:16,608 INFO  [org.jboss.as.clustering.infinispan] (MSC service thread 1-3) JBAS010281: Started ___hotRodTopologyCache cache from clustered container

              19:28:16,703 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:9990/management

              19:28:16,703 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:9990

              19:28:16,704 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 6.0.0.Alpha1 (AS 7.2.0.Final) started in 8283ms - Started 93 of 142 services (49 services are passive or on-demand)

               

               

              And I don't think so they have formed a cluster of two nodes. Is there any other things missing? Might be multicast issue? How do I make sure multicast is enabled properly and any other settings are also configured properly?

               

              Thanks for the help on this.

              • 6. Re: Not able to form a cluster of two nodes using Infinispan server
                Raihan Jamal Newbie

                This is the ifconfig for the other node-

                 

                root@rj-raptor-vm1-14938:/home/rjamal/infinispan/infinispan-server-6.0.0.Alpha1/bin# ifconfig

                eth0      Link encap:Ethernet  HWaddr fa:16:3e:92:59:12

                          inet addr:10.9.251.184  Bcast:10.9.251.255  Mask:255.255.255.0

                          inet6 addr: fe80::f816:3eff:fe92:5912/64 Scope:Link

                          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

                          RX packets:604506 errors:0 dropped:0 overruns:0 frame:0

                          TX packets:376884 errors:0 dropped:0 overruns:0 carrier:0

                          collisions:0 txqueuelen:1000

                          RX bytes:838973473 (838.9 MB)  TX bytes:67117844 (67.1 MB)

                 

                 

                lo        Link encap:Local Loopback

                          inet addr:127.0.0.1  Mask:255.0.0.0

                          inet6 addr: ::1/128 Scope:Host

                          UP LOOPBACK RUNNING  MTU:16436  Metric:1

                          RX packets:34128 errors:0 dropped:0 overruns:0 frame:0

                          TX packets:34128 errors:0 dropped:0 overruns:0 carrier:0

                          collisions:0 txqueuelen:0

                          RX bytes:5990228 (5.9 MB)  TX bytes:5990228 (5.9 MB)

                 

                One quick thing how the other node will figure out that any other nodes is joining. Bcoz I am not changing any config file which will tell about the other nodes.

                • 7. Re: Not able to form a cluster of two nodes using Infinispan server
                  Tristan Tarrant Master

                   

                  One quick thing how the other node will figure out that any other nodes is joining. Bcoz I am not changing any config file which will tell about the other nodes.

                   

                  It can be done in a variety of ways. By default using UDP multicast to broadcast an "I want to join" message to an existing cluster. This generally works for all nodes on the same subnet, which is the problem in your configuration:

                   

                  the two nodes belong to different subnets:

                  Node 1 inet addr:10.9.240.220  Bcast:10.9.240.255  Mask:255.255.255.0

                  Node 2 inet addr:10.9.251.184  Bcast:10.9.251.255  Mask:255.255.255.0

                   

                  Node 1 belongs to a network 10.9.240.0/24

                  Node 2 belongs to a network 10.9.251.0/24

                   

                  Make them belong to the same network and you're set.

                   

                  Tristan