12 Replies Latest reply on Sep 19, 2013 10:49 AM by dan.berindei

    Infinispan cluster with Invalidation

    jrox

      I'm currently trying to run an infinispan cluster (5.1.4.FINAL) with two machines using invalidation. Right now I'm trying to run two processes on the same machine that should send invalidation messages to each other but I cant get them to see each other and properly send messages through.

       

      As soon as the second process starts up they start sending messages but both of them drop messages with the error that the sender is not in table. I tried the McastSenderTest/McastReceiverTest which seems to work. When having a McastReceiverTest running I seem to get the updates from the infinispan nodes as well although it looks a bit strange. Anyone know what I'm doing wrong here?

       

      Node 1:

      10:07:18,148 INFO  [JGroupsTransport] ISPN000078: Starting JGroups Channel

      10:07:18,805 WARN  [UDP] send buffer of socket java.net.DatagramSocket@359ba2ce was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      10:07:18,805 WARN  [UDP] receive buffer of socket java.net.DatagramSocket@359ba2ce was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      10:07:18,805 WARN  [UDP] send buffer of socket java.net.MulticastSocket@7cdd9de0 was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      10:07:18,805 WARN  [UDP] receive buffer of socket java.net.MulticastSocket@7cdd9de0 was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      10:07:21,842 INFO  [JGroupsTransport] ISPN000094: Received new cluster view: [pbapp6-59135|0] [pbapp6-59135]

      10:07:21,843 INFO  [JGroupsTransport] ISPN000079: Cache local address is pbapp6-59135, physical addresses are [fe80:0:0:0:216:3eff:fe5a:3f76%3:32830]

      10:07:21,853 INFO  [GlobalComponentRegistry] ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.4.FINAL

      10:18:38,465 WARN  [NAKACK] pbapp6-59135: dropped message 1 from pbapp6-49654 (sender not in table [pbapp6-59135]), view=[pbapp6-59135|0] [pbapp6-59135]

      10:18:38,847 WARN  [NAKACK] pbapp6-59135: dropped message 2 from pbapp6-49654 (sender not in table [pbapp6-59135]), view=[pbapp6-59135|0] [pbapp6-59135]

      10:18:39,475 WARN  [NAKACK] pbapp6-59135: dropped message 3 from pbapp6-49654 (sender not in table [pbapp6-59135]), view=[pbapp6-59135|0] [pbapp6-59135]

      10:18:39,648 WARN  [NAKACK] pbapp6-59135: dropped message 4 from pbapp6-49654 (sender not in table [pbapp6-59135]), view=[pbapp6-59135|0] [pbapp6-59135]

      10:18:41,567 WARN  [NAKACK] pbapp6-59135: dropped message 5 from pbapp6-49654 (sender not in table [pbapp6-59135]), view=[pbapp6-59135|0] [pbapp6-59135]

      ....

       

       

      Node 2:

      10:18:27,831 INFO  [JGroupsTransport] ISPN000078: Starting JGroups Channel

      10:18:28,461 WARN  [UDP] send buffer of socket java.net.DatagramSocket@7cdd9de0 was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      10:18:28,461 WARN  [UDP] receive buffer of socket java.net.DatagramSocket@7cdd9de0 was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      10:18:28,462 WARN  [UDP] send buffer of socket java.net.MulticastSocket@4c130f9f was set to 640KB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)

      10:18:28,462 WARN  [UDP] receive buffer of socket java.net.MulticastSocket@4c130f9f was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)

      10:18:31,490 INFO  [JGroupsTransport] ISPN000094: Received new cluster view: [pbapp6-49654|0] [pbapp6-49654]

      10:18:31,492 INFO  [JGroupsTransport] ISPN000079: Cache local address is pbapp6-49654, physical addresses are [fe80:0:0:0:216:3eff:fe5a:3f76%3:32832]

      10:18:31,501 INFO  [GlobalComponentRegistry] ISPN000128: Infinispan version: Infinispan 'Brahma' 5.1.4.FINAL

      10:18:36,984 WARN  [NAKACK] pbapp6-49654: dropped message 245 from 724282bc-9cac-33a8-511a-d94c444222f4 (sender not in table [pbapp6-49654]), view=[pbapp6-49654|0] [pbapp6-49654]

      10:18:37,258 WARN  [NAKACK] pbapp6-49654: dropped message 246 from 724282bc-9cac-33a8-511a-d94c444222f4 (sender not in table [pbapp6-49654]), view=[pbapp6-49654|0] [pbapp6-49654]

      10:18:41,934 WARN  [UDP] pbapp6-49654: no physical address for 724282bc-9cac-33a8-511a-d94c444222f4, dropping message

      10:18:44,906 WARN  [UDP] pbapp6-49654: no physical address for 724282bc-9cac-33a8-511a-d94c444222f4, dropping message

      10:18:46,491 WARN  [NAKACK] pbapp6-49654: dropped message 247 from 724282bc-9cac-33a8-511a-d94c444222f4 (sender not in table [pbapp6-49654]), view=[pbapp6-49654|0] [pbapp6-49654]

      10:18:46,827 WARN  [NAKACK] pbapp6-49654: dropped message 248 from 724282bc-9cac-33a8-511a-d94c444222f4 (sender not in table [pbapp6-49654]), view=[pbapp6-49654|0] [pbapp6-49654]

      10:18:49,431 WARN  [NAKACK] pbapp6-49654: dropped message 249 from 724282bc-9cac-33a8-511a-d94c444222f4 (sender not in table [pbapp6-49654]), view=[pbapp6-49654|0] [pbapp6-49654]

      10:18:49,903 WARN  [NAKACK] pbapp6-49654: dropped message 250 from 724282bc-9cac-33a8-511a-d94c444222f4 (sender not in table [pbapp6-49654]), view=[pbapp6-49654|0] [pbapp6-49654]

      ....

       

       

      McastSenderTest:

      -bash-3.1$ java -cp jgroups-3.0.9.Final.jar org.jgroups.tests.McastSenderTest -m                     cast_addr 228.6.7.8 -port 46655

      Socket #1=0.0.0.0/0.0.0.0:46655, ttl=32, bind interface=/fe80:0:0:0:216:3eff:fe5                     a:3f76%3

      Socket #2=0.0.0.0/0.0.0.0:46655, ttl=32, bind interface=/192.168.6.102

      Socket #3=0.0.0.0/0.0.0.0:46655, ttl=32, bind interface=/0:0:0:0:0:0:0:1%1

      Socket #4=0.0.0.0/0.0.0.0:46655, ttl=32, bind interface=/127.0.0.1

      > << Received packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:        çd'°µÑö6                     ÿòçd'°µÑö6ÿò§ª8ISPN

      << Received packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:  çd'°µÑö6ÿòçd'°µÑ                     ö6ÿò§ª8ISPN

      << Received packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:  çd'°µÑö6ÿòçd'°µÑ                     ö6ÿò§ª8ISPN

      << Received packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:  çd'°µÑö6ÿòçd'°µÑ                     ö6ÿò§ª8ISPN

      <8ISPNeived packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:  ¬!Àµ{   V !Àµ{ V                     <8ISPNeived packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:  ¬!Àµ{   V !Àµ{ V                     <8ISPNeived packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:  ¬!Àµ{   V !Àµ{ V                     <8ISPNeived packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:  ¬!Àµ{   V !Àµ{ V                     << Received packet from fe80:0:0:0:216:3eff:fe5a:3f76%3:46655:  ¬!Àµ{   V !Àµ{ V                      8ISPN

        • 1. Re: Infinispan cluster with Invalidation
          dudalov

          That sounds like JGroup configuration problem. Nodes didn't find each other. Node 1 knows only about itself:

                 Received new cluster view: [pbapp6-59135|0] [pbapp6-59135]

          So the Node 2:

                 Received new cluster view: [pbapp6-49654|0] [pbapp6-49654]

           

          It's supposed to be like:  [pbapp6-59135|0] [pbapp6-59135, pbapp6-49654]

           

          Is it related to misconfigured UDP send buffer? I'm not sure.

          • 2. Re: Infinispan cluster with Invalidation
            dudalov

            I actually have another question about Invalidation mode. Do I get it right that after cached object is updated on one of the cluster nodes then the other nodes should eventually retrieve the updated data from the store? Can the object be replicated from the node where it was modified? I'm trying to avoid expensive DB queries for complex objects and reuse updated cache object on other nodes when it's requested. Something similar to replicated mode, but only "on demand". Can it be achieved?

             

            And one more. Are the objects on other nodes invalidated automatically or I should use @CacheEntryInvalidated listener to do it from the client?

            Any links to best practices for invalidated cache?

             

            Thanks,

            Dmitry

            • 3. Re: Infinispan cluster with Invalidation
              jrox

              I'm using the out of the box jgroups-udp.xml configuration file. I'm not sure what you mean by send buffer being wrong. Anyone else got an idea why they are not able to find each other but are able to send messages to each other (resulting in a drop though).

              • 4. Re: Infinispan cluster with Invalidation
                galder.zamarreno

                Try jgroups-tcp.xml

                • 5. Re: Infinispan cluster with Invalidation
                  mgencur

                  I was getting a similar error lately and the following snippet being added to jgroups stack helped:

                  <protocol type="PING">
                      <property name="timeout">5000</property>
                      <property name="num_initial_members">5</property>
                  </protocol>
                  
                  • 6. Re: Infinispan cluster with Invalidation
                    mgencur

                    Btw, you can achieve lazy loading in replicated mode by disabling state-transfer and using Cluster cache loader. Invalidation mode needs a cache store, indeed.

                    • 7. Re: Infinispan cluster with Invalidation
                      dudalov

                      Hi Martin,

                       

                      Where do you suggest to add that PING element, not to jgroups configuration file, doesn't it? In jgroups xml-file there is a different element already:

                      <PING timeout="5000" num_initial_members="5"/>

                      • 8. Re: Infinispan cluster with Invalidation
                        mgencur

                        Ah, I meant this one, sorry:) What I actually used was a config for JBoss AS 7. Do you really use PING with timeout 5000? I'm seeing a config file at https://github.com/infinispan/infinispan/blob/9ba7cf53c164b07c6111357f2b21913ea0dff982/core/src/main/resources/jgroups-udp.xml#L62 and it has only 3000. So you've already changed this?

                        • 9. Re: Infinispan cluster with Invalidation
                          dan.berindei

                          @Andre, I have a hunch you'd solve your problem if you forced IPv4 with -Djava.net.preferIPv4Stack=true.

                           

                          If it still doesn't work, please attach your JGroups configuration.

                          • 10. Re: Infinispan cluster with Invalidation
                            jrox

                            Thanks Dan, that seems to have fixed that issue.

                            • 11. Re: Infinispan cluster with Invalidation
                              vbalaut

                              I am also facing the same issue which is mentioned in this post..

                               

                              I am having application which is internally creating REPL cache using EmbeddedCache manager. I am using jgroups-udp with default setting...

                              When I am trying to run on windows, its working fine and cluster is able to form.

                               

                              But when I am trying the same code on linux machine, cluster is not forming.

                              I have tried this setting -Djava.net.preferIPv4Stack=true but seems to be no luck. I have verified that there is no firewall which is blocking....


                              Can you suggesst some way to find out exact issue and resolve that...

                               

                              tomcat 1 log
                              ----------------------------------------------------------------------
                              2013-09-12 07:51:24,480|main|INFO |com.enlighta.cluster.cache.AbstractCacheManager.<init>(AbstractCacheManager.java:61) - Trying to create infinispan cache programatically.
                              2013-09-12 07:51:24,885|main|DEBUG|org.infinispan.util.ModuleProperties.resolveModuleLifecycles(ModuleProperties.java:50) - Loading lifecycle SPI class: org.infinispan.query.impl.LifecycleManager@3352d4
                              2013-09-12 07:51:24,994|main|DEBUG|org.infinispan.util.ModuleProperties.loadModuleCommandHandlers(ModuleProperties.java:79) - Loading module command extension SPI class: org.infinispan.query.QueryModuleCommandExtensions@6e8deccd
                              2013-09-12 07:51:25,026|main|DEBUG|org.infinispan.marshall.core.JBossMarshaller.inject(JBossMarshaller.java:50) - Using JBoss Marshalling
                              2013-09-12 07:51:25,150|main|DEBUG|org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:688) - Started cache manager ISPN on null
                              2013-09-12 07:51:25,166|main|INFO |com.enlighta.cluster.cache.ClusterCacheService.<init>(ClusterCacheService.java:38) - Trying to create MAP of caches so it can be used by components..
                              2013-09-12 07:51:25,369|main|INFO |org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:182) - ISPN000078: Starting JGroups Channel
                              2013-09-12 07:51:25,852|main|WARN |org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1134) - [JGRP00014] TP.discard_incompatible_packets has been deprecated: incompatible packets are discarded anyway
                              2013-09-12 07:51:25,852|main|WARN |org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1134) - [JGRP00014] TP.enable_bundling has been deprecated: will be ignored as bundling is on by default
                              2013-09-12 07:51:25,899|main|WARN |org.jgroups.stack.Configurator.resolveAndInvokePropertyMethod(Configurator.java:1087) - [JGRP00014] UNICAST.setTimeout has been deprecated: not used anymore
                              2013-09-12 07:51:26,086|main|DEBUG|org.jgroups.stack.Configurator.setDefaultValues(Configurator.java:823) - set property UDP.bind_addr to default value /fe80:0:0:0:d572:3236:54a6:1ff0%13
                              2013-09-12 07:51:26,086|main|DEBUG|org.jgroups.stack.Configurator.setDefaultValues(Configurator.java:823) - set property UDP.diagnostics_addr to default value /ff0e:0:0:0:0:0:75:75
                              2013-09-12 07:51:26,227|main|DEBUG|org.jgroups.protocols.FRAG2.up(FRAG2.java:173) - received CONFIG event: {bind_addr=/fe80:0:0:0:d572:3236:54a6:1ff0%13}
                              2013-09-12 07:51:26,258|main|DEBUG|org.jgroups.protocols.UDP.createSockets(UDP.java:323) - sockets will use interface fe80:0:0:0:d572:3236:54a6:1ff0%13
                              2013-09-12 07:51:26,508|main|DEBUG|org.jgroups.protocols.UDP.createSockets(UDP.java:393) - socket information:
                              mcast_addr=ff0e:0:0:0:0:0:e406:708:46655, bind_addr=/fe80:0:0:0:d572:3236:54a6:1ff0%13, ttl=2
                              sock: bound to fe80:0:0:0:d572:3236:54a6:1ff0%13:62204, receive buffer size=20000000, send buffer size=640000
                              mcast_sock: bound to fe80:0:0:0:d572:3236:54a6:1ff0%13:46655, send buffer size=640000, receive buffer size=25000000
                              2013-09-12 07:51:26,523|main|DEBUG|org.jgroups.protocols.pbcast.GMS.down(GMS.java:924) - address=taru_laptop-45266, cluster=ISPN, physical address=fe80:0:0:0:d572:3236:54a6:1ff0%13:62204
                              2013-09-12 07:51:56,584|main|DEBUG|org.jgroups.protocols.pbcast.NAKACK.setDigest(NAKACK.java:1216) -
                              [taru_laptop-45266 setDigest()]
                              existing digest:  []
                              new digest:       taru_laptop-45266: [0 (0)]
                              resulting digest: taru_laptop-45266: [0 (0)]
                              2013-09-12 07:51:56,584|main|DEBUG|org.jgroups.protocols.pbcast.GMS.installView(GMS.java:595) - taru_laptop-45266: installing [taru_laptop-45266|0] [taru_laptop-45266]
                              2013-09-12 07:51:56,584|main|DEBUG|org.jgroups.protocols.pbcast.STABLE.resume(STABLE.java:206) - resuming message garbage collection
                              2013-09-12 07:51:56,584|main|DEBUG|org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:328) - VIEW_CHANGE received: [taru_laptop-45266]
                              2013-09-12 07:51:56,584|main|DEBUG|org.infinispan.remoting.transport.jgroups.JGroupsTransport.viewAccepted(JGroupsTransport.java:617) - New view accepted: [taru_laptop-45266|0] [taru_laptop-45266]
                              2013-09-12 07:51:56,584|main|INFO |org.infinispan.remoting.transport.jgroups.JGroupsTransport.viewAccepted(JGroupsTransport.java:644) - ISPN000094: Received new cluster view: [taru_laptop-45266|0] [taru_laptop-45266]
                              2013-09-12 07:51:56,600|main|INFO |com.enlighta.cluster.JMSClientListener.observeViewChanged(JMSClientListener.java:34) - View Changed : EventImpl{type=VIEW_CHANGED, newMembers=[taru_laptop-45266], oldMembers=null, localAddress=taru_laptop-45266, viewId=0, subgroupsMerged=null, mergeView=false}
                              2013-09-12 07:51:56,600|main|DEBUG|org.jgroups.protocols.pbcast.STABLE.resume(STABLE.java:206) - resuming message garbage collection
                              2013-09-12 07:51:56,600|main|DEBUG|org.jgroups.protocols.pbcast.ClientGmsImpl.becomeSingletonMember(ClientGmsImpl.java:336) - created group (first member). My view is [taru_laptop-45266|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl

                              *** After tomcat 2 comes up.. here is view change message

                               

                              2013-09-12 07:52:34,488|ViewHandler,taru_laptop-45266|DEBUG|org.jgroups.protocols.pbcast.STABLE.suspend(STABLE.java:190) - suspending message garbage collection
                              2013-09-12 07:52:34,488|ViewHandler,taru_laptop-45266|DEBUG|org.jgroups.protocols.pbcast.STABLE.startResumeTask(STABLE.java:519) - resume task started, max_suspend_time=33000
                              2013-09-12 07:52:34,488|Incoming-1,taru_laptop-45266|DEBUG|org.jgroups.protocols.pbcast.NAKACK.setDigest(NAKACK.java:1216) -
                              [taru_laptop-45266 setDigest()]
                              existing digest:  taru_laptop-45266: [1 (1)]
                              new digest:       taru_laptop-45266: [0 (0)], taru_laptop-12830: [0 (0)]
                              resulting digest: taru_laptop-12830: [0 (0)], taru_laptop-45266: [1 (1)]
                              2013-09-12 07:52:34,488|Incoming-1,taru_laptop-45266|DEBUG|org.jgroups.protocols.pbcast.GMS.installView(GMS.java:595) - taru_laptop-45266: installing [taru_laptop-45266|1] [taru_laptop-45266, taru_laptop-12830]
                              2013-09-12 07:52:34,503|Incoming-1,taru_laptop-45266|DEBUG|org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:328) - VIEW_CHANGE received: [taru_laptop-45266, taru_laptop-12830]
                              2013-09-12 07:52:34,503|FD_SOCK pinger,taru_laptop-45266|DEBUG|org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:386) - ping_dest is taru_laptop-12830, pingable_mbrs=[taru_laptop-45266, taru_laptop-12830]

                               

                               

                              tomcat 2 log
                              ------------------------------------------------------------
                              2013-09-12 07:52:33,162|main|INFO |org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:182) - ISPN000078: Starting JGroups Channel
                              2013-09-12 07:52:33,614|main|WARN |org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1134) - [JGRP00014] TP.discard_incompatible_packets has been deprecated: incompatible packets are discarded anyway
                              2013-09-12 07:52:33,614|main|WARN |org.jgroups.stack.Configurator.resolveAndAssignField(Configurator.java:1134) - [JGRP00014] TP.enable_bundling has been deprecated: will be ignored as bundling is on by default
                              2013-09-12 07:52:33,661|main|WARN |org.jgroups.stack.Configurator.resolveAndInvokePropertyMethod(Configurator.java:1087) - [JGRP00014] UNICAST.setTimeout has been deprecated: not used anymore
                              2013-09-12 07:52:33,864|main|DEBUG|org.jgroups.stack.Configurator.setDefaultValues(Configurator.java:823) - set property UDP.bind_addr to default value /fe80:0:0:0:d572:3236:54a6:1ff0%13
                              2013-09-12 07:52:33,864|main|DEBUG|org.jgroups.stack.Configurator.setDefaultValues(Configurator.java:823) - set property UDP.diagnostics_addr to default value /ff0e:0:0:0:0:0:75:75
                              2013-09-12 07:52:33,989|main|DEBUG|org.jgroups.protocols.FRAG2.up(FRAG2.java:173) - received CONFIG event: {bind_addr=/fe80:0:0:0:d572:3236:54a6:1ff0%13}
                              2013-09-12 07:52:34,020|main|DEBUG|org.jgroups.protocols.UDP.createSockets(UDP.java:323) - sockets will use interface fe80:0:0:0:d572:3236:54a6:1ff0%13
                              2013-09-12 07:52:34,332|main|DEBUG|org.jgroups.protocols.UDP.createSockets(UDP.java:393) - socket information:
                              mcast_addr=ff0e:0:0:0:0:0:e406:708:46655, bind_addr=/fe80:0:0:0:d572:3236:54a6:1ff0%13, ttl=2
                              sock: bound to fe80:0:0:0:d572:3236:54a6:1ff0%13:61344, receive buffer size=20000000, send buffer size=640000
                              mcast_sock: bound to fe80:0:0:0:d572:3236:54a6:1ff0%13:46655, send buffer size=640000, receive buffer size=25000000
                              2013-09-12 07:52:34,347|main|DEBUG|org.jgroups.protocols.pbcast.GMS.down(GMS.java:924) - address=taru_laptop-12830, cluster=ISPN, physical address=fe80:0:0:0:d572:3236:54a6:1ff0%13:61344
                              2013-09-12 07:52:34,410|main|DEBUG|org.jgroups.protocols.pbcast.ClientGmsImpl.determineCoord(ClientGmsImpl.java:298) - election results: {taru_laptop-45266=1}
                              2013-09-12 07:52:34,410|main|DEBUG|org.jgroups.protocols.pbcast.ClientGmsImpl.joinInternal(ClientGmsImpl.java:133) - sending JOIN(taru_laptop-12830) to taru_laptop-45266
                              2013-09-12 07:52:34,987|main|DEBUG|org.jgroups.protocols.pbcast.NAKACK.setDigest(NAKACK.java:1216) -
                              [taru_laptop-12830 setDigest()]
                              existing digest:  []
                              new digest:       taru_laptop-45266: [0 (0)], taru_laptop-12830: [0 (0)]
                              resulting digest: taru_laptop-12830: [0 (0)], taru_laptop-45266: [0 (0)]
                              2013-09-12 07:52:34,987|main|DEBUG|org.jgroups.protocols.pbcast.GMS.installView(GMS.java:595) - taru_laptop-12830: installing [taru_laptop-45266|1] [taru_laptop-45266, taru_laptop-12830]
                              2013-09-12 07:52:35,003|main|DEBUG|org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:328) - VIEW_CHANGE received: [taru_laptop-45266, taru_laptop-12830]
                              2013-09-12 07:52:35,003|FD_SOCK pinger,taru_laptop-12830|DEBUG|org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:386) - ping_dest is taru_laptop-45266, pingable_mbrs=[taru_laptop-45266, taru_laptop-12830]
                              2013-09-12 07:52:35,003|main|DEBUG|org.infinispan.remoting.transport.jgroups.JGroupsTransport.viewAccepted(JGroupsTransport.java:617) - New view accepted: [taru_laptop-45266|1] [taru_laptop-45266, taru_laptop-12830]
                              2013-09-12 07:52:35,003|main|INFO |org.infinispan.remoting.transport.jgroups.JGroupsTransport.viewAccepted(JGroupsTransport.java:644) - ISPN000094: Received new cluster view: [taru_laptop-45266|1] [taru_laptop-45266, taru_laptop-12830]
                              2013-09-12 07:52:35,018|main|INFO |com.enlighta.cluster.JMSClientListener.observeViewChanged(JMSClientListener.java:34) - View Changed : EventImpl{type=VIEW_CHANGED, newMembers=[taru_laptop-45266, taru_laptop-12830], oldMembers=null, localAddress=taru_laptop-12830, viewId=1, subgroupsMerged=null, mergeView=false}
                              2013-09-12 07:52:35,018|main|INFO |org.infinispan.remoting.transport.jgroups.JGroupsTransport.startJGroupsChannelIfNeeded(JGroupsTransport.java:220) - ISPN000079: Cache local address is taru_laptop-12830, physical addresses are [fe80:0:0:0:d572:3236:54a6:1ff0%13:61344]
                              2013-09-12

                               

                               

                               

                               

                              Here is the logs where tomcat nodes start on linux server and they are not able to join cluster.

                               

                              tomcat 1 logs
                              -----------------------------------------------------------------------------------

                                  2013-09-12 07:39:31,785|main|INFO  - ISPN000078: Starting JGroups Channel
                                  2013-09-12 07:39:31,903|main|WARN  - [JGRP00014] TP.discard_incompatible_packets has been deprecated: incompatible packets are discarded anyway
                                   2013-09-12 07:39:31,903|main|WARN  - [JGRP00014] TP.enable_bundling has been deprecated: will be ignored as bundling is on by default
                                  2013-09-12 07:39:31,925|main|WARN  - [JGRP00014] UNICAST.setTimeout has been deprecated: not used anymore
                                   2013-09-12 07:39:31,971|main|DEBUG - set property UDP.bind_addr to default value /192.168.0.5
                                  2013-09-12 07:39:31,971|main|DEBUG - set property UDP.diagnostics_addr to default value /224.0.75.75
                                   2013-09-12 07:39:31,997|main|DEBUG - received CONFIG event: {bind_addr=/192.168.0.5}
                                  2013-09-12 07:39:32,007|main|DEBUG - sockets will use interface 192.168.0.5
                                  2013-09-12 07:39:32,008|main|WARN  - [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
                                  2013-09-12 07:39:32,008|main|WARN  - [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to per
                                   formance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
                                  2013-09-12 07:39:32,009|main|WARN  - [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to perf
                                   ormance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
                                  2013-09-12 07:39:32,009|main|WARN  - [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to pe
                                   rformance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
                                  2013-09-12 07:39:32,010|main|DEBUG - socket information:
                                  mcast_addr=228.6.7.8:46655, bind_addr=/192.168.0.5, ttl=2
                                   sock: bound to 192.168.0.5:33562, receive buffer size=212992, send buffer size=212992
                                  mcast_sock: bound to 192.168.0.5:46655, send buffer size=212992, receive buffer size=212992
                                   2013-09-12 07:39:32,012|main|DEBUG - address=localhost-45145, cluster=ISPN, physical address=192.168.0.5:33562
                                  2013-09-12 07:40:02,030|main|DEBUG -
                                  [localhost-45145 setDigest()]
                                   existing digest:  []
                                  new digest:       localhost-45145: [0 (0)]
                                  resulting digest: localhost-45145: [0 (0)]
                                  2013-09-12 07:40:02,031|main|DEBUG - localhost-45145: installing [localhost-45145|0] [localhost-45145]
                                   2013-09-12 07:40:02,033|main|DEBUG - resuming message garbage collection
                                  2013-09-12 07:40:02,034|main|DEBUG - VIEW_CHANGE received: [localhost-45145]
                                  2013-09-12 07:40:02,035|main|DEBUG - New view accepted: [localhost-45145|0] [localhost-45145]
                                   2013-09-12 07:40:02,036|main|INFO  - ISPN000094: Received new cluster view: [localhost-45145|0] [localhost-45145]
                                  2013-09-12 07:40:02,038|main|INFO  - View Changed : EventImpl{type=VIEW_CHANGED, newMembers=[localhost-45145], oldMembers=null, localAddress=localhost-45145, viewId=0,
                                   subgroupsMerged=null, mergeView=false}
                                  2013-09-12 07:40:02,038|main|DEBUG - resuming message garbage collection
                                  2013-09-12 07:40:02,039|main|DEBUG - created group (first member). My view is [localhost-45145|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl
                                   2013-09-12 07:40:02,039|main|INFO  - ISPN000079: Cache local address is localhost-45145, physical addresses are [192.168.0.5:33562]

                               

                               

                               

                              tomcat 2 log

                               

                              ----------------------------------------------------------------------------------------------------


                              2013-09-12 07:38:39,740|main|INFO  - Trying to create MAP of caches so it can be used by components..
                              2013-09-12 07:38:39,816|main|INFO  - ISPN000078: Starting JGroups Channel
                              2013-09-12 07:38:39,931|main|WARN  - [JGRP00014] TP.discard_incompatible_packets has been deprecated: incompatible packets are discarded anyway
                              2013-09-12 07:38:39,932|main|WARN  - [JGRP00014] TP.enable_bundling has been deprecated: will be ignored as bundling is on by default
                              2013-09-12 07:38:39,953|main|WARN  - [JGRP00014] UNICAST.setTimeout has been deprecated: not used anymore
                              2013-09-12 07:38:39,998|main|DEBUG - set property UDP.bind_addr to default value /192.168.0.5
                              2013-09-12 07:38:39,998|main|DEBUG - set property UDP.diagnostics_addr to default value /224.0.75.75
                              2013-09-12 07:38:40,025|main|DEBUG - received CONFIG event: {bind_addr=/192.168.0.5}
                              2013-09-12 07:38:40,034|main|DEBUG - sockets will use interface 192.168.0.5
                              2013-09-12 07:38:40,036|main|WARN  - [JGRP00015] the send buffer of socket DatagramSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to perfo
                              rmance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
                              2013-09-12 07:38:40,036|main|WARN  - [JGRP00015] the receive buffer of socket DatagramSocket was set to 20MB, but the OS only allocated 212.99KB. This might lead to per
                              formance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
                              2013-09-12 07:38:40,037|main|WARN  - [JGRP00015] the send buffer of socket MulticastSocket was set to 640KB, but the OS only allocated 212.99KB. This might lead to perf
                              ormance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux)
                              2013-09-12 07:38:40,037|main|WARN  - [JGRP00015] the receive buffer of socket MulticastSocket was set to 25MB, but the OS only allocated 212.99KB. This might lead to pe
                              rformance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
                              2013-09-12 07:38:40,038|main|DEBUG - socket information:
                              mcast_addr=228.6.7.8:46655, bind_addr=/192.168.0.5, ttl=2
                              sock: bound to 192.168.0.5:46484, receive buffer size=212992, send buffer size=212992
                              mcast_sock: bound to 192.168.0.5:46655, send buffer size=212992, receive buffer size=212992
                              2013-09-12 07:38:40,040|main|DEBUG - address=localhost-28168, cluster=ISPN, physical address=192.168.0.5:46484
                              2013-09-12 07:39:10,056|main|DEBUG -
                              [localhost-28168 setDigest()]
                              existing digest:  []
                              new digest:       localhost-28168: [0 (0)]
                              resulting digest: localhost-28168: [0 (0)]
                              2013-09-12 07:39:10,056|main|DEBUG - localhost-28168: installing [localhost-28168|0] [localhost-28168]
                              2013-09-12 07:39:10,057|main|DEBUG - resuming message garbage collection
                              2013-09-12 07:39:10,058|main|DEBUG - VIEW_CHANGE received: [localhost-28168]
                              2013-09-12 07:39:10,060|main|DEBUG - New view accepted: [localhost-28168|0] [localhost-28168]
                              2013-09-12 07:39:10,061|main|INFO  - ISPN000094: Received new cluster view: [localhost-28168|0] [localhost-28168]
                              2013-09-12 07:39:10,063|main|INFO  - View Changed : EventImpl{type=VIEW_CHANGED, newMembers=[localhost-28168], oldMembers=null, localAddress=localhost-28168, viewId=0,

                               

                              subgroupsMerged=null, mergeView=false}
                              2013-09-12 07:39:10,063|main|DEBUG - resuming message garbage collection
                              2013-09-12 07:39:10,064|main|DEBUG - created group (first member). My view is [localhost-28168|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl
                              2013-09-12 07:39:10,065|main|INFO  - ISPN000079: Cache local address is localhost-28168, physical addresses are [192.168.0.5:46484]
                              2013-09-12 07:39:10,065|main|DEBUG - Waiting on view being accepted

                               

                              • 12. Re: Infinispan cluster with Invalidation
                                dan.berindei

                                Viren, what discovery protocol are you using?

                                 

                                You should definitely enable TRACE logging for org.jgroups when debugging this kind of problem. You should also try to form a cluster just with JGroups: http://www.jgroups.org/manual/html/ch02.html#ItDoesntWork