9 Replies Latest reply on Jul 12, 2011 6:44 AM by Pete Muir

    How to set up infinispan over LAN.

    Marcin Matuszak Newbie

      Hi there,

      Maybe my questions will be silly, but you have to forgive me, because I'm just starting my journey with Infinispan and data grids in general. So here is my first problem. I wrote a simple Java application, which should enable simple operations like adding, deleting etc. data to/from data grid. It works when I run 2 instances of that application on one computer, but when I try to run the third instance on another PC then they recognize, that this grid has 3 nodes, but I can't see this data on the third node. I tried to run JGroups Draw demo and it worked perfectly, so now I wonder what I'm doing wrong?

      Here is some of my code, when you see:

      .
      .
      .
      


      that means that I removed some of my code to make it shorter.

       .
       .
       .
       CacheManager cm = null;
       try {
       cm = new DefaultCacheManager("cfg.xml");
       } catch (IOException e1) {
       // TODO Auto-generated catch block
       System.out.println("Nie wczytano pliku konfiguracyjnego");
       e1.printStackTrace();
       }
       Cache customCache = cm.getCache();
      
       System.out.println(customCache.getName()+" - rozmiar: "+customCache.size());
      .
      .
      .
      
       switch(choice){
       case 1: {
      .
      .
      .
       System.out.println(str + " - " +customCache.get(str));
       break;
       }
       case 2: {
      .
      .
      .
      
       customCache.put(str, str2);
       System.out.println("dodano: " +customCache.get(str));
       break;
       }
       case 3: {
       System.out.println("#putIfAbsent# Podaj klucz: ");
      .
      .
      .
       customCache.putIfAbsent(str, str2);
       System.out.println("dodano: " +customCache.get(str));
       break;
       }
       case 4: {
      .
      .
      .
       customCache.replace(str, str2);
       System.out.println("zmieniono na: " +customCache.get(str));
       break;
       }
       case 5: {
      .
      .
      .
      
       customCache.remove(str);
       if(customCache.get(str)==null)
       System.out.println("klucz: "+str+" usunieto.");
       else
       System.out.println("Blad podczas usuwania.");
       break;
       }
       case 6: {
      .
      .
      .
      
       Map<String, String> rand = new HashMap<String, String>();
       int i=0;
       while (rand.size() < amount){
       rand.put("a"+(i+number), "b"+(i+number));
       i++;
       }
      
       customCache.putAll(rand);
      
       System.out.println("");
      
       System.out.println("dodano "+amount+" kluczy od "+number+" do "+ (amount-1+number)+", typu k: a1 v: b1 itd..");
       break;
      
       }
       case 7:{
       System.out.println(customCache.getName()+" - rozmiar: "+customCache.size());
       break;
       }
      
       case 8:{
      .
      .
      .
      
       for(int i = from; i<=to; i++){
       System.out.println("a"+i+" "+customCache.get("a"+i));
       if(customCache.get("a"+i)==null){
       System.out.println("**** **** **** **** **** ****");
       }
       }
       break;
       }
       case 10: {
       choice = 9;
       if(customCache.size()!=0)
       customCache.clear();
       break;
       }
       default: break;
       }
       }
      
       System.out.println(customCache.getName()+" - rozmiar: "+customCache.size());
       System.out.println("KONIEC");
       }
      }
      


      and here is my cfg.xml:

      <?xml version="1.0" encoding="UTF-8"?>
      
      <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0">
       <global>
       <transport transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport" clusterName="infinispan-cluster" distributedSyncTimeout="50000">
      
       </transport>
       </global>
      
       <default>
       <clustering mode="distribution">
       <l1 enabled="true" lifespan="10000"/>
       <hash numOwners="2"/>
       <sync/>
       </clustering>
       </default>
      </infinispan>
      


      Could you please help me with that?

      Best regards,
      Martin

        • 1. Re: How to set up infinispan over LAN.
          Galder Zamarreño Master

          The first thing to check is whether at the transport level, the 3 nodes are part of the JGroups view. In other words, enabling logging in each node, http://www.jboss.org/community/wiki/InfinispanTechnicalFAQs#How_can_I_enable_logging and check if logs for all 3 nodes shows a view of 3 members.

          Chances are it you won't see 3 nodes, which will indicate that there's some kind of multicast traffic issue which is not flowing between the two physical machines.

          • 2. Re: How to set up infinispan over LAN.
            Marcin Matuszak Newbie

            thanks for quick response. Here is a piece of my log:

            2009-09-03 11:24:21,562 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,infinispan-cluster,osoz-fd029efaa5-45095) Received new cluster view: [osoz-fd029efaa5-45095|24] [osoz-fd029efaa5-45095, esbopen-25477, osoz-fd029efaa5-34016]
            


            esbopen-25477 is the one that's on another PC. So I guess they "see" each other. But I'm also wondering why there are 4 nodes if I started my application only three times?

            I also tried McastReceiverTest and McastSenderTest that are provided with JGroups and it worked just fine.

            • 3. Re: How to set up infinispan over LAN.
              Galder Zamarreño Master

              You must have a 3rd process running in the machine.

              Also, note that you're using distribution with number of nodes 2, so when data is stored, is only stored in 2 nodes in the cluster and hence, this data might only be stored in the two nodes that are local. However, if you run a get() call from the other machine, it should pull the data from the other nodes. I suppose you're calling get() from the 3rd node and it's not retuning anything?

              Check the logs on the other machine to verify the view making sure that it contains all 3 nodes and check what happens in the logs when get() is called. It should either find it locally, or ask the cluster to retrieve the data.

              Finally, the view number in the log presented shows 24, so there're seem to have been 24 view changes. You might wanna shut down all processes and start from scratch to verify that you get a cluster only formed by the nodes intended.

              • 4. Re: How to set up infinispan over LAN.
                Marcin Matuszak Newbie

                this is going to be a long post due to code amount, sorry for that :)
                So here' s the thing:

                last time I checked when I called get() from that 3rd node i got "null", but now things are different. I must have changed something.

                This is what I do:

                1) I start 2 clasters on local machine and one on the other. I use the following cfg.xml:

                <?xml version="1.0" encoding="UTF-8"?>
                
                <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0">
                 <global>
                 <transport transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport" clusterName="infinispan-cluster" distributedSyncTimeout="50000">
                
                 </transport>
                 </global>
                
                 <default>
                 <clustering mode="distribution">
                 <l1 enabled="true" lifespan="10000"/>
                 <hash numOwners="3"/>
                 <sync/>
                 </clustering>
                 </default>
                </infinispan>
                


                2) I try to add 10 items to one of the local clusters and then it suspends and I get error that you can see in this log (local machine, because on the 3rd one there is no new log):

                2009-09-04 12:31:19,156 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
                2009-09-04 12:31:19,203 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
                2009-09-04 12:31:19,203 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Starting JGroups Channel
                2009-09-04 12:31:19,203 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
                2009-09-04 12:31:19,562 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [osoz-fd029efaa5-47180|1] [osoz-fd029efaa5-47180, osoz-fd029efaa5-46630]
                2009-09-04 12:31:19,609 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Cache local address is osoz-fd029efaa5-46630
                2009-09-04 12:31:19,625 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain size: 6
                2009-09-04 12:31:19,625 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain is:
                 >> org.infinispan.interceptors.InvocationContextInterceptor
                 >> org.infinispan.interceptors.DistTxInterceptor
                 >> org.infinispan.interceptors.NotificationInterceptor
                 >> org.infinispan.interceptors.DistributionInterceptor
                 >> org.infinispan.interceptors.DistLockingInterceptor
                 >> org.infinispan.interceptors.CallInterceptor
                2009-09-04 12:31:19,625 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (main) Invoked with command SizeCommand{containerSize=0} and InvocationContext [NonTxInvocationContext{flags=null}]
                2009-09-04 12:31:19,625 TRACE [org.infinispan.interceptors.CallInterceptor] (main) Executing command: SizeCommand{containerSize=0}.
                2009-09-04 12:31:51,203 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-46630) Received new cluster view: [osoz-fd029efaa5-47180|2] [osoz-fd029efaa5-47180, o2009-09-04 12:31:51,203 INFO [org.in2009-09-04 12:32:09,468 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,osoz-fd029efaa5-46630) Read version 400
                2009-09-04 12:32:09,468 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,osoz-fd029efaa5-46630) Attempting to execute command: SingleRpcCommand{interceptorChain=null, cacheName='org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME', command=PutMapCommand{map={a9=b9, a0=b0, a1=b1, a2=b2, a3=b3, a4=b4, a5=b5, a6=b6, a7=b7, a8=b8}}} [sender=osoz-fd029efaa5-47180]
                2009-09-04 12:32:09,468 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,osoz-fd029efaa5-46630) Enough waiting; replayIgnored = false, sr STATE_PREEXISTED
                2009-09-04 12:32:09,468 TRACE [org.infinispan.commands.remote.BaseRpcCommand] (Incoming-2,osoz-fd029efaa5-46630) Invoking command PutMapCommand{map={a9=b9, a0=b0, a1=b1, a2=b2, a3=b3, a4=b4, a5=b5, a6=b6, a7=b7, a8=b8}}, with originLocal flag set to false.
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Invoked with command PutMapCommand{map={a9=b9, a0=b0, a1=b1, a2=b2, a3=b3, a4=b4, a5=b5, a6=b6, a7=b7, a8=b8}} and InvocationContext [NonTxInvocationContext{flags=null}]
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a9 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a0 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a1 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a2 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a3 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a4 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a5 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a6 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a7 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a8 since entry is mapped to current node, or is in L1
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a9 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a0 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a1 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a2 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a3 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a4 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a5 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a6 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a7 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a8 with acquisition timeout of 10000 millis
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.CallInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Executing command: PutMapCommand{map={a9=b9, a0=b0, a1=b1, a2=b2, a3=b3, a4=b4, a5=b5, a6=b6, a7=b7, a8=b8}}.
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Number of entries in context: 10
                2009-09-04 12:32:09,484 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a8 removed=false valid=true changed=true created=true value=b8]
                2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a8] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a8
                2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a7 removed=false valid=true changed=true created=true value=b7]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a7] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a7
                2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a6 removed=false valid=true changed=true created=true value=b6]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a6] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a6
                2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a5 removed=false valid=true changed=true created=true value=b5]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a5] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a5
                2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a4 removed=false valid=true changed=true created=true value=b4]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a4] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a4
                2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a3 removed=false valid=true changed=true created=true value=b3]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a3] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a3
                2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a2 removed=false valid=true changed=true created=true value=b2]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a2] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a2
                2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a1 removed=false valid=true changed=true created=true value=b1]
                2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInter2009-09-04 12:32:09,531 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (OOB-4,osoz-fd029efaa5-47180) Read version 400
                2009-09-04 12:32:24,406 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (main) responses: [sender=osoz-fd029efaa5-46630, retval=SuccessfulResponse, received=true, suspected=false]
                [sender=esbopen-9704, retval=null, received=false, suspected=false]
                
                2009-09-04 12:32:24,406 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Cache [osoz-fd029efaa5-47180]: responses for command SingleRpcCommand:
                [sender=osoz-fd029efaa5-46630, retval=SuccessfulResponse, received=true, suspected=false]
                [sender=esbopen-9704, retval=null, received=false, suspected=false]
                
                2009-09-04 12:32:24,406 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (main) replication exception:
                org.infinispan.util.concurrent.TimeoutException: Replication timeout for esbopen-9704
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:326)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:87)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:103)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:211)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:193)
                 at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:265)
                 at org.infinispan.interceptors.DistributionInterceptor.visitPutMapCommand(DistributionInterceptor.java:148)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
                 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)
                 at org.infinispan.interceptors.TxInterceptor.visitPutMapCommand(TxInterceptor.java:145)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
                 at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
                 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:237)
                 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:553)
                 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:187)
                 at inf.infini2.main(infini2.java:232)
                2009-09-04 12:32:24,421 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Replication exception
                org.infinispan.util.concurrent.TimeoutException: Replication timeout for esbopen-9704
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:326)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:87)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:103)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:211)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:193)
                 at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:265)
                 at org.infinispan.interceptors.DistributionInterceptor.visitPutMapCommand(DistributionInterceptor.java:148)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
                 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)
                 at org.infinispan.interceptors.TxInterceptor.visitPutMapCommand(TxInterceptor.java:145)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
                 at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
                 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:237)
                 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:553)
                 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:187)
                 at inf.infini2.main(infini2.java:232)
                2009-09-04 12:32:24,421 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (main) Execution error:
                org.infinispan.util.concurrent.TimeoutException: Replication timeout for esbopen-9704
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:326)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:87)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:103)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:211)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
                 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:193)
                 at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:265)
                 at org.infinispan.interceptors.DistributionInterceptor.visitPutMapCommand(DistributionInterceptor.java:148)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
                 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)
                 at org.infinispan.interceptors.TxInterceptor.visitPutMapCommand(TxInterceptor.java:145)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
                 at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
                 at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
                 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
                 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
                 at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:237)
                 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:553)
                 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:187)
                 at inf.infini2.main(infini2.java:232)
                
                


                So I try to do this in a different way:

                1) start three nodes with cfg.xml shown below:

                <?xml version="1.0" encoding="UTF-8"?>
                
                <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0">
                 <global>
                 <transport clusterName="demoCluster"/>
                 </global>
                
                 <default>
                 <clustering mode="replication">
                 <stateRetrieval timeout="20000" fetchInMemoryState="true"/>
                 <sync replTimeout="20000"/>
                 </clustering>
                 </default>
                </infinispan>
                


                And here I've got 2 logs, first is from the local machine:

                2009-09-04 12:49:53,218 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
                2009-09-04 12:49:53,265 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
                2009-09-04 12:49:53,265 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Starting JGroups Channel
                2009-09-04 12:49:53,265 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
                2009-09-04 12:49:53,625 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [osoz-fd029efaa5-60337|1] [osoz-fd029efaa5-60337, osoz-fd029efaa5-51501]
                2009-09-04 12:49:53,656 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Cache local address is osoz-fd029efaa5-51501
                2009-09-04 12:49:53,671 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain size: 6
                2009-09-04 12:49:53,671 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain is:
                 >> org.infinispan.interceptors.InvocationContextInterceptor
                 >> org.infinispan.interceptors.TxInterceptor
                 >> org.infinispan.interceptors.NotificationInterceptor
                 >> org.infinispan.interceptors.ReplicationInterceptor
                 >> org.infinispan.interceptors.LockingInterceptor
                 >> org.infinispan.interceptors.CallInterceptor
                2009-09-04 12:49:53,671 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (main) Data container is 653967
                2009-09-04 12:49:53,671 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (main) Initiating state transfer process
                2009-09-04 12:49:53,671 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
                2009-09-04 12:49:53,687 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-51501) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 12:49:53,687 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Applying state
                2009-09-04 12:49:53,703 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,osoz-fd029efaa5-51501) Read version 400
                2009-09-04 12:49:53,703 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Integrating transaction log
                2009-09-04 12:49:53,703 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Applying commit log
                2009-09-04 12:49:53,703 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Finished applying commit log
                2009-09-04 12:49:53,703 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-51501) dests=[osoz-fd029efaa5-60337], command=StateTransferControlCommand{enabled=true}, mode=SYNCHRONOUS, timeout=20000
                2009-09-04 12:49:53,718 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,osoz-fd029efaa5-51501) Wrote version 400
                2009-09-04 12:49:53,718 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,osoz-fd029efaa5-51501) real_dests=[osoz-fd029efaa5-60337]
                2009-09-04 12:49:53,734 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (OOB-1,osoz-fd029efaa5-51501) Read version 400
                2009-09-04 12:49:53,734 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,osoz-fd029efaa5-51501) responses: [sender=osoz-fd029efaa5-60337, retval=null, received=true, suspected=false]
                
                2009-09-04 12:49:53,734 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-51501) Cache [osoz-fd029efaa5-51501]: responses for command StateTransferControlCommand:
                null
                2009-09-04 12:49:53,734 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Retrieving/Applying post-flush commits
                2009-09-04 12:49:53,734 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Applying commit log
                2009-09-04 12:49:53,734 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Finished applying commit log
                2009-09-04 12:49:53,734 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Retrieving/Applying pending prepares
                2009-09-04 12:49:52009-09-04 12:49:53,734 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (OOB-3,osoz-fd029efaa5-60337) Read version 400
                2009-09-04 12:49:53,734 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (OOB-3,osoz-fd029efaa5-60337) Wrote version 400
                2009-09-04 12:49:59,031 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,osoz-fd029efaa5-60337) Received new cluster view: [osoz-fd029efaa5-60337|2] [osoz-fd029efaa5-60337, osoz-fd029efaa5-51501, esbopen-3341]
                2009-09-04 12:49:59,296 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
                2009-09-04 12:49:59,296 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? true
                2009-09-04 12:49:59,296 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
                2009-09-04 12:49:59,296 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Writing 0 StoredEntries to stream
                2009-09-04 12:49:59,296 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Transaction log size is 0
                2009-09-04 12:49:59,296 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Waiting for a distributed sync block
                2009-09-04 12:50:01,203 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
                2009-09-04 12:50:01,203 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? false
                2009-09-04 12:50:01,203 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
                2009-09-04 12:50:01,203 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Not providing state!
                2009-09-04 12:50:04,031 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
                2009-09-04 12:50:04,031 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? false
                2009-09-04 12:50:04,031 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
                2009-09-04 12:50:04,031 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Not providing state!
                2009-09-04 12:50:09,156 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
                2009-09-04 12:50:09,156 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? false
                2009-09-04 12:50:09,156 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
                2009-09-04 12:50:09,156 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Not providing state!
                2009-09-04 12:50:19,109 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
                2009-09-04 12:50:19,109 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? false
                2009-09-04 12:50:19,109 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
                2009-09-04 12:50:19,109 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Not providing state!
                2009-09-04 12:50:47,093 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,osoz-fd029efaa5-60337) Received new cluster view: [osoz-fd029efaa5-60337|3] [osoz-fd029efaa5-60337, osoz-fd029efaa5-51501]
                2009-09-04 12:50:59,296 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Caught while responding to state transfer request
                org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 60 seconds)
                 at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:164)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:78)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:456)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:782)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1484)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:644)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:573)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
                 at java.lang.Thread.run(Unknown Source)
                Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeou2009-09-04 12:50:59,703 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-51501) Caught while responding to state transfer request
                org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 60 s2009-09-04 12:51:42,015 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Thread-0) Disconnecting and closing JGroups Channel
                2009-09-04 12:51:42,078 ERROR [org.jgroups.protocols.UDP] (OOB-7,osoz-fd029efaa5-60337) failed sending message to osoz-fd029efaa5-51501 (93 bytes)
                java.lang.Exception: dest=/10.10.8.100:2669 (96 bytes)
                 at org.jgroups.protocols.UDP._send(UDP.java:208)
                 at org.jgroups.protocols.UDP.sendUnicast(UDP.java:191)
                 at org.jgroups.protocols.TP.sendToSingleMember(TP.java:1106)
                 at org.jgroups.protocols.TP.doSend(TP.java:1090)
                 at org.jgroups.protocols.TP.send(TP.java:1073)
                 at org.jgroups.protocols.TP.down(TP.java:934)
                 at org.jgroups.protocols.Discovery.down(Discovery.java:410)
                 at org.jgroups.protocols.MERGE2.down(MERGE2.java:161)
                 at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:334)
                 at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220)
                 at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:76)
                 at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:653)
                 at org.jgroups.protocols.UNICAST.sendAck(UNICAST.java:670)
                 at org.jgroups.protocols.UNICAST.sendAckForMessage(UNICAST.java:679)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:535)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.p2009-09-04 12:51:42,500 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-51501) Received new cluster view: [osoz-fd029efaa5-51501|4] [osoz-fd029efaa5-51501]
                2009-09-04 12:51:42,796 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Thread-0) Disconnecting and closing JGroups Channel
                2009-09-04 12:51:42,796 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Thread-0) Stopping the RpcDispatcher
                2009-09-04 12:51:42,796 DEBUG [org.infinispan.lifecycle.ComponentStatus] (Thread-0) Ignoring call to stop() as current state is STOPPING
                pl.send(Native Method)
                 at java.net.DatagramSocket.send(Unknown Source)
                 at org.jgroups.protocols.UDP._send(UDP.java:204)
                 ... 29 more
                2009-09-04 12:51:42,390 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Thread-0) Stopping the RpcDispatcher
                2009-09-04 12:51:42,390 DEBUG [org.infinispan.lifecycle.ComponentStatus] (Thread-0) Ignoring call to stop() as current state is STOPPING
                
                



                And the socond, from remote one:

                2009-09-04 10:48:56,684 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
                2009-09-04 10:48:56,747 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
                2009-09-04 10:48:56,762 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Starting JGroups Channel
                2009-09-04 10:48:56,762 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
                2009-09-04 10:48:57,512 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [osoz-fd029efaa5-60337|2] [osoz-fd029efaa5-60337, osoz-fd029efaa5-51501, esbopen-3341]
                2009-09-04 10:48:57,544 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Cache local address is esbopen-3341
                2009-09-04 10:48:57,591 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain size: 6
                2009-09-04 10:48:57,591 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain is:
                 >> org.infinispan.interceptors.InvocationContextInterceptor
                 >> org.infinispan.interceptors.TxInterceptor
                 >> org.infinispan.interceptors.NotificationInterceptor
                 >> org.infinispan.interceptors.ReplicationInterceptor
                 >> org.infinispan.interceptors.LockingInterceptor
                 >> org.infinispan.interceptors.CallInterceptor
                2009-09-04 10:48:57,591 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (main) Data container is 22292701
                2009-09-04 10:48:57,591 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (main) Initiating state transfer process
                2009-09-04 10:48:57,591 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
                2009-09-04 10:48:57,591 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:48:57,591 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:48:57,950 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:48:57,950 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
                org.infinispan.statetransfer.StateTransferException: java.lang.ClassCastException: java.util.HashMap
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImpl.java:333)
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:304)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1437)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
                 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
                 at org.jgroups.protocols.FC.up(FC.java:487)
                 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
                 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
                 at org.jgroups.protocols.PING.up(PING.java:282)
                 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
                 at org.jgroups.protocols.TP.access$100(TP.java:53)
                 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
                 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                Caused by: java.lang.ClassCastException: java.util.HashMap
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImpl.java:328)
                 ... 31 more
                2009-09-04 10:48:57,950 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-51501
                2009-09-04 10:48:57,950 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:48:57,950 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:48:58,356 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:48:58,356 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
                org.infinispan.statetransfer.StateTransferException: java.lang.ClassCastException: java.util.HashMap
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImpl.java:333)
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:304)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1437)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
                 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
                 at org.jgroups.protocols.FC.up(FC.java:487)
                 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
                 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
                 at org.jgroups.protocols.PING.up(PING.java:282)
                 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
                 at org.jgroups.protocols.TP.access$100(TP.java:53)
                 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
                 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                Caused by: java.lang.ClassCastException: java.util.HashMap
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImpl.java:328)
                 ... 31 more
                2009-09-04 10:48:58,356 WARN [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Could not find available peer for state, backing off and retrying
                2009-09-04 10:48:59,559 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
                2009-09-04 10:48:59,559 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:48:59,559 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:48:59,669 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:48:59,669 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
                2009-09-04 10:48:59,669 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
                org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1437)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
                 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
                 at org.jgroups.protocols.FC.up(FC.java:487)
                 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
                 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
                 at org.jgroups.protocols.PING.up(PING.java:282)
                 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
                 at org.jgroups.protocols.TP.access$100(TP.java:53)
                 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
                 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                2009-09-04 10:48:59,669 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-51501
                2009-09-04 10:48:59,669 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:48:59,669 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:48:59,856 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:48:59,856 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
                2009-09-04 10:48:59,856 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
                org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1437)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
                 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
                 at org.jgroups.protocols.FC.up(FC.java:487)
                 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
                 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
                 at org.jgroups.protocols.PING.up(PING.java:282)
                 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
                 at org.jgroups.protocols.TP.access$100(TP.java:53)
                 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
                 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                2009-09-04 10:48:59,950 WARN [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Could not find available peer for state, backing off and retrying
                2009-09-04 10:49:02,356 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
                2009-09-04 10:49:02,356 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:49:02,356 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:49:02,481 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:49:02,481 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
                2009-09-04 10:49:02,481 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
                org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1437)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
                 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
                 at org.jgroups.protocols.FC.up(FC.java:487)
                 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
                 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
                 at org.jgroups.protocols.PING.up(PING.java:282)
                 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
                 at org.jgroups.protocols.TP.access$100(TP.java:53)
                 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
                 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                2009-09-04 10:49:02,481 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-51501
                2009-09-04 10:49:02,481 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:49:02,481 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:49:02,669 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:49:02,669 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
                2009-09-04 10:49:02,669 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
                org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1437)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
                 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
                 at org.jgroups.protocols.FC.up(FC.java:487)
                 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
                 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
                 at org.jgroups.protocols.PING.up(PING.java:282)
                 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
                 at org.jgroups.protocols.TP.access$100(TP.java:53)
                 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
                 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                2009-09-04 10:49:02,684 WARN [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Could not find available peer for state, backing off and retrying
                2009-09-04 10:49:07,497 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
                2009-09-04 10:49:07,497 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:49:07,497 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:49:07,606 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:49:07,606 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
                2009-09-04 10:49:07,606 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
                org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1437)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
                 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
                 at org.jgroups.protocols.FC.up(FC.java:487)
                 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
                 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
                 at org.jgroups.protocols.PING.up(PING.java:282)
                 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
                 at org.jgroups.protocols.TP.access$100(TP.java:53)
                 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
                 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                2009-09-04 10:49:07,606 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-51501
                2009-09-04 10:49:07,622 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:49:07,622 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:49:07,809 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:49:07,809 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
                2009-09-04 10:49:07,809 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
                org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
                 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
                 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
                 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
                 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
                 at org.jgroups.JChannel.up(JChannel.java:1437)
                 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
                 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
                 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
                 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
                 at org.jgroups.protocols.FC.up(FC.java:487)
                 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
                 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
                 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
                 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
                 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
                 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
                 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
                 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
                 at org.jgroups.stack.Protocol.up(Protocol.java:336)
                 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
                 at org.jgroups.protocols.PING.up(PING.java:282)
                 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
                 at org.jgroups.protocols.TP.access$100(TP.java:53)
                 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
                 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
                 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
                 at java.lang.Thread.run(Thread.java:595)
                2009-09-04 10:49:07,809 WARN [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Could not find available peer for state, backing off and retrying
                2009-09-04 10:49:17,419 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
                2009-09-04 10:49:17,419 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
                2009-09-04 10:49:17,419 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
                2009-09-04 10:49:17,575 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
                2009-09-04 10:49:17,575 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
                2009-09-04 10:49:17,575 ERROR


                • 5. Re: How to set up infinispan over LAN.
                  Galder Zamarreño Master

                  What Infinispan version are you using? Beta1?

                  Let's start with the second option tried. I'm a bit confused with the logs attached cos they show different times (one starts at 2009-09-04 10:48:56,684 and the other at 2009-09-04 12:49:53,218). Could you please check whether clocks are synchronized and then try the test again? Please send me the logs to: galder.zamarreno@jboss.com

                  It does appear though that the new node starting up didn't like the state that was provided:

                  Caused by: java.lang.ClassCastException: java.util.HashMap
                  at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImp
                  l.java:328)

                  Had you put any data into the local node before you started the remote node (esbopen-3341)?

                  For the 1st option, esbopen-9704 hasn't replied but can't see logs for this machine. So, could you please repeat this tests and send me the logs for *all* machines? Make sure the clocks are synchronized.

                  Finally, can you make sure you have -Djava.net.preferIPv4Stack=true system property when starting these Infinispan nodes?

                  • 6. Re: How to set up infinispan over LAN.
                    Marcin Matuszak Newbie

                    OK, I send you the e-mail, and here are answers for those questions:

                    - infinispan beta1
                    - clocks are synchronized now, but still somehow there is 2 hour difference in logs.
                    - I start my app with: -Djava.net.preferIPv4Stack=true
                    - I don't remember if I have put any data into the local node before I started the remote one.

                    best regards,
                    Martin

                    • 7. Re: How to set up infinispan over LAN.
                      Marcin Matuszak Newbie

                      I would like to correct my previous post. I'm not using beta1 but alpha5. I will try beta1 now.

                      • 8. Re: How to set up infinispan over LAN.
                        Renzo De Renzi Newbie

                        Good morning guys,

                        I'm using Infinispan 4.2.1 FINAL and I'm experiencing problems with Node Discovery over LAN as well. I've tried with 2 and 3 nodes over a 192.168.0.x LAN, I've no firewalls running and I enabled logging with the log4j library. I can see that each View is gonna not to be updated with the other nodes when they try to join the cluster.

                        This is how I declare the cluster configuration in each node:

                         

                        GlobalConfiguration gc = GlobalConfiguration.getClusteredDefault();

                        Configuration c = new Configuration();

                        Properties p = new Properties();

                        p.setProperty("configurationFile", "jgroups-udp.xml");

                        gc.setTransportProperties(p);

                         

                        any help would be appreciated, thanks.

                        • 9. Re: How to set up infinispan over LAN.
                          Pete Muir Master

                          Renzo, please start a new thread for a new problem!