9 Replies Latest reply on Jul 12, 2011 6:44 AM by pmuir

How to set up infinispan over LAN.

mcmati Sep 4, 2009 2:59 AM

Hi there,

Maybe my questions will be silly, but you have to forgive me, because I'm just starting my journey with Infinispan and data grids in general. So here is my first problem. I wrote a simple Java application, which should enable simple operations like adding, deleting etc. data to/from data grid. It works when I run 2 instances of that application on one computer, but when I try to run the third instance on another PC then they recognize, that this grid has 3 nodes, but I can't see this data on the third node. I tried to run JGroups Draw demo and it worked perfectly, so now I wonder what I'm doing wrong?

Here is some of my code, when you see:

.
.
.

that means that I removed some of my code to make it shorter.

 .
 .
 .
 CacheManager cm = null;
 try {
 cm = new DefaultCacheManager("cfg.xml");
 } catch (IOException e1) {
 // TODO Auto-generated catch block
 System.out.println("Nie wczytano pliku konfiguracyjnego");
 e1.printStackTrace();
 }
 Cache customCache = cm.getCache();

 System.out.println(customCache.getName()+" - rozmiar: "+customCache.size());
.
.
.

 switch(choice){
 case 1: {
.
.
.
 System.out.println(str + " - " +customCache.get(str));
 break;
 }
 case 2: {
.
.
.

 customCache.put(str, str2);
 System.out.println("dodano: " +customCache.get(str));
 break;
 }
 case 3: {
 System.out.println("#putIfAbsent# Podaj klucz: ");
.
.
.
 customCache.putIfAbsent(str, str2);
 System.out.println("dodano: " +customCache.get(str));
 break;
 }
 case 4: {
.
.
.
 customCache.replace(str, str2);
 System.out.println("zmieniono na: " +customCache.get(str));
 break;
 }
 case 5: {
.
.
.

 customCache.remove(str);
 if(customCache.get(str)==null)
 System.out.println("klucz: "+str+" usunieto.");
 else
 System.out.println("Blad podczas usuwania.");
 break;
 }
 case 6: {
.
.
.

 Map<String, String> rand = new HashMap<String, String>();
 int i=0;
 while (rand.size() < amount){
 rand.put("a"+(i+number), "b"+(i+number));
 i++;
 }

 customCache.putAll(rand);

 System.out.println("");

 System.out.println("dodano "+amount+" kluczy od "+number+" do "+ (amount-1+number)+", typu k: a1 v: b1 itd..");
 break;

 }
 case 7:{
 System.out.println(customCache.getName()+" - rozmiar: "+customCache.size());
 break;
 }

 case 8:{
.
.
.

 for(int i = from; i<=to; i++){
 System.out.println("a"+i+" "+customCache.get("a"+i));
 if(customCache.get("a"+i)==null){
 System.out.println("**** **** **** **** **** ****");
 }
 }
 break;
 }
 case 10: {
 choice = 9;
 if(customCache.size()!=0)
 customCache.clear();
 break;
 }
 default: break;
 }
 }

 System.out.println(customCache.getName()+" - rozmiar: "+customCache.size());
 System.out.println("KONIEC");
 }
}

and here is my cfg.xml:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0">
 <global>
 <transport transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport" clusterName="infinispan-cluster" distributedSyncTimeout="50000">

 </transport>
 </global>

 <default>
 <clustering mode="distribution">
 <l1 enabled="true" lifespan="10000"/>
 <hash numOwners="2"/>
 <sync/>
 </clustering>
 </default>
</infinispan>

Could you please help me with that?

Best regards,
Martin

1. Re: How to set up infinispan over LAN.

galder.zamarreno Sep 4, 2009 3:57 AM (in response to mcmati)

The first thing to check is whether at the transport level, the 3 nodes are part of the JGroups view. In other words, enabling logging in each node, http://www.jboss.org/community/wiki/InfinispanTechnicalFAQs#How_can_I_enable_logging and check if logs for all 3 nodes shows a view of 3 members.

Chances are it you won't see 3 nodes, which will indicate that there's some kind of multicast traffic issue which is not flowing between the two physical machines.
Actions
2. Re: How to set up infinispan over LAN.

mcmati Sep 4, 2009 4:15 AM (in response to mcmati)
thanks for quick response. Here is a piece of my log:
2009-09-03 11:24:21,562 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,infinispan-cluster,osoz-fd029efaa5-45095) Received new cluster view: [osoz-fd029efaa5-45095|24] [osoz-fd029efaa5-45095, esbopen-25477, osoz-fd029efaa5-34016]

esbopen-25477 is the one that's on another PC. So I guess they "see" each other. But I'm also wondering why there are 4 nodes if I started my application only three times?

I also tried McastReceiverTest and McastSenderTest that are provided with JGroups and it worked just fine.
Actions
3. Re: How to set up infinispan over LAN.

galder.zamarreno Sep 4, 2009 5:41 AM (in response to mcmati)

You must have a 3rd process running in the machine.

Also, note that you're using distribution with number of nodes 2, so when data is stored, is only stored in 2 nodes in the cluster and hence, this data might only be stored in the two nodes that are local. However, if you run a get() call from the other machine, it should pull the data from the other nodes. I suppose you're calling get() from the 3rd node and it's not retuning anything?

Check the logs on the other machine to verify the view making sure that it contains all 3 nodes and check what happens in the logs when get() is called. It should either find it locally, or ask the cluster to retrieve the data.

Finally, the view number in the log presented shows 24, so there're seem to have been 24 view changes. You might wanna shut down all processes and start from scratch to verify that you get a cluster only formed by the nodes intended.
Actions

4. Re: How to set up infinispan over LAN.

mcmati Sep 4, 2009 7:16 AM (in response to mcmati)

this is going to be a long post due to code amount, sorry for that :)
So here' s the thing:

last time I checked when I called get() from that 3rd node i got "null", but now things are different. I must have changed something.

This is what I do:

1) I start 2 clasters on local machine and one on the other. I use the following cfg.xml:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0">
 <global>
 <transport transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport" clusterName="infinispan-cluster" distributedSyncTimeout="50000">

 </transport>
 </global>

 <default>
 <clustering mode="distribution">
 <l1 enabled="true" lifespan="10000"/>
 <hash numOwners="3"/>
 <sync/>
 </clustering>
 </default>
</infinispan>

2) I try to add 10 items to one of the local clusters and then it suspends and I get error that you can see in this log (local machine, because on the 3rd one there is no new log):

2009-09-04 12:31:19,156 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
2009-09-04 12:31:19,203 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
2009-09-04 12:31:19,203 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Starting JGroups Channel
2009-09-04 12:31:19,203 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2009-09-04 12:31:19,562 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [osoz-fd029efaa5-47180|1] [osoz-fd029efaa5-47180, osoz-fd029efaa5-46630]
2009-09-04 12:31:19,609 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Cache local address is osoz-fd029efaa5-46630
2009-09-04 12:31:19,625 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain size: 6
2009-09-04 12:31:19,625 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain is:
 >> org.infinispan.interceptors.InvocationContextInterceptor
 >> org.infinispan.interceptors.DistTxInterceptor
 >> org.infinispan.interceptors.NotificationInterceptor
 >> org.infinispan.interceptors.DistributionInterceptor
 >> org.infinispan.interceptors.DistLockingInterceptor
 >> org.infinispan.interceptors.CallInterceptor
2009-09-04 12:31:19,625 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (main) Invoked with command SizeCommand{containerSize=0} and InvocationContext [NonTxInvocationContext{flags=null}]
2009-09-04 12:31:19,625 TRACE [org.infinispan.interceptors.CallInterceptor] (main) Executing command: SizeCommand{containerSize=0}.
2009-09-04 12:31:51,203 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-46630) Received new cluster view: [osoz-fd029efaa5-47180|2] [osoz-fd029efaa5-47180, o2009-09-04 12:31:51,203 INFO [org.in2009-09-04 12:32:09,468 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,osoz-fd029efaa5-46630) Read version 400
2009-09-04 12:32:09,468 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,osoz-fd029efaa5-46630) Attempting to execute command: SingleRpcCommand{interceptorChain=null, cacheName='org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME', command=PutMapCommand{map={a9=b9, a0=b0, a1=b1, a2=b2, a3=b3, a4=b4, a5=b5, a6=b6, a7=b7, a8=b8}}} [sender=osoz-fd029efaa5-47180]
2009-09-04 12:32:09,468 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,osoz-fd029efaa5-46630) Enough waiting; replayIgnored = false, sr STATE_PREEXISTED
2009-09-04 12:32:09,468 TRACE [org.infinispan.commands.remote.BaseRpcCommand] (Incoming-2,osoz-fd029efaa5-46630) Invoking command PutMapCommand{map={a9=b9, a0=b0, a1=b1, a2=b2, a3=b3, a4=b4, a5=b5, a6=b6, a7=b7, a8=b8}}, with originLocal flag set to false.
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Invoked with command PutMapCommand{map={a9=b9, a0=b0, a1=b1, a2=b2, a3=b3, a4=b4, a5=b5, a6=b6, a7=b7, a8=b8}} and InvocationContext [NonTxInvocationContext{flags=null}]
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a9 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a0 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a1 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a2 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a3 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a4 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a5 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a6 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a7 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistributionInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Not doing a remote get for key a8 since entry is mapped to current node, or is in L1
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a9 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a0 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a1 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a2 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a3 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a4 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a5 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a6 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a7 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to lock a8 with acquisition timeout of 10000 millis
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.CallInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Executing command: PutMapCommand{map={a9=b9, a0=b0, a1=b1, a2=b2, a3=b3, a4=b4, a5=b5, a6=b6, a7=b7, a8=b8}}.
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Number of entries in context: 10
2009-09-04 12:32:09,484 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a8 removed=false valid=true changed=true created=true value=b8]
2009-09-04 12:32:09,484 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a8] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a8
2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a7 removed=false valid=true changed=true created=true value=b7]
2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a7] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a7
2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a6 removed=false valid=true changed=true created=true value=b6]
2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a6] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a6
2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a5 removed=false valid=true changed=true created=true value=b5]
2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a5] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a5
2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a4 removed=false valid=true changed=true created=true value=b4]
2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a4] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a4
2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a3 removed=false valid=true changed=true created=true value=b3]
2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a3] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a3
2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a2 removed=false valid=true changed=true created=true value=b2]
2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInterceptor] (Incoming-2,osoz-fd029efaa5-46630) Releasing lock on [a2] for owner Thread[Incoming-2,osoz-fd029efaa5-46630,5,Thread Pools]
2009-09-04 12:32:09,500 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (Incoming-2,osoz-fd029efaa5-46630) Attempting to unlock a2
2009-09-04 12:32:09,500 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (Incoming-2,osoz-fd029efaa5-46630) Updating entry (key=a1 removed=false valid=true changed=true created=true value=b1]
2009-09-04 12:32:09,500 TRACE [org.infinispan.interceptors.DistLockingInter2009-09-04 12:32:09,531 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (OOB-4,osoz-fd029efaa5-47180) Read version 400
2009-09-04 12:32:24,406 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (main) responses: [sender=osoz-fd029efaa5-46630, retval=SuccessfulResponse, received=true, suspected=false]
[sender=esbopen-9704, retval=null, received=false, suspected=false]

2009-09-04 12:32:24,406 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Cache [osoz-fd029efaa5-47180]: responses for command SingleRpcCommand:
[sender=osoz-fd029efaa5-46630, retval=SuccessfulResponse, received=true, suspected=false]
[sender=esbopen-9704, retval=null, received=false, suspected=false]

2009-09-04 12:32:24,406 TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (main) replication exception:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for esbopen-9704
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:326)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:87)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:103)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:211)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:193)
 at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:265)
 at org.infinispan.interceptors.DistributionInterceptor.visitPutMapCommand(DistributionInterceptor.java:148)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)
 at org.infinispan.interceptors.TxInterceptor.visitPutMapCommand(TxInterceptor.java:145)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
 at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:237)
 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:553)
 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:187)
 at inf.infini2.main(infini2.java:232)
2009-09-04 12:32:24,421 ERROR [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Replication exception
org.infinispan.util.concurrent.TimeoutException: Replication timeout for esbopen-9704
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:326)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:87)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:103)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:211)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:193)
 at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:265)
 at org.infinispan.interceptors.DistributionInterceptor.visitPutMapCommand(DistributionInterceptor.java:148)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)
 at org.infinispan.interceptors.TxInterceptor.visitPutMapCommand(TxInterceptor.java:145)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
 at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:237)
 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:553)
 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:187)
 at inf.infini2.main(infini2.java:232)
2009-09-04 12:32:24,421 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (main) Execution error:
org.infinispan.util.concurrent.TimeoutException: Replication timeout for esbopen-9704
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:326)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:87)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:103)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:211)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
 at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:193)
 at org.infinispan.interceptors.DistributionInterceptor.handleWriteCommand(DistributionInterceptor.java:265)
 at org.infinispan.interceptors.DistributionInterceptor.visitPutMapCommand(DistributionInterceptor.java:148)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:183)
 at org.infinispan.interceptors.TxInterceptor.visitPutMapCommand(TxInterceptor.java:145)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
 at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:48)
 at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:34)
 at org.infinispan.commands.AbstractVisitor.visitPutMapCommand(AbstractVisitor.java:72)
 at org.infinispan.commands.write.PutMapCommand.acceptVisitor(PutMapCommand.java:59)
 at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:237)
 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:553)
 at org.infinispan.CacheDelegate.putAll(CacheDelegate.java:187)
 at inf.infini2.main(infini2.java:232)

So I try to do this in a different way:

1) start three nodes with cfg.xml shown below:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0">
 <global>
 <transport clusterName="demoCluster"/>
 </global>

 <default>
 <clustering mode="replication">
 <stateRetrieval timeout="20000" fetchInMemoryState="true"/>
 <sync replTimeout="20000"/>
 </clustering>
 </default>
</infinispan>

And here I've got 2 logs, first is from the local machine:

2009-09-04 12:49:53,218 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
2009-09-04 12:49:53,265 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
2009-09-04 12:49:53,265 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Starting JGroups Channel
2009-09-04 12:49:53,265 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2009-09-04 12:49:53,625 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [osoz-fd029efaa5-60337|1] [osoz-fd029efaa5-60337, osoz-fd029efaa5-51501]
2009-09-04 12:49:53,656 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Cache local address is osoz-fd029efaa5-51501
2009-09-04 12:49:53,671 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain size: 6
2009-09-04 12:49:53,671 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain is:
 >> org.infinispan.interceptors.InvocationContextInterceptor
 >> org.infinispan.interceptors.TxInterceptor
 >> org.infinispan.interceptors.NotificationInterceptor
 >> org.infinispan.interceptors.ReplicationInterceptor
 >> org.infinispan.interceptors.LockingInterceptor
 >> org.infinispan.interceptors.CallInterceptor
2009-09-04 12:49:53,671 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (main) Data container is 653967
2009-09-04 12:49:53,671 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (main) Initiating state transfer process
2009-09-04 12:49:53,671 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
2009-09-04 12:49:53,687 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-51501) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 12:49:53,687 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Applying state
2009-09-04 12:49:53,703 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,osoz-fd029efaa5-51501) Read version 400
2009-09-04 12:49:53,703 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Integrating transaction log
2009-09-04 12:49:53,703 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Applying commit log
2009-09-04 12:49:53,703 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Finished applying commit log
2009-09-04 12:49:53,703 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-51501) dests=[osoz-fd029efaa5-60337], command=StateTransferControlCommand{enabled=true}, mode=SYNCHRONOUS, timeout=20000
2009-09-04 12:49:53,718 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,osoz-fd029efaa5-51501) Wrote version 400
2009-09-04 12:49:53,718 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,osoz-fd029efaa5-51501) real_dests=[osoz-fd029efaa5-60337]
2009-09-04 12:49:53,734 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (OOB-1,osoz-fd029efaa5-51501) Read version 400
2009-09-04 12:49:53,734 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (Incoming-2,osoz-fd029efaa5-51501) responses: [sender=osoz-fd029efaa5-60337, retval=null, received=true, suspected=false]

2009-09-04 12:49:53,734 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-51501) Cache [osoz-fd029efaa5-51501]: responses for command StateTransferControlCommand:
null
2009-09-04 12:49:53,734 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Retrieving/Applying post-flush commits
2009-09-04 12:49:53,734 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Applying commit log
2009-09-04 12:49:53,734 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Finished applying commit log
2009-09-04 12:49:53,734 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,osoz-fd029efaa5-51501) Retrieving/Applying pending prepares
2009-09-04 12:49:52009-09-04 12:49:53,734 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (OOB-3,osoz-fd029efaa5-60337) Read version 400
2009-09-04 12:49:53,734 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (OOB-3,osoz-fd029efaa5-60337) Wrote version 400
2009-09-04 12:49:59,031 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-1,osoz-fd029efaa5-60337) Received new cluster view: [osoz-fd029efaa5-60337|2] [osoz-fd029efaa5-60337, osoz-fd029efaa5-51501, esbopen-3341]
2009-09-04 12:49:59,296 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
2009-09-04 12:49:59,296 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? true
2009-09-04 12:49:59,296 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
2009-09-04 12:49:59,296 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Writing 0 StoredEntries to stream
2009-09-04 12:49:59,296 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Transaction log size is 0
2009-09-04 12:49:59,296 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Waiting for a distributed sync block
2009-09-04 12:50:01,203 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
2009-09-04 12:50:01,203 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? false
2009-09-04 12:50:01,203 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
2009-09-04 12:50:01,203 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Not providing state!
2009-09-04 12:50:04,031 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
2009-09-04 12:50:04,031 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? false
2009-09-04 12:50:04,031 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
2009-09-04 12:50:04,031 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Not providing state!
2009-09-04 12:50:09,156 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
2009-09-04 12:50:09,156 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? false
2009-09-04 12:50:09,156 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
2009-09-04 12:50:09,156 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Not providing state!
2009-09-04 12:50:19,109 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Received request to generate state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to generate state.
2009-09-04 12:50:19,109 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Generating state. Can provide? false
2009-09-04 12:50:19,109 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Wrote version 400
2009-09-04 12:50:19,109 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Not providing state!
2009-09-04 12:50:47,093 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,osoz-fd029efaa5-60337) Received new cluster view: [osoz-fd029efaa5-60337|3] [osoz-fd029efaa5-60337, osoz-fd029efaa5-51501]
2009-09-04 12:50:59,296 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-60337) Caught while responding to state transfer request
org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 60 seconds)
 at org.infinispan.statetransfer.StateTransferManagerImpl.generateState(StateTransferManagerImpl.java:164)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.generateState(InboundInvocationHandlerImpl.java:78)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.getState(JGroupsTransport.java:456)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:782)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1484)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderHandler.process(STREAMING_STATE_TRANSFER.java:644)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER$StateProviderThreadSpawner$1.run(STREAMING_STATE_TRANSFER.java:573)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
 at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeou2009-09-04 12:50:59,703 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (STREAMING_STATE_TRANSFER sender,osoz-fd029efaa5-51501) Caught while responding to state transfer request
org.infinispan.statetransfer.StateTransferException: java.util.concurrent.TimeoutException: Timed out waiting for a cluster-wide sync to be acquired. (timeout = 60 s2009-09-04 12:51:42,015 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Thread-0) Disconnecting and closing JGroups Channel
2009-09-04 12:51:42,078 ERROR [org.jgroups.protocols.UDP] (OOB-7,osoz-fd029efaa5-60337) failed sending message to osoz-fd029efaa5-51501 (93 bytes)
java.lang.Exception: dest=/10.10.8.100:2669 (96 bytes)
 at org.jgroups.protocols.UDP._send(UDP.java:208)
 at org.jgroups.protocols.UDP.sendUnicast(UDP.java:191)
 at org.jgroups.protocols.TP.sendToSingleMember(TP.java:1106)
 at org.jgroups.protocols.TP.doSend(TP.java:1090)
 at org.jgroups.protocols.TP.send(TP.java:1073)
 at org.jgroups.protocols.TP.down(TP.java:934)
 at org.jgroups.protocols.Discovery.down(Discovery.java:410)
 at org.jgroups.protocols.MERGE2.down(MERGE2.java:161)
 at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:334)
 at org.jgroups.protocols.FD_ALL.down(FD_ALL.java:220)
 at org.jgroups.protocols.VERIFY_SUSPECT.down(VERIFY_SUSPECT.java:76)
 at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:653)
 at org.jgroups.protocols.UNICAST.sendAck(UNICAST.java:670)
 at org.jgroups.protocols.UNICAST.sendAckForMessage(UNICAST.java:679)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:535)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.p2009-09-04 12:51:42,500 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,osoz-fd029efaa5-51501) Received new cluster view: [osoz-fd029efaa5-51501|4] [osoz-fd029efaa5-51501]
2009-09-04 12:51:42,796 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Thread-0) Disconnecting and closing JGroups Channel
2009-09-04 12:51:42,796 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Thread-0) Stopping the RpcDispatcher
2009-09-04 12:51:42,796 DEBUG [org.infinispan.lifecycle.ComponentStatus] (Thread-0) Ignoring call to stop() as current state is STOPPING
pl.send(Native Method)
 at java.net.DatagramSocket.send(Unknown Source)
 at org.jgroups.protocols.UDP._send(UDP.java:204)
 ... 29 more
2009-09-04 12:51:42,390 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Thread-0) Stopping the RpcDispatcher
2009-09-04 12:51:42,390 DEBUG [org.infinispan.lifecycle.ComponentStatus] (Thread-0) Ignoring call to stop() as current state is STOPPING

And the socond, from remote one:

2009-09-04 10:48:56,684 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
2009-09-04 10:48:56,747 DEBUG [org.infinispan.marshall.jboss.JBossMarshaller] (main) Using JBoss Marshalling based marshaller.
2009-09-04 10:48:56,762 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Starting JGroups Channel
2009-09-04 10:48:56,762 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Unable to use any JGroups configuration mechanisms provided in properties {}. Using default JGroups configuration!
2009-09-04 10:48:57,512 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Received new cluster view: [osoz-fd029efaa5-60337|2] [osoz-fd029efaa5-60337, osoz-fd029efaa5-51501, esbopen-3341]
2009-09-04 10:48:57,544 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (main) Cache local address is esbopen-3341
2009-09-04 10:48:57,591 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain size: 6
2009-09-04 10:48:57,591 DEBUG [org.infinispan.interceptors.InterceptorChain] (main) Interceptor chain is:
 >> org.infinispan.interceptors.InvocationContextInterceptor
 >> org.infinispan.interceptors.TxInterceptor
 >> org.infinispan.interceptors.NotificationInterceptor
 >> org.infinispan.interceptors.ReplicationInterceptor
 >> org.infinispan.interceptors.LockingInterceptor
 >> org.infinispan.interceptors.CallInterceptor
2009-09-04 10:48:57,591 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (main) Data container is 22292701
2009-09-04 10:48:57,591 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (main) Initiating state transfer process
2009-09-04 10:48:57,591 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
2009-09-04 10:48:57,591 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:48:57,591 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:48:57,950 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:48:57,950 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: java.lang.ClassCastException: java.util.HashMap
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImpl.java:333)
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:304)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1437)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
 at org.jgroups.protocols.FC.up(FC.java:487)
 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
 at org.jgroups.protocols.PING.up(PING.java:282)
 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
 at org.jgroups.protocols.TP.access$100(TP.java:53)
 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.ClassCastException: java.util.HashMap
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImpl.java:328)
 ... 31 more
2009-09-04 10:48:57,950 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-51501
2009-09-04 10:48:57,950 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:48:57,950 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:48:58,356 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:48:58,356 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: java.lang.ClassCastException: java.util.HashMap
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImpl.java:333)
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:304)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1437)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
 at org.jgroups.protocols.FC.up(FC.java:487)
 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
 at org.jgroups.protocols.PING.up(PING.java:282)
 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
 at org.jgroups.protocols.TP.access$100(TP.java:53)
 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.ClassCastException: java.util.HashMap
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImpl.java:328)
 ... 31 more
2009-09-04 10:48:58,356 WARN [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Could not find available peer for state, backing off and retrying
2009-09-04 10:48:59,559 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
2009-09-04 10:48:59,559 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:48:59,559 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:48:59,669 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:48:59,669 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
2009-09-04 10:48:59,669 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1437)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
 at org.jgroups.protocols.FC.up(FC.java:487)
 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
 at org.jgroups.protocols.PING.up(PING.java:282)
 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
 at org.jgroups.protocols.TP.access$100(TP.java:53)
 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
2009-09-04 10:48:59,669 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-51501
2009-09-04 10:48:59,669 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:48:59,669 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:48:59,856 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:48:59,856 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
2009-09-04 10:48:59,856 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1437)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
 at org.jgroups.protocols.FC.up(FC.java:487)
 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
 at org.jgroups.protocols.PING.up(PING.java:282)
 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
 at org.jgroups.protocols.TP.access$100(TP.java:53)
 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
2009-09-04 10:48:59,950 WARN [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Could not find available peer for state, backing off and retrying
2009-09-04 10:49:02,356 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
2009-09-04 10:49:02,356 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:49:02,356 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:49:02,481 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:49:02,481 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
2009-09-04 10:49:02,481 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1437)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
 at org.jgroups.protocols.FC.up(FC.java:487)
 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
 at org.jgroups.protocols.PING.up(PING.java:282)
 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
 at org.jgroups.protocols.TP.access$100(TP.java:53)
 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
2009-09-04 10:49:02,481 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-51501
2009-09-04 10:49:02,481 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:49:02,481 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:49:02,669 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:49:02,669 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
2009-09-04 10:49:02,669 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1437)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
 at org.jgroups.protocols.FC.up(FC.java:487)
 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
 at org.jgroups.protocols.PING.up(PING.java:282)
 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
 at org.jgroups.protocols.TP.access$100(TP.java:53)
 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
2009-09-04 10:49:02,684 WARN [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Could not find available peer for state, backing off and retrying
2009-09-04 10:49:07,497 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
2009-09-04 10:49:07,497 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:49:07,497 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:49:07,606 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:49:07,606 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
2009-09-04 10:49:07,606 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1437)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
 at org.jgroups.protocols.FC.up(FC.java:487)
 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
 at org.jgroups.protocols.PING.up(PING.java:282)
 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
 at org.jgroups.protocols.TP.access$100(TP.java:53)
 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
2009-09-04 10:49:07,606 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-51501
2009-09-04 10:49:07,622 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:49:07,622 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:49:07,809 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:49:07,809 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
2009-09-04 10:49:07,809 ERROR [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Caught while requesting or applying state
org.infinispan.statetransfer.StateTransferException: Provider cannot provide state!
 at org.infinispan.statetransfer.StateTransferManagerImpl.applyState(StateTransferManagerImpl.java:313)
 at org.infinispan.remoting.InboundInvocationHandlerImpl.applyState(InboundInvocationHandlerImpl.java:74)
 at org.infinispan.remoting.transport.jgroups.JGroupsTransport.setState(JGroupsTransport.java:473)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.handleUpEvent(MessageDispatcher.java:803)
 at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:863)
 at org.jgroups.JChannel.up(JChannel.java:1437)
 at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:812)
 at org.jgroups.protocols.pbcast.FLUSH.up(FLUSH.java:489)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.connectToStateProvider(STREAMING_STATE_TRANSFER.java:516)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.handleStateRsp(STREAMING_STATE_TRANSFER.java:456)
 at org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.up(STREAMING_STATE_TRANSFER.java:226)
 at org.jgroups.protocols.FRAG2.up(FRAG2.java:195)
 at org.jgroups.protocols.FC.up(FC.java:487)
 at org.jgroups.protocols.pbcast.GMS.up(GMS.java:915)
 at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:239)
 at org.jgroups.protocols.UNICAST.handleDataReceived(UNICAST.java:574)
 at org.jgroups.protocols.UNICAST.up(UNICAST.java:282)
 at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:724)
 at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)
 at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:205)
 at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:271)
 at org.jgroups.stack.Protocol.up(Protocol.java:336)
 at org.jgroups.protocols.Discovery.up(Discovery.java:235)
 at org.jgroups.protocols.PING.up(PING.java:282)
 at org.jgroups.protocols.TP.passMessageUp(TP.java:996)
 at org.jgroups.protocols.TP.access$100(TP.java:53)
 at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1502)
 at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1479)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 at java.lang.Thread.run(Thread.java:595)
2009-09-04 10:49:07,809 WARN [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Could not find available peer for state, backing off and retrying
2009-09-04 10:49:17,419 INFO [org.infinispan.remoting.rpc.RpcManagerImpl] (main) Trying to fetch state from osoz-fd029efaa5-60337
2009-09-04 10:49:17,419 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-2,esbopen-3341) Received state for cache named 'org.infinispan.manager.DefaultCacheManager.DEFAULT_CACHE_NAME'. Attempting to apply state.
2009-09-04 10:49:17,419 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Applying state
2009-09-04 10:49:17,575 TRACE [org.infinispan.marshall.VersionAwareMarshaller] (Incoming-2,esbopen-3341) Read version 400
2009-09-04 10:49:17,575 DEBUG [org.infinispan.statetransfer.StateTransferManagerImpl] (Incoming-2,esbopen-3341) Provider cannot provide state!
2009-09-04 10:49:17,575 ERROR

5. Re: How to set up infinispan over LAN.

galder.zamarreno Sep 7, 2009 6:40 AM (in response to mcmati)

What Infinispan version are you using? Beta1?

Let's start with the second option tried. I'm a bit confused with the logs attached cos they show different times (one starts at 2009-09-04 10:48:56,684 and the other at 2009-09-04 12:49:53,218). Could you please check whether clocks are synchronized and then try the test again? Please send me the logs to: galder.zamarreno@jboss.com

It does appear though that the new node starting up didn't like the state that was provided:

Caused by: java.lang.ClassCastException: java.util.HashMap
at org.infinispan.statetransfer.StateTransferManagerImpl.applyInMemoryState(StateTransferManagerImp
l.java:328)

Had you put any data into the local node before you started the remote node (esbopen-3341)?

For the 1st option, esbopen-9704 hasn't replied but can't see logs for this machine. So, could you please repeat this tests and send me the logs for *all* machines? Make sure the clocks are synchronized.

Finally, can you make sure you have -Djava.net.preferIPv4Stack=true system property when starting these Infinispan nodes?
Actions
6. Re: How to set up infinispan over LAN.

mcmati Sep 7, 2009 8:30 AM (in response to mcmati)

OK, I send you the e-mail, and here are answers for those questions:

- infinispan beta1
- clocks are synchronized now, but still somehow there is 2 hour difference in logs.
- I start my app with: -Djava.net.preferIPv4Stack=true
- I don't remember if I have put any data into the local node before I started the remote one.

best regards,
Martin
Actions
7. Re: How to set up infinispan over LAN.

mcmati Sep 7, 2009 8:49 AM (in response to mcmati)

I would like to correct my previous post. I'm not using beta1 but alpha5. I will try beta1 now.
Actions
8. Re: How to set up infinispan over LAN.

renzos Jul 12, 2011 3:41 AM (in response to mcmati)

Good morning guys,
I'm using Infinispan 4.2.1 FINAL and I'm experiencing problems with Node Discovery over LAN as well. I've tried with 2 and 3 nodes over a 192.168.0.x LAN, I've no firewalls running and I enabled logging with the log4j library. I can see that each View is gonna not to be updated with the other nodes when they try to join the cluster.
This is how I declare the cluster configuration in each node:

GlobalConfiguration gc = GlobalConfiguration.getClusteredDefault();
Configuration c = new Configuration();
Properties p = new Properties();
p.setProperty("configurationFile", "jgroups-udp.xml");
gc.setTransportProperties(p);

any help would be appreciated, thanks.
Actions
9. Re: How to set up infinispan over LAN.

pmuir Jul 12, 2011 6:44 AM (in response to renzos)

Renzo, please start a new thread for a new problem!
Actions

Go to original post