1 Reply Latest reply on Jul 31, 2012 9:59 AM by cmilu

Infinispan cluster and hotrod client are not communicating

cmilu Jul 31, 2012 4:23 AM

Hello,

I have a replication cluster whit a couple of nodes - they can communicate freely between each other. I also have a Hotrod Server (provided with Infinispan) joining the cluster (I can see that its recognized by the other nodes). But when I connect a Hotrod Client (RemoteCacheManager) to the server, and try to put something into the cache, I get an exception on:

the hotrod client: org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[4] returned server error (status=0x85): org.infinispan.CacheException: Problems invoking command.
the hotrod server: Caused by: org.infinispan.CacheException: Type of data read is unknown. Id=-2147482548 is not amongst known reader indexes.
a node with a cachestore: WARN: Problems unmarshalling remote command from byte buffer org.infinispan.CacheException: Type of data read is unknown. Id=-2147482548 is not amongst known reader indexes.

The data is not saved, and also the Hotrod client doesn't see any of the changes provided by other nodes. If there aren't any nodes in the cluster besides the hotrod server - everything works. If any other node (for example with a CacheListener registered) is available, I get the mentioned exceptions.

I use Infinispan 5.1.5.Final and JGroups 3.0.10.Final everywhere. Win7 64 Bit, Oracle Java 6_0_32 32bit. All nodes are working on one machine, and the cluster is built on TCP transport stack. Every node is a Java SE application. There was a similar post on the infinispan forum (https://community.jboss.org/thread/199539) but its for an older version (the update to 5.1.5 helped in that case).

Someone had a similar issue, or has any idea where I should look for the source of the problem?

I already posted the problem at stackoverflow, but it got very little attention there, so I tought I'll try here.

Stack trace from the hotrod server.

2012-07-30 17:03:09,308 TRACE (HotRodClientMaster-1) [org.infinispan.remoting.rpc.RpcManagerImpl] replication exception:

org.infinispan.CacheException: Problems invoking command.

at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:215)

at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)

at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)

at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)

at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:601)

at org.jgroups.JChannel.up(JChannel.java:716)

at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)

at org.jgroups.protocols.RSVP.up(RSVP.java:192)

at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)

at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)

at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)

at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:383)

at org.jgroups.protocols.pbcast.NAKACK.handleMessage(NAKACK.java:706)

at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:566)

at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)

at org.jgroups.protocols.FD.up(FD.java:273)

at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)

at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

at org.jgroups.protocols.Discovery.up(Discovery.java:359)

at org.jgroups.protocols.MPING.up(MPING.java:179)

at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)

at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)

at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

Caused by: org.infinispan.CacheException: Type of data read is unknown. Id=-2147482548 is not amongst known reader indexes.

at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:218)

at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)

at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107)

at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58)

at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:395)

at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224)

at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)

at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:162)

at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:66)

at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:395)

at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:224)

at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:163)

at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114)

at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:104)

at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)

at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200)

... 27 more

2012-07-30 17:03:09,358 TRACE (HotRodClientMaster-1) [org.infinispan.util.concurrent.locks.LockManagerImpl] Attempting to unlock ByteArrayKey{data=ByteArray{size=6, hashCode=fe0fd9, array=0x033e0368656a..}}

2012-07-30 17:03:09,368 TRACE (HotRodClientMaster-1) [org.infinispan.statetransfer.StateTransferLockImpl] Released shared state transfer shared lock, remaining holders: 0

2012-07-30 17:03:09,368 ERROR (HotRodClientMaster-1) [org.infinispan.interceptors.InvocationContextInterceptor] ISPN000136: Execution error