7 Replies Latest reply on Sep 26, 2012 2:58 PM by jchildress2

    Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?

    abelinschi

      Infinispan version 5.1.4.Final

       

      Using a cluster of 6 nodes in unicast tcp mode configured through Jgroups 2.0.9 configuration and a HotRod server that joins the cluster successfully on start up using also unicast tcp.

      After the cluster is formed correctly, using a client Java application I am trying to 'put' data in default cache in the cluster but getting the following errors in one of cluster nodes immediately after issuing the 'put("key", "value")':


      WARN: Problems unmarshalling remote command from byte buffer

      org.infinispan.CacheException: Type of data read is unknown. Id=-2147482548 is not amongst known reader indexes.

              at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:216)

              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

              at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

              at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)

              at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:107)

              at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readObject(ReplicableCommandExternalizer.java:58)

              at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391)

              at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)

              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

              at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

              at org.infinispan.marshall.exts.ReplicableCommandExternalizer.readParameters(ReplicableCommandExternalizer.java:119)

              at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:163)

              at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.readObject(CacheRpcCommandExternalizer.java:67)

              at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.readObject(ExternalizerTable.java:391)

              at org.infinispan.marshall.jboss.ExternalizerTable.readObject(ExternalizerTable.java:222)

              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:351)

              at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

              at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:37)

              at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:154)

              at org.infinispan.marshall.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:114)

              at org.infinispan.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:85)

              at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectFromBuffer(MarshallerAdapter.java:50)

              at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:200)

              at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)

              at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)

              at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)

              at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)

              at org.jgroups.JChannel.up(JChannel.java:716)

              at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)

              at org.jgroups.protocols.RSVP.up(RSVP.java:192)

              at org.jgroups.protocols.FRAG2.up(FRAG2.java:181)

              at org.jgroups.protocols.FlowControl.up(FlowControl.java:418)

              at org.jgroups.protocols.FlowControl.up(FlowControl.java:400)

              at org.jgroups.protocols.pbcast.GMS.up(GMS.java:889)

              at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:244)

              at org.jgroups.protocols.UNICAST2.handleDataReceived(UNICAST2.java:759)

              at org.jgroups.protocols.UNICAST2.up(UNICAST2.java:365)

              at org.jgroups.protocols.pbcast.NAKACK.up(NAKACK.java:602)

              at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:143)

              at org.jgroups.protocols.FD.up(FD.java:273)

              at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:288)

              at org.jgroups.protocols.MERGE2.up(MERGE2.java:205)

              at org.jgroups.protocols.Discovery.up(Discovery.java:359)

              at org.jgroups.protocols.TP.passMessageUp(TP.java:1180)

              at org.jgroups.protocols.TP$IncomingPacket.handleMyMessage(TP.java:1728)

              at org.jgroups.protocols.TP$IncomingPacket.run(TP.java:1710)

              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

              at java.lang.Thread.run(Thread.java:619)

       

      Any idea why I am getting the error?

       

      My HotRod client reports the following error:

       

      May 10, 2012 4:19:28 PM org.infinispan.client.hotrod.impl.protocol.Codec10 checkForErrorsInResponseStatus

      WARN: ISPN004005: Error received from the server: org.infinispan.CacheException: Problems invoking command.

      Exception in thread "main" org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[4] returned server error (status=0x85): org.infinispan.CacheException: Problems invoking command.

          at org.infinispan.client.hotrod.impl.protocol.Codec10.checkForErrorsInResponseStatus(Codec10.java:156)

          at org.infinispan.client.hotrod.impl.protocol.Codec10.readHeader(Codec10.java:109)

          at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:78)

          at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:72)

          at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:52)

          at org.infinispan.client.hotrod.impl.operations.PutOperation.executeOperation(PutOperation.java:41)

          at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:68)

          at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:220)

          at org.infinispan.CacheSupport.put(CacheSupport.java:53)

          at org.infinispan.quickstart.clusteredcache.distribution.HotRodClient.main(HotRodClient.java:24)

       

       

      Here is the HotRod client code:

       

      BasicCacheContainer cm = new RemoteCacheManager("myhostnam:11222", true);

      BasicCache<Object, Object> cache = cm.getCache();

      String value = cache.get("key"); // no errors

      cache.put("key", "value");  // errors out

       

      ...

       

       

      Any suggestions appreciated.

       

      thanks

        • 1. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
          galder.zamarreno

          No idea. Are you sure it's JGroups 2.0.9? Shouldn't be 3.x?

           

          Maybe each cluster server might be using different jars.

           

          I'd suggest you build a test case and attach it here.

          • 2. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
            abelinschi

            I made a typo, I was going to write that it is using JGroups 3.0.9

            • 3. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
              galder.zamarreno

              Thought that might be the case . So, can you build a test case for us? This is a weird scenario.

              • 4. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
                abelinschi

                I have just downloaded Infinispan 5.1.5.CR1 and using new jars for testing the cluster with a simple HotRod client (the code shown in previous posts). It looks like the logic is working now. Here is what I am running:

                 

                1. An Infinispan cluster of two nodes (different hosts running Solaris) clustered using TCP unicast (with TCPPING enabled) with a pre-defined list of hosts + a HotRod server (running an OpenSuse box) that joins the cluster also in TCP unicast mode (total three members in the cluster running on different hosts). The cluster forms correctly without any errors.
                2. A simple HotRod simple testing code running on Win7 64 bits. However when running the HotRod client, the HotRod server is throwing the following errors (the client doesn't report any errors and still reads and updates the cache successfully):

                 

                2012-05-15 15:05:27,583 ERROR [HotRodDecoder] (HotRodServerWorker-1-1) ISPN005009: Unexpected error before any request parameters read

                1. java.io.IOException: Connection reset by peer

                        at sun.nio.ch.FileDispatcher.read0(Native Method)

                        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)

                        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251)

                        at sun.nio.ch.IOUtil.read(IOUtil.java:218)

                        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254)

                        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:323)

                        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)

                        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)

                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

                        at java.lang.Thread.run(Thread.java:679)

                2012-05-15 15:05:27,585 ERROR [HotRodDecoder] (HotRodServerWorker-1-1) ISPN005009: Unexpected error before any request parameters read

                1. java.io.IOException: Broken pipe

                        at sun.nio.ch.FileDispatcher.write0(Native Method)

                        at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)

                        at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)

                        at sun.nio.ch.IOUtil.write(IOUtil.java:78)

                        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)

                        at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:240)

                        at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:472)

                        at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:390)

                        at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)

                        at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)

                        at org.jboss.netty.channel.Channels.write(Channels.java:632)

                        at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)

                        at org.jboss.netty.channel.Channels.write(Channels.java:611)

                        at org.jboss.netty.channel.Channels.write(Channels.java:578)

                        at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)

                        at org.infinispan.server.core.AbstractProtocolDecoder.exceptionCaught(AbstractProtocolDecoder.scala:279)

                        at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)

                        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:333)

                        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)

                        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)

                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

                        at java.lang.Thread.run(Thread.java:679)

                 

                 

                 

                 

                 

                 

                 

                Let me know if you need more details. The error is generated each time I run the HotRod client Java app.

                 

                -Andrian

                • 5. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
                  abelinschi

                  It looks like if I use 'cm.close()' before exisiting the client app then all is good, no more errors on HotRod server.

                  • 6. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
                    jchildress2

                    Andrian,

                     

                    I'm getting the same error and seem to have a similar setup to yours.

                     

                    I don't see where cm.close() is available in the API....  I'm using 5.1.6

                     

                    thanks,

                     

                    John

                    • 7. Re: Errors using Infinispan unicast cluster, HotRod server and a HotRod client - any idea?
                      jchildress2

                      It looks like stop() takes care of it...