Sometimes occurring: DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException

[tedd824]

Hi~,

Sometimes get operation occurring errors below.

org.infinispan.client.hotrod.impl.protocol.Codec13 - ISPN004005: Error received from the server: io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException

Sample data is stored normally.

But sometimes error occurred.

And retry is successfully.

I don't get it why it happens and what is the cause?

Any help is greatly appreciated.

Environments:

JDK: jdk1.7.0_51

Infinispan.version: 7.0.0.Alpha2

Infinispan Server Hotrod cluster configuration.

Exception:

org.infinispan.client.hotrod.impl.protocol.Codec13 - ISPN004005: Error received from the server: io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException

org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[8164] returned server error (status=0x85): io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException

  at org.infinispan.client.hotrod.impl.protocol.Codec10.checkForErrorsInResponseStatus(Codec10.java:143)

  at org.infinispan.client.hotrod.impl.protocol.Codec10.readHeader(Codec10.java:99)

  at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)

  at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.sendKeyOperation(AbstractKeyOperation.java:53)

  at org.infinispan.client.hotrod.impl.operations.GetOperation.executeOperation(GetOperation.java:28)

  at org.infinispan.client.hotrod.impl.operations.GetOperation.executeOperation(GetOperation.java:17)

  at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:49)

  at org.infinispan.client.hotrod.impl.RemoteCacheImpl.get(RemoteCacheImpl.java:424)

[rvansa]

Hi, we need the stack trace from server log to see where exactly this exception happened.

[tedd824]

Thanks Radim.

17:14:28,913 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:10290/management

17:14:28,914 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:10290

17:14:28,914 INFO  [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss Infinispan Server 7.0.0.Alpha2 (AS 7.2.0.Final) started in 9195ms - Started 87 of 346 services (259 services are passive or on-demand)

17:17:58,574 ERROR [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) ISPN005009: Unexpected error before any request parameters read: io.netty.handler.codec.DecoderException: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry

  at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:417) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:139) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at org.infinispan.server.core.AbstractProtocolDecoder.channelRead(AbstractProtocolDecoder.scala:398) [infinispan.jar:7.0.0.Alpha2]

  at io.netty.channel.DefaultChannelHandlerContext.invokeChannelRead(DefaultChannelHandlerContext.java:338) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.DefaultChannelHandlerContext.fireChannelRead(DefaultChannelHandlerContext.java:324) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:785) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:132) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:485) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:452) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:346) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:101) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

Caused by: org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry

  at org.infinispan.server.hotrod.HotRodDecoder.createServerException(HotRodDecoder.scala:192) [infinispan.jar:7.0.0.Alpha2]

  at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:61) [infinispan.jar:7.0.0.Alpha2]

  at io.netty.handler.codec.ReplayingDecoder.callDecode(ReplayingDecoder.java:362) [netty-all-4.0.15.Final.jar:4.0.15.Final]

  ... 11 more

Caused by: java.lang.ClassCastException: [B cannot be cast to org.infinispan.container.entries.CacheEntry

  at org.infinispan.CacheImpl.getCacheEntry(CacheImpl.java:394) [infinispan-core.jar:7.0.0.Alpha2]

  at org.infinispan.CacheImpl.getCacheEntry(CacheImpl.java:399) [infinispan-core.jar:7.0.0.Alpha2]

  at org.infinispan.AbstractDelegatingAdvancedCache.getCacheEntry(AbstractDelegatingAdvancedCache.java:185) [infinispan-core.jar:7.0.0.Alpha2]

  at org.infinispan.server.core.AbstractProtocolDecoder.get(AbstractProtocolDecoder.scala:261) [infinispan.jar:7.0.0.Alpha2]

  at org.infinispan.server.core.AbstractProtocolDecoder.decodeKey(AbstractProtocolDecoder.scala:100) [infinispan.jar:7.0.0.Alpha2]

  at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:55) [infinispan.jar:7.0.0.Alpha2]

  ... 12 more

[rvansa]

Interesting, and really strange. Do you think you could reproduce that with TRACE logging level on org.infinispan? I am not sure how many servers do you use, but as I suspect there's some code path which is not executed very often (and that's why you get this exception only once in a while), this relates to distribution -> logs from all servers are needed to trace that down. And these will be huge - probably several hundreds MB, therefore, please upload them zipped to DropBox or similar place.