0 Replies Latest reply on Mar 18, 2015 5:28 AM by David Baddeley

    Debugging serialization errors

    David Baddeley Newbie

      Hi All,

       

      I am just getting started with infinispan, during testing of a JBoss cluster I am getting the following exception during state-transfer between two nodes:

       

      09:00:43,618 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread-17) Failed to send entries to node ip-123-12-3-156.eu-west-1.compute.internal/ejb : java.lang.RuntimeException: Failure to marshal argument(s): org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)

        at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)

        at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)

        at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)

        at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:257)

        at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:187)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.6.0_34]

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_34]

        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_34]

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.6.0_34]

        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_34]

        at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_34]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [rt.jar:1.6.0_34]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.6.0_34]

        at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_34]

      Caused by: java.lang.RuntimeException: Failure to marshal argument(s)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:281)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)

        ... 14 more

      Caused by: java.lang.IllegalStateException: Cannot serialize a session while connected

        at org.hibernate.internal.SessionImpl.writeObject(SessionImpl.java:2116)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_34]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.6.0_34]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_34]

        at java.lang.reflect.Method.invoke(Method.java:622) [rt.jar:1.6.0_34]

        at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:180)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1007)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:838)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:69)

        at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:60)

        at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:54)

        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:343) [rt.jar:1.6.0_34]

        at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberImpl.writeObject(SerializationGroupMemberImpl.java:383)

        at sun.reflect.GeneratedMethodAccessor953.invoke(Unknown Source) [:1.6.0_34]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_34]

        at java.lang.reflect.Method.invoke(Method.java:622) [rt.jar:1.6.0_34]

        at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:180)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1007)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.jboss.as.clustering.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:85)

        at org.jboss.as.clustering.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:175)

        at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:47)

        at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:36)

        at org.infinispan.marshall.jboss.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:457)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:154)

        at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:150)

        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48)

        at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50)

        at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45)

        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:103)

        at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:88)

        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48)

        at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50)

        at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45)

        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:87)

        at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128)

        at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112)

        at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:73)

        at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)

        at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)

        at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)

        at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)

        at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96)

        at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92)

        at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:64)

        at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:109)

        at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45)

        at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:279)

        ... 16 more

      Caused by: an exception which occurred:

        in field session

        in field delegate

        in field entityManager

        in field instance

        in field instance

        in object org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberImpl@45017498

        in object org.jboss.as.clustering.SimpleMarshalledValue@45017498

        in object org.infinispan.container.entries.ImmortalCacheEntry@a02e9300

        in object java.util.ArrayList@1b1fde1f

        in object org.infinispan.statetransfer.StateChunk@5bb05bd9

        in object java.util.ArrayList@5bb05bf8

        in object org.infinispan.statetransfer.StateResponseCommand@c347eb1

       

      I can see from the bottom of the exception that the culprit is the hibernate session (or an ArrayList stored inside it?)

       

      Caused by: java.lang.IllegalStateException: Cannot serialize a session while connected

       

      Does this refer to the HibernateSession? Does this mean that I cannot perform state transfer while there is an active hibernate session? Or do I need to track down the exact object inside the session that is causing the serialization error?

       

      Any pointers appreciated