Debugging serialization errors
Hi All,
I am just getting started with infinispan, during testing of a JBoss cluster I am getting the following exception during state-transfer between two nodes:
09:00:43,618 ERROR [org.infinispan.statetransfer.OutboundTransferTask] (transport-thread-17) Failed to send entries to node ip-123-12-3-156.eu-west-1.compute.internal/ejb : java.lang.RuntimeException: Failure to marshal argument(s): org.infinispan.CacheException: java.lang.RuntimeException: Failure to marshal argument(s)
at org.infinispan.util.Util.rewrapAsCacheException(Util.java:542)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:169)
at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:190)
at org.infinispan.statetransfer.OutboundTransferTask.sendEntries(OutboundTransferTask.java:257)
at org.infinispan.statetransfer.OutboundTransferTask.run(OutboundTransferTask.java:187)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.6.0_34]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_34]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_34]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.6.0_34]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.6.0_34]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.6.0_34]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) [rt.jar:1.6.0_34]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.6.0_34]
at java.lang.Thread.run(Thread.java:701) [rt.jar:1.6.0_34]
Caused by: java.lang.RuntimeException: Failure to marshal argument(s)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:281)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:300)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
... 14 more
Caused by: java.lang.IllegalStateException: Cannot serialize a session while connected
at org.hibernate.internal.SessionImpl.writeObject(SessionImpl.java:2116)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.6.0_34]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.6.0_34]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_34]
at java.lang.reflect.Method.invoke(Method.java:622) [rt.jar:1.6.0_34]
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:180)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1007)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:838)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:998)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1063)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1019)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.river.BlockMarshaller.doWriteObject(BlockMarshaller.java:69)
at org.jboss.marshalling.river.BlockMarshaller.writeObject(BlockMarshaller.java:60)
at org.jboss.marshalling.MarshallerObjectOutputStream.writeObjectOverride(MarshallerObjectOutputStream.java:54)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:343) [rt.jar:1.6.0_34]
at org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberImpl.writeObject(SerializationGroupMemberImpl.java:383)
at sun.reflect.GeneratedMethodAccessor953.invoke(Unknown Source) [:1.6.0_34]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.6.0_34]
at java.lang.reflect.Method.invoke(Method.java:622) [rt.jar:1.6.0_34]
at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:180)
at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1007)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:885)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at org.jboss.as.clustering.SimpleMarshalledValue.getBytes(SimpleMarshalledValue.java:85)
at org.jboss.as.clustering.SimpleMarshalledValue.writeExternal(SimpleMarshalledValue.java:175)
at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:47)
at org.jboss.as.clustering.infinispan.io.ExternalizableExternalizer.writeObject(ExternalizableExternalizer.java:36)
at org.infinispan.marshall.jboss.ExternalizerTable$ForeignExternalizerAdapter.writeObject(ExternalizerTable.java:457)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:154)
at org.infinispan.container.entries.ImmortalCacheEntry$Externalizer.writeObject(ImmortalCacheEntry.java:150)
at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48)
at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50)
at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45)
at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:103)
at org.infinispan.statetransfer.StateChunk$Externalizer.writeObject(StateChunk.java:88)
at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at org.infinispan.marshall.MarshallUtil.marshallCollection(MarshallUtil.java:48)
at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:50)
at org.infinispan.marshall.exts.ArrayListExternalizer.writeObject(ArrayListExternalizer.java:45)
at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at org.infinispan.marshall.exts.ReplicableCommandExternalizer.writeCommandParameters(ReplicableCommandExternalizer.java:87)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.marshallParameters(CacheRpcCommandExternalizer.java:128)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:112)
at org.infinispan.marshall.exts.CacheRpcCommandExternalizer.writeObject(CacheRpcCommandExternalizer.java:73)
at org.infinispan.marshall.jboss.ExternalizerTable$ExternalizerAdapter.writeObject(ExternalizerTable.java:408)
at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:145)
at org.jboss.marshalling.AbstractObjectOutput.writeObject(AbstractObjectOutput.java:62)
at org.jboss.marshalling.AbstractMarshaller.writeObject(AbstractMarshaller.java:119)
at org.infinispan.marshall.jboss.AbstractJBossMarshaller.objectToObjectStream(AbstractJBossMarshaller.java:96)
at org.infinispan.marshall.VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:92)
at org.infinispan.marshall.AbstractMarshaller.objectToBuffer(AbstractMarshaller.java:64)
at org.infinispan.marshall.AbstractDelegatingMarshaller.objectToBuffer(AbstractDelegatingMarshaller.java:109)
at org.infinispan.remoting.transport.jgroups.MarshallerAdapter.objectToBuffer(MarshallerAdapter.java:45)
at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.marshallCall(CommandAwareRpcDispatcher.java:279)
... 16 more
Caused by: an exception which occurred:
in field session
in field delegate
in field entityManager
in field instance
in field instance
in object org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberImpl@45017498
in object org.jboss.as.clustering.SimpleMarshalledValue@45017498
in object org.infinispan.container.entries.ImmortalCacheEntry@a02e9300
in object java.util.ArrayList@1b1fde1f
in object org.infinispan.statetransfer.StateChunk@5bb05bd9
in object java.util.ArrayList@5bb05bf8
in object org.infinispan.statetransfer.StateResponseCommand@c347eb1
I can see from the bottom of the exception that the culprit is the hibernate session (or an ArrayList stored inside it?)
Caused by: java.lang.IllegalStateException: Cannot serialize a session while connected
Does this refer to the HibernateSession? Does this mean that I cannot perform state transfer while there is an active hibernate session? Or do I need to track down the exact object inside the session that is causing the serialization error?
Any pointers appreciated