Object stream corruption leading to out of memory error in the CacheMarshaller
All,
We have been using the jboss cache 3.2.5 since the past three years on our production servers with no major issues. Of late we have seen issues from two of our customer sites where the application container running the jboss cache crashed with an out of memory error. We analyzed the heap dumps from both sites and found the root cause to be the same. The cache tried to serialize an object for replication and the resulting byte array was more than 800 MB in size. Our deployment consists of two servers running on RHEL 5.5 forming two nodes of the cache, replication mode async. We further analyzed the byte stream and found out that after a certain point it is filled with a repeated pattern that continues forever. I am enclosing a portion of the byte array (below) where you can see the repeated pattern. The object that it tried to serialize was itself small in size as seen from the heap dump. Things to note are we do not synchronize when updating the object and putting it in the cache, the object has a HashSet and several Enums and we do not explicitly start transactions. We believe the issue is because of concurrent access to the HashSet while the cache was writing it to the byte array but not sure. The issue is also not reproducible easily.
The stack of the thread that crashed the JVM with out of memory error is also below -->
- java.lang.OutOfMemoryError: allocLargeObjectOrArray: [B, size 888178424
at jrockit/vm/Allocator.allocLargeObjectOrArray(JIZ)Ljava/lang/Object;(Native Method)
at jrockit/vm/Allocator.allocObjectOrArray(Allocator.java:349)[optimized]
at org/jboss/cache/io/ExposedByteArrayOutputStream.write(ExposedByteArrayOutputStream.java:113)[optimized]
at java/io/ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1847)[inlined]
at java/io/ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1756)[optimized]
at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1169)[inlined]
at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:330)[optimized]
at java/util/HashSet.writeObject(HashSet.java:267)
at sun/reflect/GeneratedMethodAccessor210.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[inlined]
at java/io/ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:940)[inlined]
at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1469)[optimized]
at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)[optimized]
at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)[optimized]
at java/io/ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1518)[inlined]
at java/io/ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1483)[optimized]
at java/io/ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1400)[inlined]
at java/io/ObjectOutputStream.writeObject0(ObjectOutputStream.java:1158)[inlined]
at java/io/ObjectOutputStream.writeObject(ObjectOutputStream.java:330)[optimized]
at org/jboss/cache/marshall/CacheMarshaller200.marshallObject(CacheMarshaller200.java:460)
at org/jboss/cache/marshall/CacheMarshaller300.marshallObject(CacheMarshaller300.java:47)
at org/jboss/cache/marshall/CacheMarshaller200.marshallCommand(CacheMarshaller200.java:519)
at org/jboss/cache/marshall/CacheMarshaller200.marshallObject(CacheMarshaller200.java:314)
at org/jboss/cache/marshall/CacheMarshaller300.marshallObject(CacheMarshaller300.java:47)
at org/jboss/cache/marshall/CacheMarshaller200.marshallCommand(CacheMarshaller200.java:519)
at org/jboss/cache/marshall/CacheMarshaller200.marshallObject(CacheMarshaller200.java:314)
at org/jboss/cache/marshall/CacheMarshaller300.marshallObject(CacheMarshaller300.java:47)
at org/jboss/cache/marshall/CacheMarshaller200.objectToObjectStream(CacheMarshaller200.java:191)
at org/jboss/cache/marshall/CacheMarshaller200.objectToObjectStream(CacheMarshaller200.java:136)
at org/jboss/cache/marshall/VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:183)
at org/jboss/cache/marshall/VersionAwareMarshaller.objectToBuffer(VersionAwareMarshaller.java:52)
at org/jboss/cache/marshall/CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:383)
at org/jboss/cache/marshall/CommandAwareRpcDispatcher$ReplicationTask.call(CommandAwareRpcDispatcher.java:355)
at org/jboss/cache/util/concurrent/WithinThreadExecutor.submit(WithinThreadExecutor.java:82)
at org/jboss/cache/marshall/CommandAwareRpcDispatcher.invokeRemoteCommands(CommandAwareRpcDispatcher.java:210)
at org/jboss/cache/RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:744)
at org/jboss/cache/RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:712)
at org/jboss/cache/RPCManagerImpl.callRemoteMethods(RPCManagerImpl.java:717)
at org/jboss/cache/interceptors/BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:161)[inlined]
at org/jboss/cache/interceptors/BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:135)[inlined]
at org/jboss/cache/interceptors/BaseRpcInterceptor.replicateCall(BaseRpcInterceptor.java:107)[inlined]
at org/jboss/cache/interceptors/ReplicationInterceptor.handleCrudMethod(ReplicationInterceptor.java:160)[optimized]
at org/jboss/cache/interceptors/ReplicationInterceptor.visitPutKeyValueCommand(ReplicationInterceptor.java:107)
at org/jboss/cache/commands/write/PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)[optimized]
at org/jboss/cache/interceptors/base/CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)[optimized]
at org/jboss/cache/interceptors/base/CommandInterceptor.handleDefault(CommandInterceptor.java:131)
at org/jboss/cache/commands/AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:65)[optimized]
at org/jboss/cache/commands/write/PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)[optimized]
at org/jboss/cache/interceptors/base/CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)[inlined]
at org/jboss/cache/interceptors/TxInterceptor.attachGtxAndPassUpChain(TxInterceptor.java:301)[inlined]
at org/jboss/cache/interceptors/TxInterceptor.handleDefault(TxInterceptor.java:283)[optimized]
at org/jboss/cache/commands/AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:65)[optimized]
at org/jboss/cache/commands/write/PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)[optimized]
at org/jboss/cache/interceptors/base/CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)[inlined]
at org/jboss/cache/interceptors/CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:119)[optimized]
at org/jboss/cache/commands/write/PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)[optimized]
at org/jboss/cache/interceptors/base/CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)[inlined]
at org/jboss/cache/interceptors/InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:178)[inlined]
at org/jboss/cache/interceptors/InvocationContextInterceptor.visitPutKeyValueCommand(InvocationContextInterceptor.java:82)[optimized]
at org/jboss/cache/commands/write/PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:100)[optimized]
at org/jboss/cache/interceptors/InterceptorChain.invoke(InterceptorChain.java:287)
at org/jboss/cache/invocation/CacheInvocationDelegate.put(CacheInvocationDelegate.java:555)
at org/jboss/cache/invocation/NodeInvocationDelegate.put(NodeInvocationDelegate.java:377)
A portion of the byte stream with corruption seen towards the end. The byte pattern that you see in the end continues to grow the array to > 800 MB in size.
0000690: 0005 4c00 0e63 6861 6e67 6554 7261 636b ..L..changeTrack
00006a0: 5365 7471 007e 0011 4c00 0c6c 6173 744d Setq.~..L..lastM
00006b0: 6f64 6966 6965 6474 0010 4c6a 6176 612f odifiedt..Ljava/
00006c0: 7574 696c 2f44 6174 653b 4c00 0a6d 7954 util/Date;L..myT
00006d0: 696d 655a 6f6e 6574 0014 4c6a 6176 612f imeZonet..Ljava/
00006e0: 7574 696c 2f54 696d 655a 6f6e 653b 7870 util/TimeZone;xp
00006f0: 0100 0000 0000 0000 0000 0000 0000 0000 ................
0000700: 0000 0000 0000 7400 0073 7200 116a 6176 ......t..sr..jav
0000710: 612e 7574 696c 2e48 6173 6853 6574 ba44 a.util.HashSet.D
0000720: 8595 96b8 b734 0300 0078 7077 0c00 0000 .....4...xpw....
0000730: 203f 4000 0000 0000 1174 0011 7265 6d6f ?@......t..remo
0000740: 7465 4469 7370 6c61 794e 616d 6574 0006 teDisplayNamet..
0000750: 7374 6174 7573 7100 7e00 1b74 0011 7072 statusq.~..t..pr
0000760: 6576 696f 7573 5265 6d6f 7465 5552 4974 eviousRemoteURIt
0000770: 0009 6469 7265 6374 696f 6e71 007e 001d ..directionq.~..
0000780: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
0000790: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
00007a0: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
00007b0: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
00007c0: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
00007d0: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
00007e0: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
00007f0: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
0000800: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
0000810: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
0000820: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
0000830: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
0000840: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
0000850: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
0000860: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
0000870: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
0000880: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
0000890: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
00008a0: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
00008b0: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
00008c0: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
00008d0: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
00008e0: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
00008f0: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
0000900: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
0000910: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
0000920: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
0000930: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
0000940: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
0000950: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
0000960: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
0000970: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
0000980: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
0000990: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
00009a0: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
00009b0: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
00009c0: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
00009d0: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
00009e0: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
00009f0: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
0000a00: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
0000a10: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
0000a20: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
0000a30: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
0000a40: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
0000a50: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
0000a60: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
0000a70: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
0000a80: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
0000a90: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
0000aa0: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
0000ab0: 007e 001d 7100 7e00 1d71 007e 001d 7100 .~..q.~..q.~..q.
0000ac0: 7e00 1d71 007e 001d 7100 7e00 1d71 007e ~..q.~..q.~..q.~
0000ad0: 001d 7100 7e00 1d71 007e 001d 7100 7e00 ..q.~..q.~..q.~.
0000ae0: 1d71 007e 001d 7100 7e00 1d71 007e 001d .q.~..q.~..q.~..
0000af0: 7100 7e00 1d71 007e 001d 7100 7e00 1d71 q.~..q.~..q.~..q
.........and so on
Has anyone seen this type of an issue in the marshaller? Is it correct to assume that this is the result of concurrent modification to the HashSet in the object? Any insights will help.
Thanks
Mukund.