OOME from too-large arrays are usually harmless to the JVM but it does underscore that it is essential to validate data read by externalizers or custom serialization implementations.
It looks like you're seeing stream corruption. This would be very unlikely to be caused by other Externalizable classes due to safety measures employed for those object types. The possible causes for corrupted data are, in order of decreasing likelihood:
- A bug in the user (java.io.)Externalizable class itself
- A bug in a ClassTable or ObjectTable implementation which is in use by Infinispan, which trade safety for speed (this would point to Infinispan or possibly a user class delegated to by Infinispan)
- A bug in JBoss Marshalling involving a problem containing the isolation between user stream data and internal stream data
Would it be possible to see the Externalizable class in question in a pastebin?
I have provided essentialized/simplifed versions of the relevant Externalizable Objects here:
Though much of the public interface that are simply accessors and are therefore irrelevant for mashalling have been stripped away, the writeExternal and readExternal methods have not been modified from the original. As you can see, the writeExternal/readExternal methods are pretty simple and I would be surprised to see a bug there. However, I'd be happy to be proven wrong .
Note the OOME is actually being thrown by org.jboss.marshalling.UTFUtils.readUTFBytes when MField.readExternal calls in.readUTF() -- UTFUtils.readUTFBytes reads the length of the String as an int value (in the last example I saw, it reads a length of 0x2400004E or 603,979,854 chars) and it therefore tries to allocate a char, which obviously fails.
Why does your class implement org.jgroups.util.Streamable as well? I don't see any reason for your class to be implementing that. You're already Externalizable and it's Infinispan that will be dealing with the serialization. I dunno what the effects of Streamable could be here but you do not want JGroups to be doing funny things with it trying to serialize your class again, so I'd try to leave it out and the implemented methods and try your test again.
Btw, it'd be really useful if you could provide us with some test/setup that we can run to try to replicate your issue. Would that be possible?
It implements Streamable because it is also used by some other code that sends these Objects around a different non-Infinispan JGroups cluster. I can certainly try removing it. However, if the Streamable interface were causing JGroups to corrupt the stream, wouldn't the problem happen consistently instead of rarely as it does now?
In any case, if that doesn't fix the problem, I will try to create a reproducible test case. However, as always with these types of "inconsistent" problems, that may not be possible.
Update: removing the Streamable interface did not help. Updating to Infinispan 5.0.0 CR4 and the associated marshalling library (1.3.0.CR9) also did not help.
I am trying to create a test case but so far have not been successful in reproducing the issue outside of the application.
SOLVED: Problem was due to unsafe threading. Using Infinispan putAsync, but Object that was being serialized was likely modified by another Thread after putAsync was called and during the Infinispan serialization. Adding the appropriate synchronization seems to have fixed it. Thanks a lot for the help.
Could someone create a FAQ around this? No one else has reported this as yet but it is concievably something that could affect others.