Hi,
I am using infinispan's distributed execution service where I execute a callable on all the nodes of infinispan and the data returned from each node can be quite big. Recently, I have noticed that when the size of the serialized object starts going close to Integer.MAX_VALUE, the process of serialization gets painfully slower. This happens because in the method org.infinispan.io.ExposedByteArrayOutputStream.getNewBufferSize, when the value of
curSize + (curSize >> 2) exceeds Integer.MAX_VALUE, then the increment in size is only one byte. So, to write each new byte, a new buffer is initialized and the content of older buffer is copied to newer buffer. Although, this can be handled to allocate a buffer of size Integer.MAX_VALUE, but still doing this would not solve my problem as the object size can grow even more than Integer.MAX_VALUE.
So, my question is:
Is it possible to send responses close to 4-5GBs using inifinispan's distributed execution framework. I am open to suggestions including writing my own Marshallers but looking at the code, it seems unlikely that rest of the framework would be able to handle if I serialize my object into multiple bytearrays.
Also, if it does not currently support it, could you please suggest the changes in infinispan code which can make it possible to achieve this behaviour with minimal effort, I would like to add this functionality and contribute back if possible. I was thinking of sending multiple response messages for the same request messages and somehow joining them at the calling node. Does it look doable?
Regards,
Ashish
PS: I am still on infinispan-5.0.0.Final, but I doubt there are any solutions to this issue even in latest releases.
