Hi Roman, Infinispan 5.3 is more than 2 years old release, so we don't fix (neither investigate thoroughly) bugs there. However, we would be very keen to fix any memory leaks if you are able to reproduce them on Infinispan 8.
My guess is that the maps you're seeing is the marshallers' instanceCache, because the marshallers are cached in a thread-local variable. Maybe you could cut down on the size of your thread pools, or allow idle worker threads to shut down after 60 seconds?
Like Radim said, we're not going to fix the problem in 5.3, but we'd like to hear back from you in case you can reproduce it with Infinispan 8. If you do, it would be good to attach a screenshot with the incoming references to the IdentityIntMap, to clarify if the per-thread marshallers cache is at fault.
finally we did an upgrade to Infinispan 8 on our productive system and sadly we have exact the same effect as we had on Infinispan 5.3. Any suggestions?