We have encountered a strange bug that seems to occassionally result in a cache being created with a connected JGroups channel but without receiving any state from the other cache instances it says it's connected to. It's important to note that in most circumstances we have no issues and the cache state is transferred successfully to the new cluster member.
We have roughly 38,000 entries in our cache (approximately 10-14MB serialized) and yet sometimes on startup, the "getCache()" method returns a cache without an exception but the cache contains no entries, while the other cache members contain the full ~38,000 entries. We are using REPL_ASYNC mode, which I don't think should affect this.
When launching 2 machines with the same binaries, we are sometimes seeing one machine boot normally and get a full copy of the cache while the other is left without any entries, although it states itself as connected in JMX.
We are using Infinispan 5.1.6.FINAL and JGroups 3.1.0.Final. I noticed that Infinispan 5.1.6.FINAL has a dependency on JGroups 3.0.11.Final, but I wouldn't think that that would cause the inconsistent behavior we are experiencing.
Any help would be greatly appreciated, or maybe this is a bug, as one would expect an exception to be thrown in any case where the cache state could not be received in the getCache() method.