Problems with Infinispan as 2nd Level Hibernate Cache

[infinispanuser]

In a clustered environment with 4 nodes we use Infinispan as 2nd Level Hibernate Cache. We use Infinispan 8.2.5 and JGroups 3.6.7.

When there is much work load on one of the servers we already often got problems with the Infinispan cluster that suspects these node(s) and the nodes will never come back to the cluster.

Even the one clustered that was suspected and kicked out of the cluster forms his own cluster after some time which is never what we expected the behavior to be. Also replication fails with an replication timeout each time a node tries to replicate themself.

Another problem that we saw was that this also seems to lead to inconsistency between the infinispan cluster nodes and still it is not clear to us if there is anything else that will lead to problems in such a scenario.

What we saw and got were different cache entries on the different nodes, which is a big problem that is not acceptable in our case.

In the forum we already found this post https://developer.jboss.org/message/822198#822198 . It is still unanswered so does this mean that Infinispan is not able to be used as Hibernate 2nd Level Cache in a clustered environment?

Attached you will find our Infinispan configuration which is nearly the same as delivered with the official hibernate-infinispan jar (hibernate-infinispan-5.2.5.Final.jar\org\hibernate\cache\infinispan\builder\infinispan-configs.xml)

And also you can find our jGroups configuration file which also is nearly a copy of what is suggested by Infinispan in the core library (infinispan-core-8.2.5.Final.jar\default-configs\default-jgroups-tcp.xml)

It has to be mentioned that we are not able to use multicast in our environment, which is why suggested to use tcpping instead of mping in our stack.

Our main question is if there is anything wrong with our configuration and why does the default configuration example from Infinispan differ so much from the default tcp configuration suggested and delivered by jgroups (jgroups\jgroups-3.6.7.Final.jar\tcp.xml)

It seems that in the default Infinispan JGroups configuration file there is no state transfer protocol on top of the stack. Why is such a protocol missing in the stack which is available in the default tcp stack of jgroups?

Is it possible to use the default tcp stack from jgroups as delivered in their library or does this not work with the Hibernate-Infinispan implementation?

I hope someone can help us further with our questions so that we can still use Infinispan in our project(s)?

[galder.zamarreno]

infinispanuser  wrote:

In a clustered environment with 4 nodes we use Infinispan as 2nd Level Hibernate Cache. We use Infinispan 8.2.5 and JGroups 3.6.7.

When there is much work load on one of the servers we already often got problems with the Infinispan cluster that suspects these node(s) and the nodes will never come back to the cluster. Even the one clustered that was suspected and kicked out of the cluster forms his own cluster after some time which is never what we expected the behavior to be. Also replication fails with an replication timeout each time a node tries to replicate themself.

If a node is suspected when there's a lot of load, it's probably due to heavy garbage collection. We should try to give the JVMs more room to work with, or tweak GC settings. You should check your garbage collection logs, check size of JVM, get memory dumps to see what's taking memory...etc. You can also try to increase FD timeouts in JGroups stack. Even if kicked out, if MERGE3 protocol is present, it should eventually reunite.

infinispanuser  wrote:

Another problem that we saw was that this also seems to lead to inconsistency between the infinispan cluster nodes and still it is not clear to us if there is anything else that will lead to problems in such a scenario.

What we saw and got were different cache entries on the different nodes, which is a big problem that is not acceptable in our case.

In the forum we already found this post Is Clustered/Distributed Hibernate 2nd Level Cache possible? . It is still unanswered so does this mean that Infinispan is not able to be used as Hibernate 2nd Level Cache in a clustered environment?

Infinispan Hibernate 2LC is heavily used by many people. Wildfly/EAP includes it as default and many users use it clustered, so it's been heavily used for the past 8 years since its inception.

How have you arrived at the conclusion that each node has different data? It's unfortunate that this happens. I'm not aware of any specific issue right now, but it'd be good if you could replicate in a test environment, where you can get TRACE logs on org.hibernate and org.infinispan...etc.

infinispanuser  wrote:

Our main question is if there is anything wrong with our configuration and why does the default configuration example from Infinispan differ so much from the default tcp configuration suggested and delivered by jgroups (jgroups\jgroups-3.6.7.Final.jar\tcp.xml)

It seems that in the default Infinispan JGroups configuration file there is no state transfer protocol on top of the stack. Why is such a protocol missing in the stack which is available in the default tcp stack of jgroups?

Is it possible to use the default tcp stack from jgroups as delivered in their library or does this not work with the Hibernate-Infinispan implementation?

You should stick to the JGroups configuration provided by Infinispan itself. The one used by JGroups by default is not designed for the Infinispan use case.

[infinispanuser]

First of all thank you for your answer.

We came to the conclusion because we have had a scenario where there was different data on one of the nodes. On all other nodes for example we were able to edit a user in our web application and assign some role to him, but on the nodes with the data not being synchronized we were not able to save the data. Also even further, if the user got unassigned from one or more of its roles, he still had these roles on the "corrupt" nodes and therefore had still the oppertunity to edit thing that he wasn't allowed to edit anymore. The web application is a web application with angularjs Front-End and Spring Backend (Controllers -> BusinessLogic -> Hibernate for Persistence and Infinispan as 2nd Level Cache for Hibernate -> Oracle DB).

We also thought that the exiting MERGE3 should handle such situations for us but this seems not to be the case in our scenario. Is there anything else /any other information or tip/help you can give us?

Additionally ... in your opinion, which is the best source/manual/documentation in the web for the configuration and usage of Infinispan as a 2nd Level Cache in a clustered environment?

[infinispanuser]

Is there any update or help/advice you can give us regarding the questions from my last post/answer?

[galder.zamarreno]

Not really, you need to figure out first why nodes get suspected. As mentioned before, check GC logs...etc. You still have not provided such info so can't help further. If GC is the problem, try to cache less things or expire them from cache more quickly. Also, be mindful which queries you cache...etc.

Finally, Infinispan 9.2.0.CR1 will be released in the next day or so, which will come with Hibernate 5.2 cache provider. This can easily be used with a Spring application too. Might be worth trying that out when that's out.