Infinispan 9.4.x: Lots of errors in cross-datacenter replication mode, in a Keycloak installation.

[jafarre.viewnext]

Hello,

We've built an installation of Keycloak 6.0.1 with cross-datacenter

replication using Infinispan Server 9.4.15.Final.

About three minutes after starting Keycloak, we start getting lots of

sequences of the following three errors:

ISPN000136: Error executing command RemoveExpiredCommand on Cache 'work'

ISPN000136: Error executing command RemoveCommand on Cache 'work'

ISPN000071: Caught exception when handling command SingleXSiteRpcCommand

Please, see the attached log for more detail.

The attached log is the server.log one of the Infinispan servers.

There you can see that around 17:49 the Keycloak server started and a

couple of warnings regarding cache activity are shown. These warnings are

normal as per Keycloak documentation troubleshooting.

At 17:52, the errors start appearing in a non-stop fashion.

This goes on until one of the infinispan sites automatically goes offline

(only one repetition included in the log file, because it was too big).

The errors are always related to the work cache only.

Note that the log has been anonymized and host, node, site names, IP's,

ports, etc. do not correspond to the real ones. We have made a best effort

not to obfuscate any relevant information, however.

This log corresponds to an Infinispan 9.4.15 and a Keycloak 4.8.3, both in

domain mode, with one node per data center each.

No error appears in Keycloak logs, even in debug mode (or we haven't found

it).

To discard possible mistakes in the configuration, we have also tested it

with the exact configuration in:

https://www.keycloak.org/docs/latest/server_installation/index.html#setup

These are the results for serveral combinations of versions, always with

the exact example configuration from the docs:

Keycloak        Infinispan      Result

========        ========        ======

6.0.1           9.4.15          Error

4.8.3           9.4.15          Error

4.8.3           9.4.0           Error

4.8.3           9.3.6           Works

6.0.1           9.3.6           Does not connect

5.0.0           9.3.6           Does not connect

In the first three setups, the log looks exactly the same as the attached

log.

In the fourth setup, everything works fine.

In the last two setups, Keycloak couldn't connect to infinispan, probably

because it's too old.

The bug shows up only for Infinispan 9.3.x, independently of the version of Keycloak.

For this reason this is likely a problem in Infinispan.

However, we have not been able to reproduce the problem by directly writing data to infinispan caches. We need Keycloak to reproduce it.

Any help will be appreciated. If you need any further information, don't

hesitate to ask.

Thank you very much,

Juan

[pruivo]

Hi Juan,

Can you share the server.log from site2? It should contain more info.

Pedro

[jafarre.viewnext]

Hi, Pedro:

Thanks for your quick answer.

I didn't share it, because it's totally equivalent (symmetric). You just need to exchange node names.

I should repeat the test to be able to send it to you, if you really find it necessary.

Kind regards,

Juan

[jafarre.viewnext]

Hi again, Pedro:

I have found the corresponding log in the second server.

I attach it. I hope it helps.

Thank you,

Juan

[pruivo]

Hi Juan,

You are hitting [ISPN-10650] Expiration should not be replicated cross site - JBoss Issue Tracker

The issue will be fixed in the next 9.4.x release (it is already fixed in 10.0.0.CR3).

It is basically a deadlock. When site2 receives an operation for site1 (site1 keeps the locks until site2 finishes the operation), it checks for the expiration and, in this case, it triggers a cluster expiration event.
The event reaches site1 which fails to acquire the lock.

Pedro

[jafarre.viewnext]

Thank you, Pedro!

So it is a bug and not something I may fix in my configuration?

Should this be affecting versions 9.3.x too?

And last, but not least, is there any ETA for version 9.4.17.Final?

Kind regards,

Juan

[pruivo]

Hi,

#1 The only workaround is disabling expiration everywhere (in cache configuration and avoid using put(key, value, max-idle, lifespan) and similar methods in your code)

#2 yes, cluster expiration was introduced in 9.3.0.Final. We are not sure if the issue is there since the beginning of if it was introduced recently (a user detected it last month)

#3 9.4.17.Final should be available by the end of the month (more or less on the same week as 10.0.0.Final).

Pedro

[jafarre.viewnext]

Thank you very much! That was of great help!

[jafarre.viewnext]

Hello, Pedro,

One question: do you know whether this fix will be backported to the 9.3.x release series?

Thanks a lot!

[jafarre.viewnext]

Hi again, Pedro,

I can see that Infinispan version 10.0.1 was released the first of November, but I can't find any information about version 9.4.17, yet.

Do you know whether it is still planned or there will be no more 9.4.x releases after version 10?

In my case, I can't move to version 10, as it wouldn't be compatible with Keycloak.

Thank you,

Juan

[pruivo]

Hi Juan,

9.3.x is "dead"; no more 9.3.x relases.

I'm sorry I did a mistake. There is no more 9.4.x releases after 10.0.0.Final. But, the 9.4.x branch is patched and you can compile yourself and use the patched server zip (and or other jars).

I guess you can also ping the keycloak team to see if they are going to update to Infinispan 10 and how long will would take.

Cheers,

Pedro

[jafarre.viewnext]

Thanks again, Pedro!