Hello,

We've built an installation of Keycloak 6.0.1 with cross-datacenter

replication using Infinispan Server 9.4.15.Final.

About three minutes after starting Keycloak, we start getting lots of

sequences of the following three errors:

ISPN000136: Error executing command RemoveExpiredCommand on Cache 'work'

ISPN000136: Error executing command RemoveCommand on Cache 'work'

ISPN000071: Caught exception when handling command SingleXSiteRpcCommand

Please, see the attached log for more detail.

The attached log is the server.log one of the Infinispan servers.

There you can see that around 17:49 the Keycloak server started and a

couple of warnings regarding cache activity are shown. These warnings are

normal as per Keycloak documentation troubleshooting.

At 17:52, the errors start appearing in a non-stop fashion.

This goes on until one of the infinispan sites automatically goes offline

(only one repetition included in the log file, because it was too big).

The errors are always related to the work cache only.

Note that the log has been anonymized and host, node, site names, IP's,

ports, etc. do not correspond to the real ones. We have made a best effort

not to obfuscate any relevant information, however.

This log corresponds to an Infinispan 9.4.15 and a Keycloak 4.8.3, both in

domain mode, with one node per data center each.

No error appears in Keycloak logs, even in debug mode (or we haven't found

it).

To discard possible mistakes in the configuration, we have also tested it

with the exact configuration in:

https://www.keycloak.org/docs/latest/server_installation/index.html#setup

These are the results for serveral combinations of versions, always with

the exact example configuration from the docs:

Keycloak        Infinispan      Result

========        ========        ======

6.0.1           9.4.15          Error

4.8.3           9.4.15          Error

4.8.3           9.4.0           Error

4.8.3           9.3.6           Works

6.0.1           9.3.6           Does not connect

5.0.0           9.3.6           Does not connect

In the first three setups, the log looks exactly the same as the attached

log.

In the fourth setup, everything works fine.

In the last two setups, Keycloak couldn't connect to infinispan, probably

because it's too old.

The bug shows up only for Infinispan 9.3.x, independently of the version of Keycloak.

For this reason this is likely a problem in Infinispan.

However, we have not been able to reproduce the problem by directly writing data to infinispan caches. We need Keycloak to reproduce it.

Any help will be appreciated. If you need any further information, don't

hesitate to ask.

Thank you very much,

Juan