We've built an installation of Keycloak 6.0.1 with cross-datacenter
replication using Infinispan Server 9.4.15.Final.
About three minutes after starting Keycloak, we start getting lots of
sequences of the following three errors:
ISPN000136: Error executing command RemoveExpiredCommand on Cache 'work'
ISPN000136: Error executing command RemoveCommand on Cache 'work'
ISPN000071: Caught exception when handling command SingleXSiteRpcCommand
Please, see the attached log for more detail.
The attached log is the server.log one of the Infinispan servers.
There you can see that around 17:49 the Keycloak server started and a
couple of warnings regarding cache activity are shown. These warnings are
normal as per Keycloak documentation troubleshooting.
At 17:52, the errors start appearing in a non-stop fashion.
This goes on until one of the infinispan sites automatically goes offline
(only one repetition included in the log file, because it was too big).
The errors are always related to the work cache only.
Note that the log has been anonymized and host, node, site names, IP's,
ports, etc. do not correspond to the real ones. We have made a best effort
not to obfuscate any relevant information, however.
This log corresponds to an Infinispan 9.4.15 and a Keycloak 4.8.3, both in
domain mode, with one node per data center each.
No error appears in Keycloak logs, even in debug mode (or we haven't found
To discard possible mistakes in the configuration, we have also tested it
with the exact configuration in:
These are the results for serveral combinations of versions, always with
the exact example configuration from the docs:
Keycloak Infinispan Result
======== ======== ======
6.0.1 9.4.15 Error
4.8.3 9.4.15 Error
4.8.3 9.4.0 Error
4.8.3 9.3.6 Works
6.0.1 9.3.6 Does not connect
5.0.0 9.3.6 Does not connect
In the first three setups, the log looks exactly the same as the attached
In the fourth setup, everything works fine.
In the last two setups, Keycloak couldn't connect to infinispan, probably
because it's too old.
The bug shows up only for Infinispan 9.3.x, independently of the version of Keycloak.
For this reason this is likely a problem in Infinispan.
However, we have not been able to reproduce the problem by directly writing data to infinispan caches. We need Keycloak to reproduce it.
Any help will be appreciated. If you need any further information, don't
hesitate to ask.
Thank you very much,
server.log.zip 13.4 KB