11 Replies Latest reply on Nov 14, 2019 2:39 AM by Juan Antonio Farré Basurte

    Infinispan 9.4.x: Lots of errors in cross-datacenter replication mode, in a Keycloak installation.

    Juan Antonio Farré Basurte Newbie

      Hello,

       

      We've built an installation of Keycloak 6.0.1 with cross-datacenter

      replication using Infinispan Server 9.4.15.Final.

       

      About three minutes after starting Keycloak, we start getting lots of

      sequences of the following three errors:

      ISPN000136: Error executing command RemoveExpiredCommand on Cache 'work'

      ISPN000136: Error executing command RemoveCommand on Cache 'work'

      ISPN000071: Caught exception when handling command SingleXSiteRpcCommand

       

      Please, see the attached log for more detail.

       

      The attached log is the server.log one of the Infinispan servers.

      There you can see that around 17:49 the Keycloak server started and a

      couple of warnings regarding cache activity are shown. These warnings are

      normal as per Keycloak documentation troubleshooting.

      At 17:52, the errors start appearing in a non-stop fashion.

      This goes on until one of the infinispan sites automatically goes offline

      (only one repetition included in the log file, because it was too big).

      The errors are always related to the work cache only.

      Note that the log has been anonymized and host, node, site names, IP's,

      ports, etc. do not correspond to the real ones. We have made a best effort

      not to obfuscate any relevant information, however.

       

      This log corresponds to an Infinispan 9.4.15 and a Keycloak 4.8.3, both in

      domain mode, with one node per data center each.

      No error appears in Keycloak logs, even in debug mode (or we haven't found

      it).

       

      To discard possible mistakes in the configuration, we have also tested it

      with the exact configuration in:

      https://www.keycloak.org/docs/latest/server_installation/index.html#setup

       

      These are the results for serveral combinations of versions, always with

      the exact example configuration from the docs:

       

      Keycloak        Infinispan      Result

      ========        ========        ======

      6.0.1           9.4.15          Error

      4.8.3           9.4.15          Error

      4.8.3           9.4.0           Error

      4.8.3           9.3.6           Works

      6.0.1           9.3.6           Does not connect

      5.0.0           9.3.6           Does not connect

       

      In the first three setups, the log looks exactly the same as the attached

      log.

      In the fourth setup, everything works fine.

      In the last two setups, Keycloak couldn't connect to infinispan, probably

      because it's too old.

       

      The bug shows up only for Infinispan 9.3.x, independently of the version of Keycloak.

      For this reason this is likely a problem in Infinispan.

      However, we have not been able to reproduce the problem by directly writing data to infinispan caches. We need Keycloak to reproduce it.

       

      Any help will be appreciated. If you need any further information, don't

      hesitate to ask.

       

      Thank you very much,

       

      Juan