How to Reduce the Amount of Exception Data Transmitted

[jkharness87]

Hello -

We've noticed that whenever we have a timeout exception (ISPN000476), this information gets sent to other nodes in the cluster who log it with the ISPN000217 code. Recently we experienced a flood of timeout errors (thousands) which, in turn, led to a bunch of network transmissions to communicate this fact to the other nodes. The result was that a 20 Gig network link was saturated which, in turn, caused a cascading failure (since it induced more timeouts, etc).

Is there any way to control this behavior or the amount of data that gets serialized? It appears that entire stack traces are sent which gets very heavy from a payload size.

I can increase the timeout from the 15 second default but feel like that would just be covering up the issue as far as how the application reactes when there's a timeout (or mass of timeouts).

We're using keycloak 6.0.0 which is based on Wildfly 16 (which uses Infinispan 9.4.8).

Thanks!

Josh

[dan.berindei]

See also Communicating Infinispan Remote Exceptions Generates Excessive Network Traffic - Stack Overflow

In general I would not recommend increasing the timeout, because 15 seconds is already pretty bad as a response time. However, I don't a better alternative, unless you happen to be using RocksDBStore

[jkharness87]

While we can't control the amount of data transmitted, we did prevent the errors/exceptions that were happening by switching from the UDP stack to the TCP stack for our small cluster. This removed the errors seen under load and thus fixed the issue for us. More details here:

keycloak - Communicating Infinispan Remote Exceptions Generates Excessive Network Traffic - Stack Overflow