Wildfly 10 very slow in cluster with session replication

[lagoria]

I have two nodes (both in localhost at the moment) running standalone-full-ha profile. Both nodes are running the same JSF application (state is set to server side).

The application runs very slow when both nodes are up. Whilst If I shutdown one of them, the application just runs fast.

Trying to figure out what was going on I have taken many thread dumps (this is the JVM that is currently serving the request), all look like this:

2017-11-25 10:02:01 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.151- - Pastebin.com

where  you can see that infinispan threads are waiting for something (responses from other node?). These thread seems to be related to the session cache because I see the following line towards the root of the thread:

org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:148)

Other JVM thread dump (taken at the same time) shows nothing, a part from the "idle" threads obviously.

The session  cache is the default you can find in Wildfly 10 standalone-full-ha profile (I have just removed file-store):

<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

                <transport lock-timeout="60000"/>

                <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0" owners="2">

                    <locking isolation="REPEATABLE_READ"/>

                    <transaction mode="BATCH"/>

                    <!--file-store/-->

                </distributed-cache>

                <distributed-cache name="concurrent" mode="SYNC" l1-lifespan="0" owners="2">

                    <file-store/>

                </distributed-cache>

</cache-container>

Also it should be in ASYNC mode, so why these threads are locked waiting for something?

[lagoria]

Update: Now I am getting timeouts errors from the infinispan cache.

From my point of view session replication just doesn't work (and can you find other people saying the same complaining about bug ISPN000299).

Fortunately, session replication is not very important to me. I am ok if WildFly still can replicate my EclipseLink L2 cache (this is done through clustered JMS) and some Infinispan replicated custom cache I am using to save on sql queries.

At this point, my question is: can I just drop the <distributed /> tag in web.xml and still use the <distributed-cache> provided from the standalone-full-ha profile or should I configure a local cache?

[pferraro]

1. Which WildFly version are you using?
2. Don't use ASYNC mode.  This does nothing to improve performance - and will only exacerbate thread contention.  Infinispan cache txs are committed after the response is sent to the client, so the behavior of a SYNC cache is already asynchronous from the perspective of a web client.
3. Are you load balancing requests to your cluster?  If so, do you have sticky sessions enabled?  If not, performance will be impacted as transactions against the session must be coordinated by the node owning the associated Infinispan cache entries.
4. Does your application process concurrent requests for the same session?  If so, you will likely observe better performance using the "concurrent" cache configuration.

[lagoria]

Thanks for your reply.

1. Version is 10.1.0.Final

2. ok but the configuration I pasted comes straight from default standalone-full-ha profile. You should probably fix the provided configuration if that's not ideal.

3. I will, at the moment I was just sending all requests to one node though.

4. Nop, also I was testing it using JMeter that I am pretty sure does not send in session parallel requests by default (thus waits for a response before sending a new HTTP request).

Now that I am using a non clustered session configuration (no distributed tag in web.xml), performance is so much better and there is no Infinispan timeout error or slow down caused by infinispan. Anyway I am still using full-ha to leverage clustered JMS and infinispan replication (but not in web cache).

[pferraro]

1. 11.0.0.Final will deliver better performance and improved stability.
2. We did. wildfly/infinispan.xml at 11.0.0.Final · wildfly/wildfly · GitHub
3. OK - that's fine for measuring performance, as cache entries are owned by the node that created the session.  Ownership can only change in response to a cluster topology change (i.e. nodes joining/leaving).
4. If the response contains links for session specific artifacts (e.g. stylesheets, images, etc.) does JMeter send subsequent concurrent requests for these?  Or do the measurements only include the initial response?  If these linked artifacts are not user specific - make sure you do not append the JSESSIONID to these, as that will result in unnecessary operations on the Infinispan cache - and would have a sizable impact on performance.

If you don't need HA web sessions - why enable it in the first place?  It would be unreasonable to expect comparable performance between a distributed session manager and a simple in-memory session manager - regardless of vendor.

[lagoria]

I'll try WF 11 then.

As for point 4, I have excluded every css,js,images etc resources, so it's not the case.

Need for HA: would have been a nice plus in case of single node failure. However I don't expect frequent failures, so if users (once in a year hopefully) are just logged out and redirected to the page they were before, should not be a big issue after all.

Performance: yes I realise what you say now. I reckon this is especially true if you are using JSF with a good number of session or view scoped beans.

Ah, my original question still applies: if I am not using HA web sessions, should I configure a web <local-cache> instead of the one provided by default in standalone-full-ha profile?

[pferraro]

lagoria  wrote:

Ah, my original question still applies: if I am not using HA web sessions, should I configure a web <local-cache> instead of the one provided by default in standalone-full-ha profile?

You can, but only if you need the behavior it enables.  There are 2 Infinispan configurations for distributable web sessions in standalone.xml (or standalone-full.xml) that do just this, to support a distributable session manager that supports session passivation, where old sessions are persisted to disk when the number of active sessions exceeds a given threshold; and another that support persistent sessions, where session are persisted to disk per request.  The former is more performant, though a node crash will result in loss of session data.