Distributed application on JBoss EAP 7.1 loses Session attributes

[pthorson]

Our Spring MVC project is running on JBoss EAP 7.1 with remote caching to DataGrid 7.2. We have observed occasional instances of Session Attributes not being found.

Logging shows entries like:

2019-10-11 16:13:42,543 WARN  [org.wildfly.clustering.web.infinispan] (default task-97) WFLYCLWEBINF0011: Session  is missing cache entry for attribute 

And see subsequent exceptions from Spring like:

2019-10-11 16:13:42,612 SEVERE [gov.wv.dhhr.ies.cw.administration.application.presentation.util.CWExceptionHandler] (default task-85) Exception Occured: org.springframework.web.bind.ServletRequestBindingException: Missing session attribute 'SESSION_CONTEXT_USI' of type UserSessionInfo
at org.springframework.web.servlet.mvc.method.annotation.SessionAttributeMethodArgumentResolver.handleMissingValue(SessionAttributeMethodArgumentResolver.java:60)
at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.handleMissingValue(AbstractNamedValueMethodArgumentResolver.java:220)
at org.springframework.web.method.annotation.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:112)
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:124)
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:161)

We are finding it difficult to pinpoint a trace or debug that can reveal more information. As observed, the instances of this are sporadic, but can be repeated occasionally.

Session replication is configured in jboss-web.xml as follows:

    <replication-config>

        <replication-granularity>ATTRIBUTE</replication-granularity>

        <cache-name>web.session_cache</cache-name>

    </replication-config>

And cache configured as follows in standalone-ha.xml:

            <cache-container name="web" default-cache="session-cache" module="org.wildfly.clustering.web.infinispan">

                <transport lock-timeout="60000"/>

                <local-cache name="passivation">

                    <locking isolation="REPEATABLE_READ"/>

                    <transaction mode="BATCH"/>

                    <file-store passivation="true" purge="false"/>

                </local-cache>

                <invalidation-cache name="session-cache" mode="SYNC">

                    <remote-store cache="default" socket-timeout="60000" remote-servers="remote-jdg-node1" passivation="false" purge="false" shared="true"/>

                </invalidation-cache>

                <replicated-cache name="sso" mode="SYNC"/>

                <distributed-cache name="dist" mode="ASYNC" l1-lifespan="0">

                    <file-store/>

                </distributed-cache>

            </cache-container>

And socket binding like:

        <outbound-socket-binding name="remote-jdg-node1">

            <remote-destination host="${jdg.remoting.hothod.node1.addr:127.0.0.1}" port="${jdg.remoting.hothod.node1.port:11822}"/>

        </outbound-socket-binding>

Any thoughts on what the issue might be?

Thank you.

[pferraro]

ATTRIBUTE granularity stores the set of session attribute names in one cache entry, and a separate cache entry per session attribute.  When a given request loads the session, the session manager validates that all attribute entries exist an can be loaded/deserialized.  The first log entry indicates that a particular session attribute entry was not found.  There could be a number of causes - from timing issues to misconfiguration of the datagrid.

Can you paste your cache configuration used by the data grid?

Additionally, Spring has a habit of retaining references to the session and accessing them outside of the context of a request, which can be problematic for WildFly's distributed session manager.  While this is usually only problematic for transactional caches, I don't want to rule that out just yet.  Do we know the context of this method org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(...) ?

[pthorson]

In standalone-ha, the invalidation-cache, session-cache, has the remote-store cache set to default. The datagrid 7.2 server I connect to for this test is launching with clustered.xml with this configuration for the cache:

        <subsystem xmlns="urn:infinispan:server:core:8.5" default-cache-container="clustered">

            <cache-container name="clustered" default-cache="default" statistics="true">

                <transport lock-timeout="60000"/>

                <global-state/>

...

              <distributed-cache name="default"/>

...

            </cache-container>

        </subsystem>

I believe we are just using the out of the box configuration on this server.

The actual exception does come in the servicing of an AJAX request by a Spring MVC @Controller with @RequestMapping, but appears to be within Spring MVC's binding initialization.

Following this, I can see that a new session gets created. I assume that the old session can no longer be found at that point:

10:49:08,229 SEVERE [gov.wv.dhhr.ies.cw.common.util.UserSessionInfoUtils] (default task-101) UserSessionInfoUtils.getUserSessionInfo: UserSessionInfo is NULL
10:49:08,257 INFO  [gov.wv.dhhr.ies.cw.security.profile.RecentItemsSessionListener] (default task-92) Session created hvJi0BUoiWjT3uunomgW3WG75akJXKZmlS3diSHb

[pthorson]

pferraro is there a debug string we could enable on EAP or DataGrid to better isolate this issue?

[pthorson]

It appears this might be related to a known issue that is corrected by an upgrade to the latest versions of EAP 7.2 and DataGrid 7.3. We'll be upgrading and checking if this resolves the issue.