8 Replies Latest reply on Apr 4, 2019 4:12 PM by amukherjee27

Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

amukherjee27 Mar 28, 2019 8:53 PM

Our platform upgraded to Wildfly 14.0.1.FINAL and we are experiencing below issue:-

1. Intermittently one of the production nodes (we have a cluster of 3) becomes non-responsive - there are no error trace BUT server is up.
2. When looked at the Wildfly console Runtime>Web(undertow) > Deployment > app.war ---> shows 90000+ active sessions.
3. On doing rolling restart the sessions distribute over to other nodes but are never killed and this number keeps on growing until we completely stop all servers together.
4. This is a concern since it impact our production environment.

Please advise.

1. Re: Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

pferraro Mar 30, 2019 10:15 AM (in response to amukherjee27)

Please paste your infinispan subsystem configuration, web.xml and any jboss-web.xml.
Are you able to reproduce this problem with the latest WF16 release?
Actions
2. Re: Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

amukherjee27 Apr 1, 2019 1:45 PM (in response to pferraro)

Hi pferraro, really appreciate your quick response. Here are the requested details -

1. Infinispan -

<subsystem xmlns="urn:jboss:domain:infinispan:7.0">
            <cache-container name="server" aliases="ha-partition" default-cache="default" module="org.wildfly.clustering.server">
                <transport lock-timeout="60000"/>
                <replicated-cache name="default">
                    <locking isolation="REPEATABLE_READ"/>
                    <transaction mode="BATCH"/>
                </replicated-cache>
            </cache-container>
            <cache-container name="web" default-cache="repl" module="org.wildfly.clustering.web.infinispan">
                <transport lock-timeout="60000"/>
                <replicated-cache name="repl">
                    <file-store/>
                </replicated-cache>
                <replicated-cache name="sso"/>
                <distributed-cache name="dist">
                    <locking isolation="REPEATABLE_READ"/>
                    <transaction mode="BATCH"/>
                    <file-store/>
                </distributed-cache>
            </cache-container>
            <cache-container name="hibernate" module="org.infinispan.hibernate-cache">
                <transport lock-timeout="60000"/>
                <local-cache name="local-query">
                    <object-memory size="10000"/>
                    <expiration max-idle="100000"/>
                </local-cache>
                <invalidation-cache name="entity">
                    <transaction mode="NON_XA"/>
                    <object-memory size="10000"/>
                    <expiration max-idle="100000"/>
                </invalidation-cache>
                <replicated-cache name="timestamps"/>
            </cache-container>
</subsystem>

3. web.xml -

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         id="WebApp_ID" version="3.0">
    <distributable/>
    <display-name>******</display-name>
    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>
            classpath:/META-INF/spring/applicationContext.xml,
            classpath:/META-INF/spring/infrastructure.xml,
            classpath:/META-INF/spring/security-config.xml
        </param-value>
    </context-param>

    

    <filter>
        <filter-name>encoding-filter</filter-name>
        <filter-class>org.springframework.web.filter.CharacterEncodingFilter</filter-class>
        <async-supported>true</async-supported>
        <init-param>
            <param-name>encoding</param-name>
            <param-value>utf-8</param-value>
        </init-param>
        <init-param>
            <param-name>forceEncoding</param-name>
            <param-value>true</param-value>
        </init-param>
    </filter>

    

    <filter>
        <filter-name>oemInViewFilter</filter-name>
        <filter-class>org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter</filter-class>
        <async-supported>true</async-supported>
    </filter>

    <filter>
        <filter-name>*****Filter</filter-name>
        <filter-class>com.******.web.*****Filter</filter-class>
        <async-supported>true</async-supported>
    </filter>

    
    

    <filter>
        <filter-name>XForwardedFilter</filter-name>
        <filter-class>com.******.web.XForwardedFilter</filter-class>
        <async-supported>true</async-supported>
        <init-param>
            <param-name>protocolHeader</param-name>
            <param-value>x-forwarded-proto</param-value>
        </init-param>
    </filter>

    <filter>
        <filter-name>HTTPCacheFilter</filter-name>
        <filter-class>com.******.web.HttpCacheFilter</filter-class>
        <async-supported>true</async-supported>
        <init-param>
            <param-name>Cache-Control</param-name>
            <param-value>max-age=86400, public</param-value>
        </init-param>
    </filter>

    <filter>
        <filter-name>springSecurityFilterChain</filter-name>
        <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
        <async-supported>true</async-supported>
    </filter>

    <filter>
        <filter-name>apiAccessFilter</filter-name>
        <filter-class>com.******.web.api.v1.APIAccessFilter</filter-class>
        <async-supported>true</async-supported>
    </filter>

    <filter-mapping>
        <filter-name>encoding-filter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    

    <filter-mapping>
        <filter-name>oemInViewFilter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <filter-mapping>
        <filter-name>*******Filter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    
    

    <filter-mapping>
        <filter-name>XForwardedFilter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <filter-mapping>
        <filter-name>HTTPCacheFilter</filter-name>
        <url-pattern>/resources/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <filter-mapping>
        <filter-name>springSecurityFilterChain</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <filter-mapping>
        <filter-name>apiAccessFilter</filter-name>
        <url-pattern>/v1/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>

    <servlet>
        <servlet-name>spring-servlet</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>classpath:/META-INF/spring/mvc-config.xml</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
        <async-supported>true</async-supported>
        <multipart-config>
            <file-size-threshold>10000000</file-size-threshold>
        </multipart-config>
    </servlet>

    <servlet>
        <servlet-name>atmosphere-servelet</servlet-name>
        <servlet-class>org.atmosphere.cpr.MeteorServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>classpath:/META-INF/spring/mvc-config.xml</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.servlet</param-name>
            <param-value>org.springframework.web.servlet.DispatcherServlet</param-value>
        </init-param>
        <init-param>
            <param-name>dispatchOptionsRequest</param-name>
            <param-value>true</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.cpr.broadcasterClass</param-name>
            <param-value>org.atmosphere.cpr.DefaultBroadcaster</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.cpr.broadcasterCacheClass</param-name>
            <param-value>org.atmosphere.cache.UUIDBroadcasterCache</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.resumeOnBroadcast</param-name>
            <param-value>true</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.cpr.broadcasterLifeCyclePolicy</param-name>
            <param-value>IDLE</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.cpr.maxBroadcasterLifeCyclePolicyIdleTime</param-name>
            <param-value>300000</param-value>
        </init-param>
        

        
        <load-on-startup>1</load-on-startup>
        <async-supported>true</async-supported>
        <multipart-config>
            <file-size-threshold>10000000</file-size-threshold>
        </multipart-config>
    </servlet>

    <servlet-mapping>
        <servlet-name>spring-servlet</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>

    <servlet-mapping>
        <servlet-name>atmosphere-servelet</servlet-name>
        <url-pattern>/socket/*</url-pattern>
    </servlet-mapping>

    <session-config>
        <session-timeout>30</session-timeout>
        <cookie-config>
            <http-only>true</http-only>
            
        </cookie-config>
        
    </session-config>
    <welcome-file-list>
        <welcome-file>index.jsp</welcome-file>
    </welcome-file-list>

    <error-page>
        <location>/WEB-INF/views/error/default.jsp</location>
    </error-page>

    <error-page>
        <error-code>400</error-code>
        <location>/WEB-INF/views/error/400.jsp</location>
    </error-page>

    <error-page>
        <error-code>401</error-code>
        <location>/WEB-INF/views/error/401.jsp</location>
    </error-page>

    <error-page>
        <error-code>403</error-code>
        <location>/WEB-INF/views/error/403.jsp</location>
    </error-page>

    <error-page>
        <error-code>404</error-code>
        <location>/WEB-INF/views/error/404.jsp</location>
    </error-page>

    <error-page>
        <error-code>405</error-code>
        
        <location>/WEB-INF/views/error/405.jsp</location>
    </error-page>

    <error-page>
        <error-code>500</error-code>
        <location>/WEB-INF/views/error/500.jsp</location>
    </error-page>

    <error-page>
        <error-code>502</error-code>
        <location>/WEB-INF/views/error/500.jsp</location>
    </error-page>

    <error-page>
        <error-code>503</error-code>
        <location>/WEB-INF/views/error/500.jsp</location>
    </error-page>

    <error-page>
        <error-code>504</error-code>
        <location>/WEB-INF/views/error/500.jsp</location>
    </error-page>

    <jsp-config>
        <jsp-property-group>
            <url-pattern>*.jsp</url-pattern>
            <page-encoding>UTF-8</page-encoding>
        </jsp-property-group>
    </jsp-config>

</web-app>

2. jboss-web.xml

<context-root>/</context-root>
Actions
3. Re: Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

amukherjee27 Apr 1, 2019 1:45 PM (in response to pferraro)

Hi pferraro, really appreciate your quick response. Here are the requested details -

1. Infinispan -

<subsystem xmlns="urn:jboss:domain:infinispan:7.0">
            <cache-container name="server" aliases="ha-partition" default-cache="default" module="org.wildfly.clustering.server">
                <transport lock-timeout="60000"/>
                <replicated-cache name="default">
                    <locking isolation="REPEATABLE_READ"/>
                    <transaction mode="BATCH"/>
                </replicated-cache>
            </cache-container>
            <cache-container name="web" default-cache="repl" module="org.wildfly.clustering.web.infinispan">
                <transport lock-timeout="60000"/>
                <replicated-cache name="repl">
                    <file-store/>
                </replicated-cache>
                <replicated-cache name="sso"/>
                <distributed-cache name="dist">
                    <locking isolation="REPEATABLE_READ"/>
                    <transaction mode="BATCH"/>
                    <file-store/>
                </distributed-cache>
            </cache-container>
            <cache-container name="hibernate" module="org.infinispan.hibernate-cache">
                <transport lock-timeout="60000"/>
                <local-cache name="local-query">
                    <object-memory size="10000"/>
                    <expiration max-idle="100000"/>
                </local-cache>
                <invalidation-cache name="entity">
                    <transaction mode="NON_XA"/>
                    <object-memory size="10000"/>
                    <expiration max-idle="100000"/>
                </invalidation-cache>
                <replicated-cache name="timestamps"/>
            </cache-container>
</subsystem>

3. web.xml -

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         id="WebApp_ID" version="3.0">
    <distributable/>
    <display-name>******</display-name>
    <context-param>
        <param-name>contextConfigLocation</param-name>
        <param-value>
            classpath:/META-INF/spring/applicationContext.xml,
            classpath:/META-INF/spring/infrastructure.xml,
            classpath:/META-INF/spring/security-config.xml
        </param-value>
    </context-param>

    

    <filter>
        <filter-name>encoding-filter</filter-name>
        <filter-class>org.springframework.web.filter.CharacterEncodingFilter</filter-class>
        <async-supported>true</async-supported>
        <init-param>
            <param-name>encoding</param-name>
            <param-value>utf-8</param-value>
        </init-param>
        <init-param>
            <param-name>forceEncoding</param-name>
            <param-value>true</param-value>
        </init-param>
    </filter>

    

    <filter>
        <filter-name>oemInViewFilter</filter-name>
        <filter-class>org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter</filter-class>
        <async-supported>true</async-supported>
    </filter>

    <filter>
        <filter-name>*****Filter</filter-name>
        <filter-class>com.******.web.*****Filter</filter-class>
        <async-supported>true</async-supported>
    </filter>

    
    

    <filter>
        <filter-name>XForwardedFilter</filter-name>
        <filter-class>com.******.web.XForwardedFilter</filter-class>
        <async-supported>true</async-supported>
        <init-param>
            <param-name>protocolHeader</param-name>
            <param-value>x-forwarded-proto</param-value>
        </init-param>
    </filter>

    <filter>
        <filter-name>HTTPCacheFilter</filter-name>
        <filter-class>com.******.web.HttpCacheFilter</filter-class>
        <async-supported>true</async-supported>
        <init-param>
            <param-name>Cache-Control</param-name>
            <param-value>max-age=86400, public</param-value>
        </init-param>
    </filter>

    <filter>
        <filter-name>springSecurityFilterChain</filter-name>
        <filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
        <async-supported>true</async-supported>
    </filter>

    <filter>
        <filter-name>apiAccessFilter</filter-name>
        <filter-class>com.******.web.api.v1.APIAccessFilter</filter-class>
        <async-supported>true</async-supported>
    </filter>

    <filter-mapping>
        <filter-name>encoding-filter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    

    <filter-mapping>
        <filter-name>oemInViewFilter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <filter-mapping>
        <filter-name>*******Filter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    
    

    <filter-mapping>
        <filter-name>XForwardedFilter</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <filter-mapping>
        <filter-name>HTTPCacheFilter</filter-name>
        <url-pattern>/resources/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <filter-mapping>
        <filter-name>springSecurityFilterChain</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <filter-mapping>
        <filter-name>apiAccessFilter</filter-name>
        <url-pattern>/v1/*</url-pattern>
        <dispatcher>REQUEST</dispatcher>
        <dispatcher>ASYNC</dispatcher>
    </filter-mapping>

    <listener>
        <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
    </listener>

    <servlet>
        <servlet-name>spring-servlet</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>classpath:/META-INF/spring/mvc-config.xml</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
        <async-supported>true</async-supported>
        <multipart-config>
            <file-size-threshold>10000000</file-size-threshold>
        </multipart-config>
    </servlet>

    <servlet>
        <servlet-name>atmosphere-servelet</servlet-name>
        <servlet-class>org.atmosphere.cpr.MeteorServlet</servlet-class>
        <init-param>
            <param-name>contextConfigLocation</param-name>
            <param-value>classpath:/META-INF/spring/mvc-config.xml</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.servlet</param-name>
            <param-value>org.springframework.web.servlet.DispatcherServlet</param-value>
        </init-param>
        <init-param>
            <param-name>dispatchOptionsRequest</param-name>
            <param-value>true</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.cpr.broadcasterClass</param-name>
            <param-value>org.atmosphere.cpr.DefaultBroadcaster</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.cpr.broadcasterCacheClass</param-name>
            <param-value>org.atmosphere.cache.UUIDBroadcasterCache</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.resumeOnBroadcast</param-name>
            <param-value>true</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.cpr.broadcasterLifeCyclePolicy</param-name>
            <param-value>IDLE</param-value>
        </init-param>
        <init-param>
            <param-name>org.atmosphere.cpr.maxBroadcasterLifeCyclePolicyIdleTime</param-name>
            <param-value>300000</param-value>
        </init-param>
        

        
        <load-on-startup>1</load-on-startup>
        <async-supported>true</async-supported>
        <multipart-config>
            <file-size-threshold>10000000</file-size-threshold>
        </multipart-config>
    </servlet>

    <servlet-mapping>
        <servlet-name>spring-servlet</servlet-name>
        <url-pattern>/</url-pattern>
    </servlet-mapping>

    <servlet-mapping>
        <servlet-name>atmosphere-servelet</servlet-name>
        <url-pattern>/socket/*</url-pattern>
    </servlet-mapping>

    <session-config>
        <session-timeout>30</session-timeout>
        <cookie-config>
            <http-only>true</http-only>
            
        </cookie-config>
        
    </session-config>
    <welcome-file-list>
        <welcome-file>index.jsp</welcome-file>
    </welcome-file-list>

    <error-page>
        <location>/WEB-INF/views/error/default.jsp</location>
    </error-page>

    <error-page>
        <error-code>400</error-code>
        <location>/WEB-INF/views/error/400.jsp</location>
    </error-page>

    <error-page>
        <error-code>401</error-code>
        <location>/WEB-INF/views/error/401.jsp</location>
    </error-page>

    <error-page>
        <error-code>403</error-code>
        <location>/WEB-INF/views/error/403.jsp</location>
    </error-page>

    <error-page>
        <error-code>404</error-code>
        <location>/WEB-INF/views/error/404.jsp</location>
    </error-page>

    <error-page>
        <error-code>405</error-code>
        
        <location>/WEB-INF/views/error/405.jsp</location>
    </error-page>

    <error-page>
        <error-code>500</error-code>
        <location>/WEB-INF/views/error/500.jsp</location>
    </error-page>

    <error-page>
        <error-code>502</error-code>
        <location>/WEB-INF/views/error/500.jsp</location>
    </error-page>

    <error-page>
        <error-code>503</error-code>
        <location>/WEB-INF/views/error/500.jsp</location>
    </error-page>

    <error-page>
        <error-code>504</error-code>
        <location>/WEB-INF/views/error/500.jsp</location>
    </error-page>

    <jsp-config>
        <jsp-property-group>
            <url-pattern>*.jsp</url-pattern>
            <page-encoding>UTF-8</page-encoding>
        </jsp-property-group>
    </jsp-config>

</web-app>

2. jboss-web.xml

<context-root>/</context-root>
Actions
4. Re: Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

amukherjee27 Apr 1, 2019 1:46 PM (in response to amukherjee27)

Hi pferraro, Just to reiterate the core issue is sudden intermittent non responsive server nodes in the cluster (one server is simply up but no logs / response) . Rolling restart fixes it.
Actions
5. Re: Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

pferraro Apr 2, 2019 9:00 AM (in response to amukherjee27)

There were a number of fixes to Infinispan transaction handling within the context of asynchronous servlets/filters since WF14. Can you try using a recent release, e.g. WF16?
Actions
6. Re: Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

amukherjee27 Apr 2, 2019 2:25 PM (in response to pferraro)

Hi pferraro ,

We just upgraded to Wildfly 14.0.1.FINAL and released to production.
As it is, it wont be ideal for a re-effort on testing/upgrade within a short span again (but in a few months we can look into a stable version), we were hoping to get some direction in terms of this sudden non responsive nodes in the cluster with Wildfly 14 .
Please advise.

thanks,
Arin
Actions
7. Re: Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

pferraro Apr 2, 2019 4:06 PM (in response to amukherjee27)

I suspect that the unresponsive nodes are caused by resource starvation - probably caused by asynchronous requests that never actually complete.
How long running are your asynchronous requests started by your servlets? The topic description suggests that distributed sessions are not expiring in a timely fashion - but unless you complete your asynchronous requests, the sessions will never have been idle. That would explain why the number of active sessions is so high.
Actions
8. Re: Wildfly 14.0.1.FINAL does not kill distributed sessions in a cluster (even after idle timeout)

amukherjee27 Apr 4, 2019 4:12 PM (in response to pferraro)

Thanks pferraro for the update.
To clarify though - this behavior (the sessions not expiring) is noticed even when we do not have any async threads , Just ping service for the clustered nodes.
Here is what we see -

1. We see the ping calls to the service creating sessions.
2. We do see that the sessions expire after the idle time out.
3. We stop one node - all its active sessions get distributed to other nodes.
4. These distributed sessions never expire and as keep doing rolling restarts, the active session count (of the distributed sessions) keep growing and we huge numbers in all nodes.

Please advise, if you are aware of this orphan sessions not expiring kind of behavior(we see the numbers on Wildfly console Runtime>Web(undertow) > Deployment > app.war) and any apparent solutions ?
Actions

Go to original post