4 Replies Latest reply on Nov 11, 2014 6:17 PM by pferraro

Wildfly 8.1.0 + standard-full-ha in two node cluster: Infinispan: You have to start the service first!

jeroenvanvianen Oct 31, 2014 11:26 AM

Hi,

I have a two node cluster with standalone Wildfly 8.1.0 on JDK 8.0.25 started as follows:

/opt/wildfly/bin/standalone.sh -c standalone-full-ha.xml -b=192.168.11.56 -bmanagement=192.168.11.56 -u 230.0.0.4 -Djboss.node.name=FW-PROD4-01

/opt/wildfly/bin/standalone.sh -c standalone-full-ha.xml -b=192.168.11.57 -bmanagement=192.168.11.56 -u 230.0.0.4 -Djboss.node.name=FW-PROD4-02

nginx is used as a loadbalancer / proxy.

I have deployed four WAR files which contain Spring MVC and Hibernate. I want to use session replication and this works if I drop either one of above nodes from the nginx configuration.

However, after a while I get a blank page in my browser and the following stacktrace:

2014-10-31 13:50:33,273 ERROR [io.undertow.request] (default task-31) UT005023: Exception handling request to /: org.springframework.web.util.NestedServletException: Request processing failed; nested exception is java.lang.IllegalStateException: You have to start the service first!

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:978) [spring-webmvc-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:857) [spring-webmvc-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:842) [spring-webmvc-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]

at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:130) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) [spring-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:177) [spring-orm-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:132) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:85) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:61) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:113) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:56) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:45) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:61) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:70) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:25) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:240) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727) [undertow-core-1.0.15.Final.jar:1.0.15.Final]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_25]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_25]

at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_25]

Caused by: java.lang.IllegalStateException: You have to start the service first!

at org.infinispan.affinity.KeyAffinityServiceImpl.getKeyForAddress(KeyAffinityServiceImpl.java:97)

at org.wildfly.clustering.web.infinispan.AffinityIdentifierFactory.createIdentifier(AffinityIdentifierFactory.java:50)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.createIdentifier(InfinispanSessionManager.java:139)

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.createIdentifier(InfinispanSessionManager.java:71)

at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.createSession(DistributableSessionManager.java:90)

at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:692) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:353) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:358) [undertow-servlet-1.0.15.Final.jar:1.0.15.Final]

at com.mycompany.project.util.ShopSelectInterceptor.preHandle(ShopSelectInterceptor.java:37) [classes:1.0-SNAPSHOT]

at org.springframework.web.servlet.HandlerExecutionChain.applyPreHandle(HandlerExecutionChain.java:130) [spring-webmvc-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938) [spring-webmvc-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:877) [spring-webmvc-4.1.1.RELEASE.jar:4.1.1.RELEASE]

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:966) [spring-webmvc-4.1.1.RELEASE.jar:4.1.1.RELEASE]

... 40 more

In above stacktrace the last line in my code is bold and contains a call to HttpSession.getAttribute().

The infinispan configuration in standard-full-ha.xml has not been changed from its default:

<cache-container name="server" default-cache="default" module="org.wildfly.clustering.server" aliases="singleton cluster">

<replicated-cache name="default" batching="true" mode="SYNC">

</replicated-cache>

</cache-container>

<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">

<distributed-cache name="dist" batching="true" mode="ASYNC" owners="2" l1-lifespan="0">

<file-store/>

</distributed-cache>

</cache-container>

<cache-container name="ejb" default-cache="dist" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">

<distributed-cache name="dist" batching="true" mode="ASYNC" owners="2" l1-lifespan="0">

<file-store/>

</distributed-cache>

</cache-container>

<cache-container name="hibernate" default-cache="local-query" module="org.hibernate">

<local-cache name="local-query">

</local-cache>

<invalidation-cache name="entity" mode="SYNC">

</invalidation-cache>

<replicated-cache name="timestamps" mode="ASYNC">

</replicated-cache>

</cache-container>

</subsystem>

I have the following questions:

1. Why does this error occur?

2. How do I start infinispan if it's not running?

3. Why would this work from the start, but stop after a while?

Thanks,

Jeroen

1. Re: Wildfly 8.1.0 + standard-full-ha in two node cluster: Infinispan: You have to start the service first!

rhusar Oct 31, 2014 2:47 PM (in response to jeroenvanvianen)

Just judging by the title, what happens if you set the cache to start eagerly?

[standalone@localhost:9990 /] /subsystem=infinispan/cache-container=web/distributed-cache=dist/:write-attribute(name=start,value=EAGER)

1 of 1 people found this helpful
Actions
2. Re: Wildfly 8.1.0 + standard-full-ha in two node cluster: Infinispan: You have to start the service first!

jeroenvanvianen Nov 1, 2014 4:48 PM (in response to rhusar)

Thanks, this seems to solve the issue; I will monitor it some more.

Best regards,

Jeroen
Actions
3. Re: Wildfly 8.1.0 + standard-full-ha in two node cluster: Infinispan: You have to start the service first!

pferraro Nov 4, 2014 12:02 AM (in response to rhusar)

[standalone@localhost:9990 /] /subsystem=infinispan/cache-container=web/distributed-cache=dist/:write-attribute(name=start,value=EAGER)

I don't see why this would fix anything. The cache referenced here isn't the one used by the web application in question. Each web application uses its own cache instance, using configuration cloned from this cache. Rather, this looks like the web application was undeployed, but a request was still being serviced. During undeployment, the session manager is stopped, which triggers the KeyAffinityService to stop (thus any call to KeyAffinityService.getKeyForAddress(...) will throw an ISE). WildFly 9 adds clean shutdown logic which prevents this condition (i.e. the session manager will not stop until all accepted requests have completed).
In general, I strongly discourage users from starting web/ejb/jpa caches eagerly, since these cache will not stop when an application is undeployed and will result in memory/classloader leaks.
Actions
4. Re: Wildfly 8.1.0 + standard-full-ha in two node cluster: Infinispan: You have to start the service first!

pferraro Nov 11, 2014 6:17 PM (in response to jeroenvanvianen)

FYI, I've found the underlying problem: [ISPN-4969] Stopping a cache will stop all KeyAffinityServices created for other caches in the cache manager - JBoss Iss…

I've submitted an upstream fix here:
[6.0.x] ISPN-4969 Stopping a cache will stop all KeyAffinityServices created for other caches in the cache manager by pf…
[master] ISPN-4969 Stopping a cache will stop all KeyAffinityServices created for other caches in the cache manager by p…

And submitted a workaround for WildFly here:
[8.x] Add workaround for ISPN-4969 by pferraro · Pull Request #6940 · wildfly/wildfly · GitHub
and as part of:
WFLY-4067 DummyTransaction exception during session.invalidate with started conversation by pferraro · Pull Request #693…
Actions

Go to original post