Null Pointer Exception when using Clustering

[lewisifer]

Hello,

I'm currently using JBoss 7.1.1 . I set up clustering as advised here: https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto . I did this because we have been getting Null Pointer Exceptions in our prod environment when using clustering. I did this on a vanilla 7.1.1 install with minimal changes and I'm able to get the NullPointerException to occur on my test environment as well. This appears to be caused by a difficulty obtaining a lock on the active session, and we cannot use clustering on production as a result.

Steps to Replicate:

1. Keep a session open / active to the web app

2. Deploy the web app

3. Hit the reload page (once is enough) when deployment is taking place

4. Observe error below

Server:server-01] 22:43:01,638 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host]] (ajp-/192.168.11.1:8009-1) Exception sending request initialized lifecycle event to listener instance of class org.jboss.weld.servlet.WeldListener: java.lang.NullPointerException

[Server:server-01]      at org.jboss.as.clustering.lock.YieldingGloballyExclusiveClusterLockSupport.handleLockSuccess(YieldingGloballyExclusiveClusterLockSupport.java:88)

[Server:server-01]      at org.jboss.as.clustering.lock.AbstractClusterLockSupport.getLock(AbstractClusterLockSupport.java:453)

[Server:server-01]      at org.jboss.as.clustering.lock.AbstractClusterLockSupport.lock(AbstractClusterLockSupport.java:180)

[Server:server-01]      at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:436)

[Server:server-01]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:381)

[Server:server-01]      at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:529) [jboss-as-web-7.1.x.incremental.129.jar:7.1.x.incremental.129]

[Server:server-01]      at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:497) [jboss-as-web-7.1.x.incremental.129.jar:7.1.x.incremental.129]

[Server:server-01]      at org.apache.catalina.connector.Request.doGetSession(Request.java:2625) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:841) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.getSession(LazySessionBeanStore.java:72) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

[Server:server-01]      at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.<init>(LazySessionBeanStore.java:58) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

[Server:server-01]      at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:31) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

[Server:server-01]      at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:16) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

[Server:server-01]      at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:134) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

[Server:server-01]      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

[Server:server-01]

[Server:server-01] 22:43:01,643 ERROR [org.apache.coyote.ajp.AjpProcessor] (ajp-/192.168.11.1:8009-1) Error processing request: java.lang.IllegalStateException: JBAS010217: Must call start() before first call to unlock()

[Server:server-01]      at org.jboss.as.clustering.lock.YieldingGloballyExclusiveClusterLockSupport.unlock(YieldingGloballyExclusiveClusterLockSupport.java:57)

[Server:server-01]      at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.unlock(SharedLocalYieldingClusterLockManager.java:502)

[Server:server-01]      at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.relinquishSessionOwnership(DistributedCacheManager.java:392)

[Server:server-01]      at org.jboss.as.web.session.ClusteredSession.relinquishSessionOwnership(ClusteredSession.java:575) [jboss-as-web-7.1.x.incremental.129.jar:7.1.x.incremental.129]

[Server:server-01]      at org.jboss.as.web.session.ClusteredSession.endAccess(ClusteredSession.java:568) [jboss-as-web-7.1.x.incremental.129.jar:7.1.x.incremental.129]

[Server:server-01]      at org.apache.catalina.connector.Request.recycle(Request.java:567) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:411) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.17.Final.jar:]

[Server:server-01]      at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]

Is this a bug with JBoss? I've been seeing many similar reports of NPEs in the JBoss AS. Would anyone be able to shed any light on this, and how to get rid of it? Your assistance is much appreciated.

[wdfink]

As this is an old verions I suppose that this problem is fixed in a later version.

Could you try the same with the latest WildFly Alpha?

If you not try to migrate to you might use the EAP6.1 bits, if you have no subscription and install in production you can use EAP6.1.0.Alpha which is the community AS7.2.0 version.

[rhusar]

Unfortunately this is a known issue, see https://bugzilla.redhat.com/show_bug.cgi?id=900549

[lewisifer]

Thanks Radoslav, does this mean that there is no fix out for this yet in a later version as well?

[rhusar]

There is no reliable fix yet. Improvements have been done for EAP 6.1.0 (http://www.jboss.org/jbossas/downloads/).

[rhusar]

Actually, is this only problem for you while the app is still deploying? That can be solved by using mod_cluster, which registeres contexts only when they are already started.

[lewisifer]

Hello Radoslav,

Thanks for your responses. Yes, this is the only problem. We are already using mod_cluster, and are experiencing these issues in spite of doing so. Is there any specific configuration that needs to be used to get this to work?

[rhusar]

The IllegalStateException sounds like has been fixed in later versions, e.g. 7.2.0 or 7.1.3 (just need to build from tag).

Well, mod_cluster does that automatically, so the NPE is not just while the application is deploying but can happen anytime. The recommendation is to turn off session passivation if you have it configured (either in the application or remove <file-store/> element from the web cache).

[lewisifer]

The problem is that we're using Torquebox to deploy, which uses JBoss. I think upgrading the JBoss version while using torquebox tends to cause other issues, so upgrading might not be a possibility.

The NPE hasn't been observed anytime except when deployment is taking place. In fact, it doesn't happen even when I turn off / kill the server process on one box to see if failover happens to the other box. That and the fact that the only recorded instance of NPEs occurring in production is during deployment points to the issue being limited to this case.

You'd mentioned that I should remove the file-store element from the web cache. Where does this reside? I do notice the "file store" element living in domain.xml . Should that be removed as well?

Thanks!

[ctomc]

latest versions of Torquebox are all based upon AS 7.2, so it might be a good thing to upgrade to newer (latest) version of torquebox.

[lewisifer]

Tomaz,

Unfortunately this problem is occurring on Torquebox v 2.3.2 which is the latest. I also tried downloading a newer version from their git repository(their master branch), and as is consistent what Radoslav mentioned, I was able to duplicate the occurrence on that version as well, which was using a later JBoss release.