10 Replies Latest reply on Oct 17, 2013 4:27 AM by marcodanti

Exception acquiring ownership

marcodanti Oct 14, 2013 5:24 AM

Scenario:

two AS-7.2.0 nodes in standalone-ha mode, fronted by a httpd-2.4.6 reverse proxy with mod_cluster-1.2.6.Final
I removed the directives 'sticky-session="true" sticky-session-force="true" from standalone-ha.xml
when I call the application, I can see from the logs that the first jboss node starts processing the request
however, the request does not complete and on the seconf jboss node the following exception pops up every 30 seconds
the browser from which the request was originated remains hanged forever

What can be the reason for this behaviour, and what can I do to solve this?

10:58:36,196 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host]] (ajp-/0.0.0.0:7009-9) JBWEB000233: Exception sending request initialized lifecycle event to listener instance of class org.jboss.weld.servlet.WeldListener: java.lang.RuntimeException: JBAS018060: Exception acquiring ownership of MPRkQLqFI4lwiC61-Y7YyHD4

at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:497) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

at org.apache.catalina.connector.Request.doGetSession(Request.java:2612) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.apache.catalina.connector.Request.getSession(Request.java:2357) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:790) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.getSession(LazySessionBeanStore.java:72) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

at org.jboss.weld.context.beanstore.http.LazySessionBeanStore.<init>(LazySessionBeanStore.java:58) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:31) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

at org.jboss.weld.context.http.HttpSessionContextImpl.associate(HttpSessionContextImpl.java:16) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

at org.jboss.weld.servlet.WeldListener.requestInitialized(WeldListener.java:134) [weld-core-1.1.10.Final.jar:2012-10-12 10:00]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:134) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_21]

Caused by: org.jboss.as.clustering.lock.TimeoutException: JBAS010213: Cannot acquire lock default-host/retim-war/MPRkQLqFI4lwiC61-Y7YyHD4 from cluster

at org.jboss.as.clustering.lock.SharedLocalYieldingClusterLockManager.lock(SharedLocalYieldingClusterLockManager.java:439)

at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.acquireSessionOwnership(DistributedCacheManager.java:392)

at org.jboss.as.web.session.ClusteredSession.acquireSessionOwnership(ClusteredSession.java:523) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

at org.jboss.as.web.session.ClusteredSession.access(ClusteredSession.java:495) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

... 16 more

1. Re: Exception acquiring ownership

rhusar Oct 14, 2013 8:43 AM (in response to marcodanti)

You de facto need to use stiky sessions for stateful applications (and sticky-session-force should typically remain turned off, which is also the default).
Actions
2. Re: Exception acquiring ownership

marcodanti Oct 14, 2013 9:18 AM (in response to rhusar)

... sticky-session-force should typically remain turned off, which is also the default...

Actually, there is an open issue exactly on that: [MODCLUSTER-359] stickysession not working with httpd-2.4.6 / mod_cluster-1.2.5.Final - JBoss Issue Tracker)
Maybe your insight could help shedding some light on that issue...?
Actions
3. Re: Exception acquiring ownership

rhusar Oct 15, 2013 9:42 AM (in response to marcodanti)

If I am not mistaken, the point of [MODCLUSTER-359] stickysession not working with httpd-2.4.6 / mod_cluster-1.2.5.Final - JBoss Issue Tracker is that a new version of HTTPd was released and it broke compatibility with existing mod_cluster release.

So on 2.2.x version of HTTPd everything is working fine. We cannot guarantee other projects not breaking their own stuff...
Actions
4. Re: Exception acquiring ownership

marcodanti Oct 16, 2013 7:18 AM (in response to rhusar)

This particular comment [MODCLUSTER-359] stickysession not working with httpd-2.4.6 / mod_cluster-1.2.5.Final - JBoss Issue Tracker leaves me under the impression that my problem was due to a wrong configuration, and in fact if i declare *sticky-session-force="true"* things seem to work as expected. Still, to the best of my understanding sticky-session-force has a completely different purpose, but I haven't seen a reply to that, so I'm not sure how to proceed.
Actions
5. Re: Re: Exception acquiring ownership

rhusar Oct 16, 2013 7:58 AM (in response to marcodanti)

sticky-session-force="true" is for setups without session replication -- which is not your case. Leave it to default, or set to false.
Actions
6. Re: Exception acquiring ownership

marcodanti Oct 16, 2013 9:38 AM (in response to rhusar)

But that is the only thing that allows me to get sticky sessions to work (whatever the reason, may be it is an unintended side effect)... I really would like to hear what jfclere opinion is, but maybe he is away or too busy to follow up on [MODCLUSTER-359] stickysession not working with httpd-2.4.6 / mod_cluster-1.2.5.Final - JBoss Issue Tracker.
Actions
7. Re: Exception acquiring ownership

rhusar Oct 16, 2013 11:25 AM (in response to marcodanti)

Maybe re-read the docs to sort some of the confusion: mod_cluster Documentation
Actions

8. Re: Exception acquiring ownership

marcodanti Oct 16, 2013 12:00 PM (in response to rhusar)

Yes, I did read the documentation as the first thing. Then I read it again until I was reduced to tears, and then I turned to the community for help.

My understanding of the documentation does not explain what I see happening in this comment [MODCLUSTER-359] stickysession not working with httpd-2.4.6 / mod_cluster-1.2.5.Final - JBoss Issue Tracker, to which there were no replies, so I am back at square one.

stickySessionForce

sticky-session-force

false

Balancer

Indicates whether the httpd proxy should return an error in the event that the balancer is unable to route a request to the node to which it is stuck. This property is ignored if stickySession is false.

What exactly is the meaning of 'return an error', and how is that error supposed to be handled it is not clear to me.

More to the point, I don't understand why (all other thing being equal) the sticky session magically start to work only if I add sticky-session-force="true"

In my understanding of the doucmentation, the sticky session should work with no need to specify additional options, because this is supposed to be the default.

Now of course, I may have misinterpreted the documentation, and that is why this discussion is taking place. All I need is someone more into these things to clarify my doubts.

Your point of view seems to be that this is a problem inside httpd, which may be perfectly true. However, I am still one step behind because my problem is that

if i do not specify anything, sticky sessions do not work
but just by specifying sticky-session-force="true", then sticky sessions work

After all this, and after all the thread in MODCLUSTER-359, I still can't come up with an explanation and nobody seemed to come up with an answer.

9. Re: Exception acquiring ownership

jfclere Oct 17, 2013 3:20 AM (in response to marcodanti)

MODCLUSTER-359 is probably a bug... And it will be investigated when we have time. In the meantime fill free to help ;-)
If you look to http://svn.apache.org/repos/asf/httpd/httpd/branches/2.4.x/include/ap_mmn.h you will see that the has been changes in the proxy area and last time I had time to check with httpd-2.4.x is was with 2.4.2
Additionally there are new bundles available for download http://www.jboss.org/mod_cluster/downloads/1-2-6-Final-bin they should work correctly.
Actions
10. Re: Exception acquiring ownership

marcodanti Oct 17, 2013 4:27 AM (in response to jfclere)

MODCLUSTER-359 is probably a bug...
That is what I have been trying to point out all this time; it looks pretty clear that something is wrong and I am glad you finally acknowledge it. For the time being, I can go on using sticky-session-force=true until a fix will be available.
http://www.jboss.org/mod_cluster/downloads/1-2-6-Final-bin they should work correctly.
As I stated more that once in MODCLUSTER-359, I am already using 1.2.6.Final. And no, it does not work. Thanks anyway for replying.
Actions

Go to original post