Session Replication - Concurrency Problems?

[zwitsch]

Hi,

I set up a cluster which consists of 2 nodes of Jboss AS 4.0.5 with an Apache 2.0 lodabalancer in front. I also set up farming and session replication and all this just works fine.

But out of my (distributable) application I am getting the following outputs:

15:27:26,358 INFO [CacheListener] Possible concurrency problem: Replicated version id 8 matches in-memory version for session EwvKAOhv-wpLJ+1T58te1A**

Every call of the app I am getting this message with the counter "id x matches" increasing by 1.
What is wrong? Has this something to do with the serialization of the objects put in the session?

The session replication though seems to work fine. Although failover works.

Thanks
Thomas

[brian.stansberry]

Are sticky sessions working properly?

Other than that, the following comment from the code explains the only situation I know of where the code block that logs that gets executed.

// This could be an issue but can happen legitimately in unusual
// circumstances, so just log something at INFO, not WARN

// Unusual circumstance: create session; don't touch session again
// until timeout period expired; fail over to another node after
// timeout but before session expiration thread has run. Existing
// session will be expired locally on new node and a new session created.
// When that session replicates, the version id will match the still
// existing cached session on the first node. Unlikely, but due
// to design of a unit test, it happens every testsuite run :-)
log_.info("Possible concurrency problem: Replicated version id " +
 version + " matches in-memory version for session " + realId);

Doesn't sound like what you described at all. :(

[baum_robson]

Hi

I'm having same problem.
The sticky_session property is disabled (siticky_session=0) in mod_jk config.
The Sessions are replicated, I can see they when I invoke the printDetails method in the jboss.cache:service=TomcatClusteringCache MBean,
but fail over doesn't work properly and I see the message described above in the log...

How you solved the problem?

Thanks

"Zwitsch" wrote:
Hi,

I set up a cluster which consists of 2 nodes of Jboss AS 4.0.5 with an Apache 2.0 lodabalancer in front. I also set up farming and session replication and all this just works fine.

But out of my (distributable) application I am getting the following outputs:

15:27:26,358 INFO [CacheListener] Possible concurrency problem: Replicated version id 8 matches in-memory version for session EwvKAOhv-wpLJ+1T58te1A**

Every call of the app I am getting this message with the counter "id x matches" increasing by 1.
What is wrong? Has this something to do with the serialization of the objects put in the session?

The session replication though seems to work fine. Although failover works.

Thanks
Thomas

[sparrow]

Hi !

I've discavered same problem too. Mode REPL_SYNC with SERIALIZABLE with SyncCommitPhase=true and SyncRollbackPhase=true doesn't always allow to replicate session succsessfully. I would glad for any advices. Thanks.

[alpheratz-jb]

Has anyone followed up on this issue?

I am also seeing this (behind a cisco loadbalancer), my session timeout is 40 mins and I don't think that I am hitting this limit...

2007-05-08 16:14:38,538 INFO
[org.jboss.ha.framework.server.ClusterFileTransfer
] Start push of file TCEEar.ear to cluster.
2007-05-08 16:14:40,658 INFO
[org.jboss.ha.framework.server.ClusterFileTransfer
] Finished push of file TCEEar.ear to cluster.
2007-05-08 16:22:32,036 INFO
[org.jboss.web.tomcat.tc5.session.CacheListener] Possible concurrency problem: Replicated version id 4 matches in-memory version f or session Q6LCsX8DWbft3ejbmvog7Q**
2007-05-08 16:22:40,274 INFO
[org.jboss.web.tomcat.tc5.session.CacheListener] Possible concurrency problem: Replicated version id 1 matches in-memory version f or session Q6LCsX8DWbft3ejbmvog7Q**
2007-05-08 16:22:45,256 INFO
[org.jboss.web.tomcat.tc5.session.CacheListener] Possible concurrency problem: Replicated version id 2 matches in-memory version f or session Q6LCsX8DWbft3ejbmvog7Q**

Cheers,

Alph

[snasto]

I am seeing the same on 4.2.0CR2 with REPL_SYNC and REPEATABLE_READ. WE are using Cicso load balancer, no stickiness fior now as we are trying to figure out how to stick based on jsessionid on the Cisco side.
Looks like we all have in common "REPL_SYNC" | no sticky LB, don't we?

[brian.stansberry]

sparrow: Don't use SERIALIZABLE; use REPEATABLE_READ. REPL_SYNC + SERIALIZABLE will almost for sure lead to lock timeout exceptions.


Everyone else -- is anyone seeing this problem with sticky sessions enabled and working properly? You have to use sticky sessions.

[sparrow]

[e-mail]

Ok, could you comment next. I've written servlet with session supporting. I did stress test with 100 queries in sticky_mode = false. Aproximatelly 5% replications failed with concurrency problem in any synchronization modes and levels of isolation. Other test I did with sticky_mode = true. During work I broke connection between servers and often replication failed. I used version JBoss 4.0.5 GA. Thanks.

[brian.stansberry]

Open another thread please, as your replication failing question is a different topic.

[jameshurff]

We are experiencing EXACTLY the same problem with Apache 2.0 proxying requests to JBoss 4.0.5 using mod_proxy_ajp. I'd like to participate further in the discussion/resolution as this is a critical issue for us.

Should this topic be opened under a different forum group? Or is Clustering/JBoss the right topic?

2007-05-09 10:36:05,645 INFO[org.jboss.web.tomcat.tc5.session.CacheListener] Possible concurrency problem: Replicated version id 50 matches -memory version for session ZyqAtRH+rizMxXjib4eMtA**

Is this possibly a mod_proxy_ajp issue? We've successfully used mod_jk to proxy requests to JBoss in the past.

[brian.stansberry]

Folks, no offense intended, but can someone on this thread please post saying "We're using sticky sessions, we understand how to configure them and as far as I know they are working correctly, but we are still seeing the "Possible concurrency problem" message."

Once I see that I'll look into it. Until I see that, I've got a lot of other things to do.

[davewebb]

Mr Stansberry,

We're using sticky sessions, we understand how to configure them and as far as I know they are working correctly, but we are still seeing the "Possible concurrency problem" message."

To elaborate a little on our configuration, we use jBoss4.0.5GA with Apache 2.2.4 on the front end using mod_proxy_ajp for load balancing.

There are actually 2 issues we are seeing:

1) The "Possible Concurrency Problem" message. We are using REPL_ASYNC and REPEATABLE_READ in the tc5 cluster setup. I should note that we used the same setup in our last environment which was 4.0.2 and everything worked fine.

2) Although we have mod_proxy_ajp setup to use sticky sessions, it doesnt appear to be doing that, hence the discovery of #1 above b/c we get NPEs accessing session attributes as mod_proxy distributes requests across multiple JBoss instances and the session attributes arent being replicated properly.

I think a solution to either of these 2 problems helps out everyone, although I recognize #2 is probably not within the scope of this thread. I think regardless of StickySession (which I fully understand), there is an issue with Session replication.

Hope this helps and we appreciate your time and attention to this forum. Thank you.

[brian.stansberry]

Thanks. Please clarify a bit:

1) You mention that sticky sessions are working correctly, but your #2 indicates they aren't. Are you finding your #1 problem even when you know sticky sessions are working?

2) In either case, when you see issues with requests failing over and the expected state not being there, do you have any feeling as to the interval between the requests and the load the servers are under?

You can get the "Possible Concurrency" message with this scenario:

1) Request 1 goes to server A and replicates to server B.
2) Request 2 goes to server A but replication to B hasn't completed.
3) Request 3 goes to server B. Replication of Request 2 is not completed, so Request 3 sees out of date session state, which could lead to NPE or might not cause any noticeable problem. When the session replicates back to server B, B will log the "Possible Concurrency" message.

With REPL_ASYNC, there is a higher risk of this, as Request 2 is allowed to complete before replication is completed. The greater the load the system is under, the longer it might take the replication to complete. The shorter the interval between request 2 and 3, the greater the chance the request hasn't completed replication.

[davewebb]

1) I am moving from mod_proxy_ajp back to mod_jk tonight so after than I can be more sure of sticky sessions working because I know they work for sure in mod_jk. I know they are not working correctly now, although I have it setup just like the Wiki explains (http://wiki.jboss.org/wiki/Wiki.jsp?page=UsingMod_proxyWithJBoss). I have this in my configuration:

ProxyPass /ATS balancer://bmcluster stickysession=JSESSIONID lbmethod=bytraffic nofailover=Off

But subsequent requests are getting distributed between server A and B.

2) The system is not under load at all, but I see your point. I originally setup REPL_ASYNC with StickySession because latency wasnt a consideration. I totally understand why the message is beging generated and I see 2 possible solutions

a) Get Sticky Sessions working properly with REPL_ASYNC
b) Use REPL_SYNC without sticky sessions

Do you concur?

[brian.stansberry]

Even with REPL_SYNC you need sticky sessions to work. It's possible to get two concurrent requests for the same session.

1) Think IFRAMEs or similar stuff. Nothing prevents browsers making simultaneous requests.

2) Similarly, the webapp can flush or even close the HttpResponse's OutputStream/PrintWriter, pushing content back to the browser before the session repl code considers the request done and replicates the session. Browser could parse that content and generate another request.

There's a JIRA to look into preventing the webapp closing the OutputStream/PrintWriter, but preventing a flush could break some applications.

I'll look at the mod_proxy bit in the morning.

If the system's not under load and you have a reasonable delay between requests, the session should be replicating in time. One thing I just observed last week with 4.2.0 though was that there was occasional > 500 ms latency in replication message transmission when the UDP.enable_bundling attribute was set to "true" in tc5-cluster.sar/META-INF/jboss-service.xml. With that set to false the testsuite REPL_ASYNC tests worked fine with a 100 ms delay between requests. The same thing probably holds true in 4.0.5.