Session being stolen / assigned to wrong person

[javaspack]

Problem configuration:
JBoss 4.2.1, Apache 2.0.59, mod_jk 1.2.25, java 1.6.02

We have a problem were our website will run great 99% of the time. Then it starts all sorts of weirdness (partial pages, text only, completely wrong pages)

Then the problem will just disappear with no intervention. It usually doesn't last more than 5 minutes.

We have tracked the problem down to requests not being handled properly. In looking through the individual requests being processed, we can see the JSESSIONID going from one user's IP to a different user. The wrong user then becomes the original user as though they had logged in.

The worst problem is that orders are being placed under the wrong account.

Originally believing it to be an Apache issue, we stripped out all the Virtual Host settings, removed load balancing and made it as basic as possible.

Finally, we reverted JBoss to 4.0.3SP1 about 2 weeks ago and the problem hasn't happened again. I don't think ever we made it 72 hours on JBoss 4.2.1.

So, this looks to us to be a Tomcat or JBoss issue. It looks like JBoss 4.2.1 uses either Tomcat 6.0.10 or 6.0.13.

I know JBoss is now at 4.2.2 and we are planning to try that next, but it doesn't look like it uses a different version of Tomcat, so we aren't very hopeful.

Anybody else experienced this or does anybody know of any changes between the versions that could cause this?

[javaspack]

It has been almost 6 weeks since we reverted our JBoss to version
4.0.3sp1 and on that server the problem hasn't returned. That seems conclusive that the problem doesn't exist in 4.0.3sp1.

We upgraded a different JBoss server from 4.2.1 to 4.2.2 about 2 weeks ago and it still has the same problems.

We also upgraded Java to 1.6.0_04 on both servers.

One other thing to add: We thought maybe load might be a cause, but the new JBoss server has very light load and still has the problem.

Anybody have any suggestions?

[jfclere]

That looks like a mod_jk problem more than a JBossAS one.

[javaspack]

I don't follow the logic. We have the same version of mod_jk, but it works with JBoss 4.0.3sp1, but not 4.2.2?

Or are you saying it is a mod_jk problem on the JBoss side with the AJP?

[jfclere]

According to your problem description it seems there could be 2 problems the JVMRoute is of the sessionid is not handled by mod_jk or the sessions are not distributed between the nodes of the cluster.

Additionaly the JBossWeb used by 4.2.2-GA is affected by http://jira.jboss.com/jira/browse/JBPAPP-366.

[javaspack]

We don't have a clustered environment. We used to have 2 JBoss servers that were load balanced, but not distributed (we don't need session replication), but like I said, we removed all that to simplify the configuration.

Most of the time, we have no problem. Then, for no observable reason, weird stuff happens. Then it is stops.

I looked at the JIRA issue, which deals with CLOSE_WAIT. Wouldn't that be more of a memory leak? I can see how over time all those connections could stack up.

If this is our problem, it would need an event that causes it. And then something that fixes it without intervention.

Example:
Lots of AJP connections stack up over time. Then they get garbage collected or cleaned up, or a JBoss service restarts to clean up the problem?

Is that what we are talking about here?

[jfclere]

So according to your configuration and applications a user session can start in one server and go on in the other, correct?
How do you recognize session created in one server or the other? Do you use JVMRoute?

The problems caused by the CLOSE_WAIT issue don't get fixed without intervention.

[javaspack]

Incorrect. We currently have just ONE JBoss server. When we had two, we used JvmRoute. But like I said, we don't use a second server anymore because we thought that might be the problem.

The problem is that a user session can be transfered to a different user session.

Example:
1. User logs in.
2. Adds stuff to cart.
3. Goes to checkout.
4. User becomes a different user with that person's sessionId.
5. Order is logged under wrong person.

By monitoring every single request that comes to our site, we have been able to see exactly what the users are doing.

One time, while the problem was happening, I went to our website and it gave me another users sessionId. I didn't have to log in. Somehow it got me confused and assigned me their sessionId.

Since we have gone back to 4.0.3sp1, we haven't seen it. But it appears to exist in all the 4.2.x versions.

[joeyxxx]

Javaspack,
Did you evr find a resolution for this?

[javaspack]

No, we went back to JBoss 4.0.3sp1. Best guess, there is a problem with Tomcat 6. The JBoss versions that uses Tomcat 5.5 don't have the problem.

We might try it again if we can get management to go along now that we have upgraded to Apache 2.0.63 and mod_jk 1.2.26.

We can't stay on 4.0.3sp1 forever, but 4.2.2 is still the latest version.