Corrupted HttpSession with clustered JBoss 7.1.1

[safetytrick]

I'm seeing problems with HttpSession clustering in JBoss 7.1.1, occasionally a user's session will become corrupted and until they clear their JSESSIONID they get a blank result from every page. The stack trace for my issue is very similar to: https://issues.jboss.org/browse/AS7-4932

14:48:23,736 ERROR [org.apache.catalina.connector.CoyoteAdapter] (http-app03-10.41.108.33-8080-14) An exception or error occurred in the container during the request processing: java.lang.NullPointerException
     at org.jboss.as.web.session.ClusteredSession.update(ClusteredSession.java:973)
     at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1392)
     at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:686)
     at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:85)
     at org.apache.catalina.connector.CoyoteAdapter.isSessionIdValid(CoyoteAdapter.java:710)
     at org.apache.catalina.connector.CoyoteAdapter.parseSessionCookiesId(CoyoteAdapter.java:683)
     at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:609)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:365)
     at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:897)
     at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:626)
     at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2039)
     at java.lang.Thread.run(Unknown Source) [rt.jar:1.6.0_30]

If I'm understanding correctly the workaround for this issue is to disable passivation. In the JBoss 6 bug the issue is marked not critical because passivation is not the default but in the infinispan cache store config within standalone.xml passivation is the default? Is there a different layer of passivation that can be configured under jboss-web.xml? And if so I can work around this issue by setting <file-store passivation="false" /> ?

[eric2]

I work with Michael and I'm trying to provide a little more information on the problem.

Here is the source for that tag, the line that is bolded has the NPE. In summary the ClusteredSession recieves and update from another node however the payload is completely missing the DistributableSessionMetadata. The real solution should involve never transmitting the updated session w/o. But if this can't be done is is possible to update this method to just ignore or invalidate/expire this version of the session?

/**

     * {@inheritDoc}

     */

    public void update(IncomingDistributableSessionData sessionData) {

        assert sessionData != null : MESSAGES.nullSessionData();

        this.version.set(sessionData.getVersion());

        long ts = sessionData.getTimestamp();

        this.lastAccessedTime = this.thisAccessedTime = ts;

        this.timestamp.set(ts);

        DistributableSessionMetadata md = sessionData.getMetadata();

        // TODO -- get rid of these field and delegate to metadata

       this.id = md.getId();

        this.creationTime = md.getCreationTime();

        this.maxInactiveInterval = md.getMaxInactiveInterval();

        this.isNew = md.isNew();

        this.isValid = md.isValid();

        this.metadata = md;

        // Get our id without any jvmRoute appended

        parseRealId(id);

        // We no longer know if we have an activationListener

        hasActivationListener = null;

        // If the session has been replicated, any subsequent

        // access cannot be the first.

        this.firstAccess = false;

        // We don't know when we last replicated our timestamp. We may be

        // getting called due to activation, not deserialization after

        // replication, so this.timestamp may be after the last replication.

        // So use the creation time as a conservative guesstimate. Only downside

        // is we may replicate a timestamp earlier than we need to, which is not

        // a heavy cost.

        this.lastReplicated = this.creationTime;

        this.clusterStatus = new ClusteredSessionManagementStatus(this.realId, true, null, null);

        checkAlwaysReplicateTimestamp();

        populateAttributes(sessionData.getSessionAttributes());

        // TODO uncomment when work on JBAS-1900 is completed

        // // Session notes -- for FORM auth apps, allow replicated session

        // // to be used without requiring a new login

        // // We use the superclass set/removeNote calls here to bypass

        // // the custom logic we've added

        // String username = (String) in.readObject();

        // if (username != null)

        // {

        // super.setNote(Constants.SESS_USERNAME_NOTE, username);

        // }

        // else

        // {

        // super.removeNote(Constants.SESS_USERNAME_NOTE);

        // }

        // String password = (String) in.readObject();

        // if (password != null)

        // {

        // super.setNote(Constants.SESS_PASSWORD_NOTE, password);

        // }

        // else

        // {

        // super.removeNote(Constants.SESS_PASSWORD_NOTE);

        // }

        // We are no longer outdated vis a vis distributed cache

        this.outdatedTime = 0;

        // Requests must publish our full state back to the cluster in case anything got dropped.

        this.requireFullReplication();

    }

[safetytrick]

Ouch, disabling passivation is a huge performance hit. Even on a small cluster (3 nodes) it was very slow.

[ctomc]

hey guys,

can you try this with latest nightly builds(7.2.x)

as there ware many changes (fixes) in clustering after 7.1.1 release.

--

tomaz

[eric2]

This is a production server so we can't run a nightly build. We believe we have fixed the problem by wrapping the session.update(data) with a try catch in patching org.jboss.as.web.session.DistributableSessionManager

Coce:

                      try {

                                session.update(data);

                            } catch (NullPointerException npe) {

                                log.error("Session Meta Data was null for session: " +realId , npe);

                                this.distributedCacheManager.removeSession(realId);

                                session = null;

                            }

[jaikiran]

One of the reasons why we ask the users to try nightly builds is to check if the bug has been already fixed in the latest source code and if not, file a new bug so that it gets fixed. We do understand that running nightly builds in production is not an option and in fact we don't recommend that. However, if you are able to reproduce this issue on a development environment against 7.1.1, then give it a try against the nightly build in your development environment to check if it's still an issue with our latest AS7 code. If yes, we'll have to file a JIRA to fix it.

[eric2]

I completely understand where you are coming from, unfortnuatlly we haven't been able to reproduce this out side of production. We have tried, we simulated a production like enviornment and hit it wilth hundreds of thousands of requests with no luck. It still happens frequenetly in production. We did end up patching the class listed above and that did solve the problem.