Session replication in cluster

[vedanth]

Hello Everyone am struck with a problem regarding session replication in clustering. Server used: jboss-as-7.1.3.Final (built from git hub)

Am able to setup cluster successfully with 2 nodes(Say master and slave) on 2 different linux machines along wit mod_cluster. The status of the nodes in mod_cluster-manager console shows OK.

To test the cluster i have deployed a sample application on both the nodes with <distributable/> tag (JbossCluster application).

Session replication is propagated successfully from master to slave in case if the instance of master goes down.

The problem is that the session is not being propagated from slave to master if the instance of slave goes down. In this scenario i get a NullPointerException. Detailed log shown below.

Firewall is disabled on both the machines. Attachment contains the host.xml's of both master and slave along with virtual host settings of httpd.conf

ERROR [CoyoteAdapter] An exception or error occurred in the container during the request processing

[JBoss] java.lang.NullPointerException

[JBoss]      at org.jboss.web.tomcat.service.session.ClusteredSession.update(ClusteredSession.java:1097)

[JBoss]      at org.jboss.web.tomcat.service.session.SessionBasedClusteredSession.update(SessionBasedClusteredSession.java:154)

[JBoss]      at org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.JBossCacheService.loadSession(JBossCacheService.java:344)

[JBoss]      at org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1432)

[JBoss]      at org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:491)

[JBoss]      at org.apache.catalina.connector.Request.isRequestedSessionIdValid(Request.java:2160)

[JBoss]      at org.apache.catalina.connector.CoyoteAdapter.parseSessionCookiesId(CoyoteAdapter.java:626)

[JBoss]      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:551)

[JBoss]      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:322)

[JBoss]      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)

[JBoss]      at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)

[JBoss]      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

[JBoss]      at java.lang.Thread.run(Thread.java:619)

Thanks in advance

Vedanth

[pferraro]

Are you certain this stacktrace came from AS 7.1.3?

Several of the classes (specifically the org.jboss.web.tomcat package) do not exist in AS7.  This looks more like an AS5 stacktrace.

[vedanth]

Sorry for the wrong stack trace... Here is the actual one.

[Server:server-three] 21:25:01,283 ERROR [org.apache.catalina.connector.CoyoteAdapter] (ajp-/192.168.2.41:8259-5) An exception or error occurred in the container during the request processing: java.lang.NullPointerException

[Server:server-three]   at org.jboss.as.web.session.ClusteredSession.update(ClusteredSession.java:979) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

[Server:server-three]   at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1377) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

[Server:server-three]   at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:673) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

[Server:server-three]   at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:84) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

[Server:server-three]   at org.apache.catalina.connector.Request.doGetSession(Request.java:2618) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:96) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

[Server:server-three]   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.17.Final.jar:]

[Server:server-three]   at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_22]

[Server:server-three]

[mjhero]

I am facing the same issue on my environment. (2 machins and 2 JVM on each machine, and I am not using mod_cluster. There is no mod_cluster settings at my standalone.xml)

Am very looking foward to hearing good answer from someone.

I attached my standalone.xml for you.

[mjhero]

Hi, I've reviewed the relevant source codes tagged 7.1.3 Final at Github.

There is some logical problem at ClusterdSession.java.

If a session outgoing sets its dirty flag to no, the receiver can't process session meta data because the meta data is null.

I modified the source code like below.

<SessionBasedClusterdSession.java>

@Override

    protected OutgoingSessionGranularitySessionData getOutgoingSessionData() {

        Map<String, Object> attrs = isSessionAttributeMapDirty() ? getSessionAttributeMap() : null;

       //DistributableSessionMetadata metadata = isSessionMetadataDirty() ? getSessionMetadata() : null;    <-- Original Source code

        DistributableSessionMetadata metadata = getSessionMetadata(); // <-- Modified

        Long timestamp = attrs != null || metadata != null || getMustReplicateTimestamp() ? Long.valueOf(getSessionTimestamp())

                : null;

        return new OutgoingData(getRealId(), getVersion(), timestamp, metadata, attrs);

    }

The commented line has the logical error.. I metioned above, the clean(no dirty flag) outgoing replication has null metadata. Its problem, null pointer exception at other node.

I can't undestand how the source code could be tagged as "FINAL". It still has bug.

Anyway, I modified this code and recomplieded, the whole problem has gone away.

[erasmomarciano]

Are you sure that the master end slale are in cluster?

Have you check on log file?

[softmahi]

Hi MJ Kim,

We are also facing same issue in jboss 7.2 .

Still its persisting in 7.2 ?  And We have to incorporate ur code changes of <SessionBasedClusterdSession.java>

container during the request processing: java.lang.NullPointerException

[Server:server-four]     at org.jboss.as.web.session.ClusteredSession.update(ClusteredSession.java:973) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1389) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:681) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:84) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.catalina.connector.Request.doGetSession(Request.java:2605) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.catalina.connector.Request.getSession(Request.java:2357) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:99) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

[Server:server-four]     at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_13]

[Server:server-four]

Need help on this ............

[pferraro]

FYI - We identified the cause of this issue earlier this week.  It's caused by: https://issues.jboss.org/browse/ISPN-2903

The fix is available in AS master.

[softmahi]

Hi ,

The AS Master(JBoss 7.2 ) has some compliation issues.

One more thing observed , when slave node is receiving application requests and thats goes down session is replicated to master Node without any issues.

But if application requests are serving by Master Node , and that went down then there is no session replication.

Gives the error 

[Server:server-six] 06:53:43,684 ERROR [org.apache.catalina.connector] (ajp-/192.168.8.153:8259-12) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.NullPointerException

This issue will be resolved by taking  below bug Fix

https://issues.jboss.org/browse/ISPN-2903

Thanks,

Mahesh

Message was edited by: Mahesh H