8 Replies Latest reply on Mar 14, 2013 8:20 AM by softmahi

    Session replication in cluster

    vedanth k r Newbie

      Hello Everyone am struck with a problem regarding session replication in clustering. Server used: jboss-as-7.1.3.Final (built from git hub)

       

      Am able to setup cluster successfully with 2 nodes(Say master and slave) on 2 different linux machines along wit mod_cluster. The status of the nodes in mod_cluster-manager console shows OK.

      To test the cluster i have deployed a sample application on both the nodes with <distributable/> tag (JbossCluster application).

       

      Session replication is propagated successfully from master to slave in case if the instance of master goes down.

       

      The problem is that the session is not being propagated from slave to master if the instance of slave goes down. In this scenario i get a NullPointerException. Detailed log shown below.

      Firewall is disabled on both the machines. Attachment contains the host.xml's of both master and slave along with virtual host settings of httpd.conf

       

      ERROR [CoyoteAdapter] An exception or error occurred in the container during the request processing

      [JBoss] java.lang.NullPointerException

      [JBoss]      at org.jboss.web.tomcat.service.session.ClusteredSession.update(ClusteredSession.java:1097)

      [JBoss]      at org.jboss.web.tomcat.service.session.SessionBasedClusteredSession.update(SessionBasedClusteredSession.java:154)

      [JBoss]      at org.jboss.web.tomcat.service.session.distributedcache.impl.jbc.JBossCacheService.loadSession(JBossCacheService.java:344)

      [JBoss]      at org.jboss.web.tomcat.service.session.JBossCacheManager.loadSession(JBossCacheManager.java:1432)

      [JBoss]      at org.jboss.web.tomcat.service.session.JBossCacheManager.findSession(JBossCacheManager.java:491)

      [JBoss]      at org.apache.catalina.connector.Request.isRequestedSessionIdValid(Request.java:2160)

      [JBoss]      at org.apache.catalina.connector.CoyoteAdapter.parseSessionCookiesId(CoyoteAdapter.java:626)

      [JBoss]      at org.apache.catalina.connector.CoyoteAdapter.postParseRequest(CoyoteAdapter.java:551)

      [JBoss]      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:322)

      [JBoss]      at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:437)

      [JBoss]      at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:384)

      [JBoss]      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)

      [JBoss]      at java.lang.Thread.run(Thread.java:619)

       

       

      Thanks in advance

      Vedanth

        • 1. Re: Session replication in cluster
          Paul Ferraro Master

          Are you certain this stacktrace came from AS 7.1.3?

          Several of the classes (specifically the org.jboss.web.tomcat package) do not exist in AS7.  This looks more like an AS5 stacktrace.

          • 2. Re: Session replication in cluster
            vedanth k r Newbie

            Sorry for the wrong stack trace... Here is the actual one.

             

            [Server:server-three] 21:25:01,283 ERROR [org.apache.catalina.connector.CoyoteAdapter] (ajp-/192.168.2.41:8259-5) An exception or error occurred in the container during the request processing: java.lang.NullPointerException

            [Server:server-three]   at org.jboss.as.web.session.ClusteredSession.update(ClusteredSession.java:979) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

            [Server:server-three]   at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1377) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

            [Server:server-three]   at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:673) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

            [Server:server-three]   at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:84) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

            [Server:server-three]   at org.apache.catalina.connector.Request.doGetSession(Request.java:2618) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at org.apache.catalina.connector.Request.getSession(Request.java:2375) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:96) [jboss-as-web-7.1.3.Final.jar:7.1.3.Final]

            [Server:server-three]   at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:372) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:453) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.17.Final.jar:]

            [Server:server-three]   at java.lang.Thread.run(Thread.java:679) [rt.jar:1.6.0_22]

            [Server:server-three]

            • 3. Re: Session replication in cluster
              MJ Kim Newbie

              I am facing the same issue on my environment. (2 machins and 2 JVM on each machine, and I am not using mod_cluster. There is no mod_cluster settings at my standalone.xml)

               

              Am very looking foward to hearing good answer from someone.

               

              I attached my standalone.xml for you.

              • 4. Re: Session replication in cluster
                MJ Kim Newbie

                Hi, I've reviewed the relevant source codes tagged 7.1.3 Final at Github.

                 

                There is some logical problem at ClusterdSession.java.

                If a session outgoing sets its dirty flag to no, the receiver can't process session meta data because the meta data is null.

                 

                I modified the source code like below.

                 

                <SessionBasedClusterdSession.java>

                 

                @Override

                    protected OutgoingSessionGranularitySessionData getOutgoingSessionData() {

                        Map<String, Object> attrs = isSessionAttributeMapDirty() ? getSessionAttributeMap() : null;

                 

                       //DistributableSessionMetadata metadata = isSessionMetadataDirty() ? getSessionMetadata() : null;    <-- Original Source code

                        DistributableSessionMetadata metadata = getSessionMetadata(); // <-- Modified

                 

                        Long timestamp = attrs != null || metadata != null || getMustReplicateTimestamp() ? Long.valueOf(getSessionTimestamp())

                                : null;

                        return new OutgoingData(getRealId(), getVersion(), timestamp, metadata, attrs);

                    }

                 

                The commented line has the logical error.. I metioned above, the clean(no dirty flag) outgoing replication has null metadata. Its problem, null pointer exception at other node.

                 

                I can't undestand how the source code could be tagged as "FINAL". It still has bug.

                 

                Anyway, I modified this code and recomplieded, the whole problem has gone away.

                • 5. Re: Session replication in cluster
                  erasmo2 marciano2 Master

                  Are you sure that the master end slale are in cluster?

                   

                  Have you check on log file?

                  • 6. Re: Session replication in cluster
                    softmahi Newbie

                    Hi MJ Kim,

                     

                     

                    We are also facing same issue in jboss 7.2 .

                     

                    Still its persisting in 7.2 ?  And We have to incorporate ur code changes of <SessionBasedClusterdSession.java>

                     

                     

                     

                    container during the request processing: java.lang.NullPointerException

                    [Server:server-four]     at org.jboss.as.web.session.ClusteredSession.update(ClusteredSession.java:973) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.jboss.as.web.session.DistributableSessionManager.loadSession(DistributableSessionManager.java:1389) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:681) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.jboss.as.web.session.DistributableSessionManager.findSession(DistributableSessionManager.java:84) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.catalina.connector.Request.doGetSession(Request.java:2605) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.catalina.connector.Request.getSession(Request.java:2357) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:99) [jboss-as-web-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final.jar:7.2.0.Final]

                    [Server:server-four]     at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_13]

                    [Server:server-four]

                     

                    Need help on this ............

                    • 7. Re: Session replication in cluster
                      Paul Ferraro Master

                      FYI - We identified the cause of this issue earlier this week.  It's caused by: https://issues.jboss.org/browse/ISPN-2903

                      The fix is available in AS master.

                      • 8. Re: Session replication in cluster
                        softmahi Newbie

                        Hi ,

                         

                        The AS Master(JBoss 7.2 ) has some compliation issues.

                         

                        One more thing observed , when slave node is receiving application requests and thats goes down session is replicated to master Node without any issues.

                         

                        But if application requests are serving by Master Node , and that went down then there is no session replication.

                         

                        Gives the error 

                         

                        [Server:server-six] 06:53:43,684 ERROR [org.apache.catalina.connector] (ajp-/192.168.8.153:8259-12) JBWEB001018: An exception or error occurred in the container during the request processing: java.lang.NullPointerException

                         

                         

                        This issue will be resolved by taking  below bug Fix

                         

                        https://issues.jboss.org/browse/ISPN-2903

                         

                         

                        Thanks,

                        Mahesh

                         

                        Message was edited by: Mahesh H