7 Replies Latest reply on Oct 7, 2002 9:32 PM by integ

    balancing beans

    darkeye

      Hi,

      I'm a newbie to JBoss, and I've found the new clustering feature of JBoss 3 very exciting.

      I made some small trials, in the following setup:

      - DBS: a database server with a PostreSQL database
      - A: a machine running JBoss 3.0.1RC1
      - B: a machine running JBoss 3.0.1RC1

      Machines A and B are running in a cluster. They recognize each other, and I can deploy to either of them using farming. The configuration of these JBoss instances are the same.

      I created a simple container managed entity bean, that stores its info in the PostreSQL database of DBS. I deployed this bean successfully using farming.

      The idea behind the trial is to have lots of beans in JBoss, which get distributed among the cluster nodes. What I would expect is that if I create or instancieate a lot of beans, they would be spread approx. evenly between nodes A and B. To check this feature, I wrote a simple function into the test bean to tell what machine it is running on:

      public java.net.InetAddress
      getHost()
      {
      try {
      return java.net.InetAddress.getLocalHost();
      } catch ( java.net.UnknownHostException e ) {
      return null;
      }
      }


      Now what happens is very strange. When I create new beans (with previously non-existent primary keys), at the time of creation they get distributed evenly between machine A and B. But when I access aleardy existing beans with a findByPrimaryKey() for a findAll() function, all beans get to be either on machine A or machine B. They are never mixed between the two machines. It is also interesting, that even though at creation they are mixed between A and B, but right after that they all get re-grouped to one of the machines.


      Is this bevatiour intentional? Am I doing something wrong? Is it so that I didn't create too many beans to have the clustering algorithm bother about spreading the load?


      Akos

        • 1. Re: balancing beans
          slaboure

          This is dependant on the LoadBalancing policy that is set when you deploy your bean (see the doco). Furthermore, if you never you use the same home/remote proxy, the policy is reinitialized (because it is a new proxy) and may re-start always using the same target node.

          • 2. Re: balancing beans
            romalley

            I have a similar but different experience with jbossha-httpsession.

            Before I deployed the clustering stuff, the iis-redirect.dll load balancer was using round-robbin to load balance a new user to an app server. Sticky load balancing was in effect so that if I started 10 browsers, on average 5 users were 'stuck' to server A and 5 were stuck to server B. Of course, when server A was stopped, all 5 of those users were routed to B on their next request and they all got new sessions.

            But with jbossha-httpsession deployed, if I open 10 browsers, they ALL end up opening on one of the servers -- say server A. Then if I stop server A, those users get routed correctly to server B and they have the same session that they had on server A.

            So, the clustering failover is working great but my routine load balancing isn't utilizing both machines.

            I can't find anything on 'setting load balance policy' in the clustering document. How do I set up round robin for this service?

            Bob O'Malley

            • 3. Re: balancing beans
              slaboure

              Hello,

              You say that you are using an IIS DLL for your load-balancing. Then it is not JBoss but this DLL that you need to configure so that the load-balancing is done correctly.

              Cheers,


              Sacha

              • 4. Re: balancing beans
                romalley

                Sacha

                1) For the Jakarta Tomcat redirector dll plugin, there is a simple piece of round robin load balancing that Tomcat must do. I think this redirector does it's round robin work by looking at a portion of the session cookie. To control this piece of the cookie, there is field called 'jvmRoute' that Tomcat places at the end of the session ID. The Tomcat configuration approach is to put the jvmRoute attribute into the Engine entry in the tomcat4-service.xml file like this:





                Then, for the load balance redirector within the worker.properties file, you set up a worker with the same name like this:

                worker.serverOne.port=8009
                worker.serverOne.host=localhost
                worker.serverOne.type=ajp13
                worker.serverOne.cachesize=20
                worker.serverOne.lbfactor=2

                Then, you can see in my example the session.getId() method returns a value like this:

                7F9086AFF3E0E6DDE974452847CBB3BB.serverOne

                Sticky load balancing works -- all requests for this session are forwarded to the worker called serverOne.

                But when I use the jbossha-httpsession.sar, the session does NOT have the jvmRoute in it, instead the session looks like this with clustering turned on:

                dc9A8mxS88aHU8Z8C3sXfA**

                So, the IIS redirector does not know what to do because it does not find the .serverOne at the end of the session. It no longer does sticky load balancing.


                2) I was somewhat incorrect in the way I reported it's behavior. What it actually does, is to round robin every request when clustering is turned on. I think this is not so bad as long as our application doesn't modify the session for every request. I might leave it alone and leave the session replication so it replicates immediately (the default I think). We'll load test and see if there is a performance hit.

                3) Ideally, the jbossha-httpsession would continue to append the 'jvmRoute' of the 1st server hit to the end of the session id and I think sticky load balancing for the Tomcat redirector would be fixed.

                Bob O'Malley

                • 5. Re: balancing beans
                  romalley

                  Sacha,

                  I posted a detailed message but it did not make it to the forum (yet -- I still hope it does).

                  In summary, With Tomcat, there is a 'relationship' when using sticky load balancing between the Tomcat isapi_redirect.dll plugin, and the attribute called 'jvmRoute' set up in the Tomcat server configuration.

                  Call it 'proprietary load balancing'. You set up Tomcat with an Engine that has the 'jvmRoute' attribute in it (Jboss treats this as part of the 'config' Element for Tomcat). This jvmRoute name must match exactly the the name of a 'worker' you set up in your redirector.dll configuration.

                  When you do, each session id has the jvmRoute appended to it like this, where serverOne is my jvmRoute:

                  D1100F27231B789676CE555AF590DFED.serverOne

                  So, the isapi_redirect.dll load balancer just looks at the end of the sessionid and decides where to send the request. Simple.


                  I think this has been lost in the jbossha-httpsession.sar implementation. It might be Okay not to use sticky load balancing except that now I routinely get long exception traces in my console after a session expires. I think the server that is sharing the session does not know that the session has been removed. Here is the exception trace:

                  13:14:59,082 INFO [EmbeddedCatalinaServiceSX] javax.ejb.EJBException: Exception in setHttpSession: javax.ejb.FinderException: IlQ2aHC56M6P-F+KkFWOSA** does not exist
                  13:14:59,082 ERROR [LogInterceptor] TransactionRolledbackException, causedBy:
                  java.lang.NullPointerException
                  at java.io.ByteArrayInputStream.(ByteArrayInputStream.java:90)
                  at org.jboss.ejb.plugins.CMPClusteredInMemoryPersistenceManager.loadEntity(CMPClusteredInMemoryPersistenceManager.java:352)
                  at org.jboss.ejb.plugins.CMPPersistenceManager.loadEntity(CMPPersistenceManager.java:410)
                  at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invoke(EntitySynchronizationInterceptor.java:262)
                  at org.jboss.ejb.plugins.EntityInstanceInterceptor.invoke(EntityInstanceInterceptor.java:152)
                  at org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java:107)
                  at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:96)
                  at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:228)
                  at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:61)
                  at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:183)
                  at org.jboss.ejb.EntityContainer.invoke(EntityContainer.java:493)
                  at org.jboss.ejb.Container.invoke(Container.java:705)
                  at org.jboss.ejb.EntityContainer.invoke(EntityContainer.java:1055)
                  at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:491)
                  at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:98)
                  at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:102)
                  at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:73)
                  at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:189)
                  at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:76)
                  at $Proxy13.remove(Unknown Source)
                  at org.jboss.ha.httpsession.server.ClusteredHTTPSessionService.removeHttpSession(ClusteredHTTPSessionService.java:153)
                  at java.lang.reflect.Method.invoke(Native Method)
                  at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
                  at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:491)
                  at org.jboss.util.jmx.MBeanProxy.invoke(MBeanProxy.java:174)
                  at $Proxy26.removeHttpSession(Unknown Source)
                  at org.jboss.web.catalina.session.ClusterManager.removeSession(ClusterManager.java:401)
                  at org.jboss.web.catalina.session.ClusterManager.remove(ClusterManager.java:201)
                  at org.jboss.web.catalina.session.ClusteredSession.expire(ClusteredSession.java:565)
                  at org.jboss.web.catalina.session.ClusteredSessionCleanup.processExpires(ClusteredSessionCleanup.java:102)
                  at org.jboss.web.catalina.session.ClusteredSessionCleanup.run(ClusteredSessionCleanup.java:167)
                  at java.lang.Thread.run(Thread.java:484)
                  13:14:59,082 INFO [EmbeddedCatalinaServiceSX] Removing a session out of the clustered store failedjavax.ejb.EJBException: Exception in removeHttpSession: javax.transaction.TransactionRolledbackException: Load failed; CausedByException is: null; nested exception is:
                  javax.ejb.EJBException: Load failed; CausedByException is:
                  null


                  Bob O'Malley

                  • 6. Re: balancing beans
                    integ

                    Any updates on this one? As far as i know, Jboss3.0.3_tomcat4.1.12 does NOT have the jvmRoute appended to session id. So i guess "stickiness" is not going to be there with session replication?

                    • 7. Re: balancing beans
                      integ

                      oops!! was using old config docs..jvmRoute is no longer in server.xml but in tomcat4-service.xml. Works perfectly fine in Jboss3.0.3_tomcat4.0.5..