1 2 Previous Next 25 Replies Latest reply on Sep 6, 2012 9:50 AM by ksermas

    How to cluster two Content Engine server to share CPU load

    ksermas

      I'm running jboss 5.1.  I've read the manual on how to cluster two Content Engine servers but it isn't very clear.

        • 1. Re: How to cluster two Content Engine server to share CPU load
          wdfink

          What you mean by 'cluster two content engine servers'? Can you explain a bit what you want to achieve.

          • 2. Re: How to cluster two Content Engine server to share CPU load
            ksermas

            Thank you sir!

             

            I have two servers (Server A and Server B)

             

            that run IBM's FileNet with Jboss running under it.  Server A has sixteen 2TB LUNs attached to it.  Server A then NFS exports the filesystems on those LUNs to Server B.

             

            We have four other 'Application Engine' servers that pull the files from Server A and Server B.

             

            The problem is that Server A's CPU load is ten times higher than the CPU load on Server B.  I want to make it so there is a load balance between the two servers.

            I was told that the only way to do that is to do a jboss cluster of Server A and Server B but the documentation is unclear to me as to where that is set.

             

            Currently we have a load-balancer setup on the network for Server A and Server B but it obviously is not working.  We have a clustered nodename

            for the two servers that is already set on this current load-balancer that is pingable.  From what I read, Server A and Server B just have to have

            a common ip in a certain config file but I'm not sure which file that is.  The documentation says there should be a deploy/cluster-service.xml

            file that contains this syntax:

             

            <mbean code="org.jboss.ha.framework.server.ClusterPartition"
                name="jboss:service=DefaultPartition">
                    
                <! -- Name of the partition being built -->
                <attribute name="PartitionName">
                    ${jboss.partition.name:DefaultPartition}
                </attribute>

             

                <! -- The address used to determine the node name -->
                <attribute name="NodeAddress">${jboss.bind.address}</attribute>

             

                <! -- Determine if deadlock detection is enabled -->
                <attribute name="DeadlockDetection">False</attribute>
                
                <! -- Max time (in ms) to wait for state transfer to complete.
                    Increase for large states -->
                <attribute name="StateTransferTimeout">30000</attribute>

             

                <! -- The JGroups protocol configuration -->
                <attribute name="PartitionConfig">
                    ... ...
                </attribute>
            </mbean>

             

            Is this all that is required to get Server A and Server B to be load balanced?

             

            Thanks....Danka!!

            • 3. Re: How to cluster two Content Engine server to share CPU load
              wdfink

              What kind of application run on the JBoss instances?

              Is the JBoss process consuming the CPU?

              • 4. Re: How to cluster two Content Engine server to share CPU load
                ksermas

                IBM FileNet

                • 5. Re: How to cluster two Content Engine server to share CPU load
                  ksermas

                  ...and yes the Jboss process is consuming at least half of the CPU on the one Content Engine server and it spikes to over 100% of the CPU.  The other is barely getting

                  over 10% of the CPU.

                  • 6. Re: How to cluster two Content Engine server to share CPU load
                    wdfink

                    How do you start the instances?

                    normally if you start "run.sh ... -g<partittionName> -u <udp multicast address> ..."

                    and -g -u is the same for both nodes you will have a log entry like "No. of Members #2" in your logfiles.

                     

                    But it depends on the applciation type whether it is loadbalanced.

                    If you use EJB remote access the loadbalancing is done by the remote proxies of the EJB.

                    If you use http access the load balancing must be done by an external LB like apache (mod_jk) or a HW solution.

                     

                    And I don't know what you meant by

                     

                    ksermas wrote:

                    .IBM FileNet

                     

                    Also you should check whether it is not the GC, the application does not respond and the CPU load is high.

                    If you not have it add "-verbose:gc -Xloggc:<LOGDIR>/gc_`date +%Y%m%d%H%M%S`.log -XX:+PrintGCDetails" to the JAVA_OPTS in run.conf to check whether the GarbageCollector will run at this time.

                    Also you might check the Threads if the CPU load is high whether the GC threads are active.

                    • 7. Re: How to cluster two Content Engine server to share CPU load
                      ksermas

                      Let me give you some background on my situation.  Our document servers servers store

                       

                      .pdf and.tiff documents on the LUNs attached to them.  A web server makes calls to these servers to retrieve these documents and display and sort them.  The traffic web servers generates to our document servers is via the IBM FileNet software that uses the jnp protocol through port 1099.  Our hardware load-balancer can't use that port so I have to make out two document servers to share that load from the web servers.

                       

                      The jboss instances are started with the startup file in /etc/init.d/jboss.  The instance on both servers is the same:

                       

                      jboss     3424     1  1 Jun04 ?        00:24:01 /usr/lib/jvm/java/bin/java -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.messaging.ServerPeerID=2 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/HeapDump -Dcom.sun.jndi.ldap.connect.pool.maxsize=20 -Dcom.sun.jndi.ldap.connect.pool.prefsize=10 -Dcom.sun.jndi.ldap.connect.pool.timeout=300000 -Djava.endorsed.dirs=/home/jboss/jboss-as/lib/endorsed -classpath /home/jboss/jboss-as/bin/run.jar:/usr/lib/jvm/java/lib/tools.jar org.jboss.Main -c ContentEngine -g ContentEngine -b 0.0.0.0

                       

                      The log file shows only one ip being clustered:

                       

                      ---------------------------------------------------------
                      GMS: address is 172.24.177.63:38056 (cluster=ContentEngine)
                      ---------------------------------------------------------
                      04:06:09,350 INFO  [ContentEngine] Number of cluster members: 1
                      04:06:09,351 INFO  [ContentEngine] Other members: 0
                      04:06:12,401 INFO  [PlatformMBeanServerRegistration] JBossCache MBeans were successfully registered to the platfor
                      m mbean server.
                      04:06:12,562 INFO  [STDOUT]
                      ---------------------------------------------------------
                      GMS: address is 172.24.177.63:38056 (cluster=ContentEngine-HAPartitionCache)
                      ---------------------------------------------------------
                      04:06:14,567 INFO  [RPCManagerImpl] Received new cluster view: [xx.xxx.177.63:38056|0] [xxx.xxx.177.63:38056]
                      04:06:14,569 INFO  [RPCManagerImpl] Cache local address is xxx.xxx.177.63:38056
                      04:06:14,572 INFO  [RPCManagerImpl] state was retrieved successfully (in 2.06 seconds)
                      04:06:14,594 INFO  [ComponentRegistry] JBoss Cache version: JBossCache 'Cascabel' 3.1.0.GA
                      04:06:14,594 INFO  [ContentEngine] Fetching serviceState (will wait for 30000 milliseconds):
                      04:06:14,595 INFO  [ContentEngine] State could not be retrieved (we are the first member in group)
                      04:06:14,716 INFO  [HANamingService] Started HAJNDI bootstrap; jnpPort=1100, backlog=50, bindAddress=/0.0.0.0
                      04:06:14,729 INFO  [DetachedHANamingService$AutomaticDiscovery] Listening on /0.0.0.0:1102, group=230.0.0.4, HA-JN
                      DI address=xxx.xxx.177.63:1100

                       

                      I'm wondering where is the configuration file to set-up the load-balancing between the two servers?  The way it looks now is that there is only one ip in that configuration.

                       

                      Thanks very much Herr Fink.

                      • 8. Re: How to cluster two Content Engine server to share CPU load
                        wdfink

                        So I suppose that you access the EJB's direct  and create the initial context with JNP from the web app.

                         

                        So in this case you have to fix the cluster problem, the servers must show the message "Number of cluster members: 2".

                        In that case the EJB proxy will do the loadbalancing after the lookup via "jnp:host:port".

                         

                        If you don't change the configuration of both instances the -g ContentEngine should separate the cluster from others (-u is default 230.0.0.4).

                        So as JGroups uses the UDP multicast it should work if you are in the same sub-network with both JBoss instances and have no firewall inbetween.

                         

                        You can check this with the JGroups test (no poblem in production) https://community.jboss.org/wiki/TestingJBoss

                        If you don't have a chance to get this work you might change to TCP stack instead. See JGroups configuration for that

                        • 9. Re: How to cluster two Content Engine server to share CPU load
                          ksermas

                          What is an EJB proxy?  I don't know where the configuration is for the 'instances'.  Does there have to be two separate jboss processes running on each ContentEngine server?

                          • 10. Re: How to cluster two Content Engine server to share CPU load
                            wdfink

                            An EJB proxy is the part at client side (aka stub or proxy) which is the implementation of your business interface, but include some code to handle the client-server communication.

                            The 'configuration of both instances' I meant whether you change ports/ip addresses by configuration file or stat parameter in JBoss.

                             

                            As I understand

                            * you start two different JBoss on two physical boxes.

                            * change jboss.messaging.ServerPeerID

                            * add a partition name with -g

                            * bind against all interfaces with -b 0.0.0.0

                             

                            But what I wonder about is that you have different IP addresses in your log

                            GMS: address is 172.24.177.63:38056 (cluster=ContentEngine)

                            Received new cluster view: [xx.xxx.177.63:38056|0] [xxx.xxx.177.63:38056]

                            Listening on /0.0.0.0:1102, group=230.0.0.4, HA-JNDI address=xxx.xxx.177.63:1100

                             

                            So I would use -b 172.24.x.x (or the 177.63 address). I don't see the other JBoss IP.

                            You have to ensure that multicast will work on your interfaces and there is no firewall (or a well configurued one) in between.

                            • 11. Re: How to cluster two Content Engine server to share CPU load
                              ksermas

                              It looks like we will be need to run one of our document servers as a Stub as our load-balancer can't handle the jnp traffice on port 1099 coming from our Application Engine servers.  I don't know

                              the exact file where these ips are configure but I did find something in the jmx-console that did.  I don't know the correct file to configure this in.  It seems like jboss has no constant set of files

                              in which to configure things.

                               

                               

                              The org.jboss.ha.jndi.HANamingService in the jmx-console had the configuration for the multi-cast ip 230.0.0.4 with a bind address of 0.0.0.0.  The jboss instances are started

                              with the ip 0.0.0.0 on both document servers.  The LoadBalancePolicy is set to org.jboss.ha.framework.interfaces.RoundRobin.

                               

                              The org.jboss.ha.framework.server.ClusterPartition has the CurrentView [xx.xx.177.62:1099, xx.xx.177.63:1099]  with a NodeAddress of 0.0.0.0

                               

                              The org.jboss.cache.jmx.CacheJmxWrapper has both ips of the document servers in it.  It looks like it is not being read by the server.

                              • 12. Re: How to cluster two Content Engine server to share CPU load
                                wdfink

                                In difference to your first post your instances are now clustered.

                                Does your SLSB's you deploy to both JBoss instances are marked as 'Clustered'?

                                If yes a lookup to one of the instances with jnp:x.x.x.x:1099 should give you a 'cluster view' of the SLSB and the requests should be balanced.

                                • 13. Re: How to cluster two Content Engine server to share CPU load
                                  ksermas

                                  I look at the output from 'ps -ef | grep jboss' and I don't see anything that says 'clustered' there.  How do I 'look up' the  SLSB instances with jnp:x.x.x.x:1099?

                                   

                                  Thanks for you help Herr Fink!

                                  • 14. Re: How to cluster two Content Engine server to share CPU load
                                    wdfink

                                    If you start a JBoss with '-g something' it mean that this instance will join the cluster with the name 'something'.

                                    In the logfile of each started instance you will s a message:

                                    "Number of cluster members: #"  => where # the current number of members is.

                                    So in your case after start the second instance you should see "Number of cluster members: 2" in both logfiles.

                                     

                                    Is that right?

                                     

                                    With JBoss5.1 you are able to lookup by setting JNP like "xx.xx.177.62:1099, xx.xx.177.63:1099" or only one of it (only necesarry for the initial lookup).

                                    After the first lookup you should store the proxy and reuse it every time and every request should go to the next JBoss instance follow the RoundRobin ...

                                    1 2 Previous Next