-
1. Re: How to cluster two Content Engine server to share CPU load
wdfink Jun 1, 2012 2:00 AM (in response to ksermas)What you mean by 'cluster two content engine servers'? Can you explain a bit what you want to achieve.
-
2. Re: How to cluster two Content Engine server to share CPU load
ksermas Jun 1, 2012 9:57 AM (in response to wdfink)Thank you sir!
I have two servers (Server A and Server B)
We have four other 'Application Engine' servers that pull the files from Server A and Server B.
The problem is that Server A's CPU load is ten times higher than the CPU load on Server B. I want to make it so there is a load balance between the two servers.
I was told that the only way to do that is to do a jboss cluster of Server A and Server B but the documentation is unclear to me as to where that is set.
Currently we have a load-balancer setup on the network for Server A and Server B but it obviously is not working. We have a clustered nodename
for the two servers that is already set on this current load-balancer that is pingable. From what I read, Server A and Server B just have to have
a common ip in a certain config file but I'm not sure which file that is. The documentation says there should be a deploy/cluster-service.xml
file that contains this syntax:
<mbean code="org.jboss.ha.framework.server.ClusterPartition"
name="jboss:service=DefaultPartition">
<! -- Name of the partition being built -->
<attribute name="PartitionName">
${jboss.partition.name:DefaultPartition}
</attribute><! -- The address used to determine the node name -->
<attribute name="NodeAddress">${jboss.bind.address}</attribute><! -- Determine if deadlock detection is enabled -->
<attribute name="DeadlockDetection">False</attribute>
<! -- Max time (in ms) to wait for state transfer to complete.
Increase for large states -->
<attribute name="StateTransferTimeout">30000</attribute><! -- The JGroups protocol configuration -->
<attribute name="PartitionConfig">
... ...
</attribute>
</mbean>Is this all that is required to get Server A and Server B to be load balanced?
Thanks....Danka!!
-
3. Re: How to cluster two Content Engine server to share CPU load
wdfink Jun 1, 2012 12:22 PM (in response to ksermas)What kind of application run on the JBoss instances?
Is the JBoss process consuming the CPU?
-
4. Re: How to cluster two Content Engine server to share CPU load
ksermas Jun 1, 2012 12:25 PM (in response to wdfink)IBM FileNet
-
5. Re: How to cluster two Content Engine server to share CPU load
ksermas Jun 5, 2012 10:27 AM (in response to wdfink)...and yes the Jboss process is consuming at least half of the CPU on the one Content Engine server and it spikes to over 100% of the CPU. The other is barely getting
over 10% of the CPU.
-
6. Re: How to cluster two Content Engine server to share CPU load
wdfink Jun 5, 2012 11:36 AM (in response to ksermas)How do you start the instances?
normally if you start "run.sh ... -g<partittionName> -u <udp multicast address> ..."
and -g -u is the same for both nodes you will have a log entry like "No. of Members #2" in your logfiles.
But it depends on the applciation type whether it is loadbalanced.
If you use EJB remote access the loadbalancing is done by the remote proxies of the EJB.
If you use http access the load balancing must be done by an external LB like apache (mod_jk) or a HW solution.
And I don't know what you meant by
ksermas wrote:
.IBM FileNet
Also you should check whether it is not the GC, the application does not respond and the CPU load is high.
If you not have it add "-verbose:gc -Xloggc:<LOGDIR>/gc_`date +%Y%m%d%H%M%S`.log -XX:+PrintGCDetails" to the JAVA_OPTS in run.conf to check whether the GarbageCollector will run at this time.
Also you might check the Threads if the CPU load is high whether the GC threads are active.
-
7. Re: How to cluster two Content Engine server to share CPU load
ksermas Jun 5, 2012 12:10 PM (in response to wdfink)Let me give you some background on my situation. Our document servers servers store
The jboss instances are started with the startup file in /etc/init.d/jboss. The instance on both servers is the same:
jboss 3424 1 1 Jun04 ? 00:24:01 /usr/lib/jvm/java/bin/java -Xms2048m -Xmx2048m -XX:MaxPermSize=512m -Dorg.jboss.resolver.warning=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djboss.messaging.ServerPeerID=2 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/HeapDump -Dcom.sun.jndi.ldap.connect.pool.maxsize=20 -Dcom.sun.jndi.ldap.connect.pool.prefsize=10 -Dcom.sun.jndi.ldap.connect.pool.timeout=300000 -Djava.endorsed.dirs=/home/jboss/jboss-as/lib/endorsed -classpath /home/jboss/jboss-as/bin/run.jar:/usr/lib/jvm/java/lib/tools.jar org.jboss.Main -c ContentEngine -g ContentEngine -b 0.0.0.0
The log file shows only one ip being clustered:
---------------------------------------------------------
GMS: address is 172.24.177.63:38056 (cluster=ContentEngine)
---------------------------------------------------------
04:06:09,350 INFO [ContentEngine] Number of cluster members: 1
04:06:09,351 INFO [ContentEngine] Other members: 0
04:06:12,401 INFO [PlatformMBeanServerRegistration] JBossCache MBeans were successfully registered to the platfor
m mbean server.
04:06:12,562 INFO [STDOUT]
---------------------------------------------------------
GMS: address is 172.24.177.63:38056 (cluster=ContentEngine-HAPartitionCache)
---------------------------------------------------------
04:06:14,567 INFO [RPCManagerImpl] Received new cluster view: [xx.xxx.177.63:38056|0] [xxx.xxx.177.63:38056]
04:06:14,569 INFO [RPCManagerImpl] Cache local address is xxx.xxx.177.63:38056
04:06:14,572 INFO [RPCManagerImpl] state was retrieved successfully (in 2.06 seconds)
04:06:14,594 INFO [ComponentRegistry] JBoss Cache version: JBossCache 'Cascabel' 3.1.0.GA
04:06:14,594 INFO [ContentEngine] Fetching serviceState (will wait for 30000 milliseconds):
04:06:14,595 INFO [ContentEngine] State could not be retrieved (we are the first member in group)
04:06:14,716 INFO [HANamingService] Started HAJNDI bootstrap; jnpPort=1100, backlog=50, bindAddress=/0.0.0.0
04:06:14,729 INFO [DetachedHANamingService$AutomaticDiscovery] Listening on /0.0.0.0:1102, group=230.0.0.4, HA-JN
DI address=xxx.xxx.177.63:1100I'm wondering where is the configuration file to set-up the load-balancing between the two servers? The way it looks now is that there is only one ip in that configuration.
Thanks very much Herr Fink.
-
8. Re: How to cluster two Content Engine server to share CPU load
wdfink Jun 5, 2012 1:16 PM (in response to ksermas)So I suppose that you access the EJB's direct and create the initial context with JNP from the web app.
So in this case you have to fix the cluster problem, the servers must show the message "Number of cluster members: 2".
In that case the EJB proxy will do the loadbalancing after the lookup via "jnp:host:port".
If you don't change the configuration of both instances the -g ContentEngine should separate the cluster from others (-u is default 230.0.0.4).
So as JGroups uses the UDP multicast it should work if you are in the same sub-network with both JBoss instances and have no firewall inbetween.
You can check this with the JGroups test (no poblem in production) https://community.jboss.org/wiki/TestingJBoss
If you don't have a chance to get this work you might change to TCP stack instead. See JGroups configuration for that
-
9. Re: How to cluster two Content Engine server to share CPU load
ksermas Jun 5, 2012 3:02 PM (in response to wdfink)What is an EJB proxy? I don't know where the configuration is for the 'instances'. Does there have to be two separate jboss processes running on each ContentEngine server?
-
10. Re: How to cluster two Content Engine server to share CPU load
wdfink Jun 6, 2012 2:16 AM (in response to ksermas)An EJB proxy is the part at client side (aka stub or proxy) which is the implementation of your business interface, but include some code to handle the client-server communication.
The 'configuration of both instances' I meant whether you change ports/ip addresses by configuration file or stat parameter in JBoss.
As I understand
* you start two different JBoss on two physical boxes.
* change jboss.messaging.ServerPeerID
* add a partition name with -g
* bind against all interfaces with -b 0.0.0.0
But what I wonder about is that you have different IP addresses in your log
GMS: address is 172.24.177.63:38056 (cluster=ContentEngine)
Received new cluster view: [xx.xxx.177.63:38056|0] [xxx.xxx.177.63:38056]
Listening on /0.0.0.0:1102, group=230.0.0.4, HA-JNDI address=xxx.xxx.177.63:1100
So I would use -b 172.24.x.x (or the 177.63 address). I don't see the other JBoss IP.
You have to ensure that multicast will work on your interfaces and there is no firewall (or a well configurued one) in between.
-
11. Re: How to cluster two Content Engine server to share CPU load
ksermas Jun 13, 2012 6:18 PM (in response to wdfink)It looks like we will be need to run one of our document servers as a Stub as our load-balancer can't handle the jnp traffice on port 1099 coming from our Application Engine servers. I don't know
the exact file where these ips are configure but I did find something in the jmx-console that did. I don't know the correct file to configure this in. It seems like jboss has no constant set of files
in which to configure things.
The org.jboss.ha.jndi.HANamingService in the jmx-console had the configuration for the multi-cast ip 230.0.0.4 with a bind address of 0.0.0.0. The jboss instances are started
with the ip 0.0.0.0 on both document servers. The LoadBalancePolicy is set to org.jboss.ha.framework.interfaces.RoundRobin.
The org.jboss.ha.framework.server.ClusterPartition has the CurrentView [xx.xx.177.62:1099, xx.xx.177.63:1099] with a NodeAddress of 0.0.0.0
The org.jboss.cache.jmx.CacheJmxWrapper has both ips of the document servers in it. It looks like it is not being read by the server.
-
12. Re: How to cluster two Content Engine server to share CPU load
wdfink Jun 14, 2012 2:13 AM (in response to ksermas)In difference to your first post your instances are now clustered.
Does your SLSB's you deploy to both JBoss instances are marked as 'Clustered'?
If yes a lookup to one of the instances with jnp:x.x.x.x:1099 should give you a 'cluster view' of the SLSB and the requests should be balanced.
-
13. Re: How to cluster two Content Engine server to share CPU load
ksermas Jun 14, 2012 10:29 AM (in response to wdfink)I look at the output from 'ps -ef | grep jboss' and I don't see anything that says 'clustered' there. How do I 'look up' the SLSB instances with jnp:x.x.x.x:1099?
Thanks for you help Herr Fink!
-
14. Re: How to cluster two Content Engine server to share CPU load
wdfink Jun 14, 2012 10:50 AM (in response to ksermas)If you start a JBoss with '-g something' it mean that this instance will join the cluster with the name 'something'.
In the logfile of each started instance you will s a message:
"Number of cluster members: #" => where # the current number of members is.
So in your case after start the second instance you should see "Number of cluster members: 2" in both logfiles.
Is that right?
With JBoss5.1 you are able to lookup by setting JNP like "xx.xx.177.62:1099, xx.xx.177.63:1099" or only one of it (only necesarry for the initial lookup).
After the first lookup you should store the proxy and reuse it every time and every request should go to the next JBoss instance follow the RoundRobin ...