org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to
johanvermeij Aug 26, 2012 2:58 AMI am testing clustering using JBoss 7.1.1 with a master and 2 slaves.
I am doing this in preparation for use in a production environment.
All machines run in domain mode.
The master is HP Proliant miniserver under Centos 6 64 bit.
This machine has 8Gb RAM.
It runs:
httpd mod_cluster load balancer
JBoss 7.1.1 master instance
Oracle 11g database
Slave 1 is a Linux PC under Centos 6 64 bit.
This machine has 4Gb RAM.
It runs:
JBoss 7.1.1 slave instance.
Slave 2 is an iMac under Mac OS.
This machine has 4Gb or RAM
It runs:
JBoss 7.1.1 slave instance.
mod_cluster is working fine, mod_cluster manager does not report any irregularities.
I use a jmeter injector to put load on the system but even when I use 2 terminals and hold the enter key down I get the problem described below.
With the master running as the only cluster member everything is OK and processing is very fast.
As soon as I bring up a slave I see the following symptoms:
1) The Oracle database seems to be locking up. The Oracle DBA who had a look at this defines the locking as a symptom of a problem in the JBoss configuration.
2) I see Infinispan/Hibernate/JGroups timeout related errors and warnings on both server and slave(2)
-------------------------------------------------
12:47:35,056 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (ajp-VBSHPServer-192.168.2.12-8259-35) ISPN000136: Execution error: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to slave2:server-three-slave-PC/hibernate
12:47:35,298 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp-VBSHPServer-192.168.2.12-8259-35) DDF0002,CashWithdrawal:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to slave2:server-three-slave-PC/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to slave2:server-three-slave-PC/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to slave2:server-three-slave-PC/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to slave2:server-three-slave-PC/hibernate
12:47:04,898 WARN [org.jboss.modules] (ajp-VBSHPServer-192.168.2.12-8259-47) Failed to define class org.infinispan.remoting.transport.jgroups.JGroupsTransport in Module "deployment.Capital.ear:main" from Service Module Loader: java.lang.LinkageError: Failed to link org/infinispan/remoting/transport/jgroups/JGroupsTransport (Module "deployment.Capital.ear:main" from Service Module Loader)
12:47:04,981 WARN [com.arjuna.ats.arjuna] (ajp-VBSHPServer-192.168.2.12-8259-47) ARJUNA012127: TwoPhaseCoordinator.afterCompletion - returned failure for SynchronizationImple< 0:ffffc0a8020c:-565e6ad8:5039b6cc:449, org.hibernate.engine.transaction.synchronization.internal.RegisteredSynchronization@23ca25c4 >
12:47:19,765 WARN [org.jboss.modules] (ajp-VBSHPServer-192.168.2.12-8259-29) Failed to define class org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher in Module "deployment.Capital.ear:main" from Service Module Loader: java.lang.LinkageError: Failed to link org/infinispan/remoting/transport/jgroups/CommandAwareRpcDispatcher (Module "deployment.Capital.ear:main" from Service Module Loader)
-------------------------------------------------
I have Googled what I could find and I have tried many things, all to no avail.
I attached httpd.conf, domain.xml, master host.xml, slave host.xml and master and slave server logs.
Your advise is much appreaciated.
-
server_log_slave.zip 78.6 KB
-
server_log_master.zip 122.1 KB
-
httpd.conf.zip 6.5 KB
-
host.xml_slave.zip 1.0 KB
-
host.xml_master.zip 1.0 KB
-
domain.xml 82.0 KB