JGroups timing call inomalities
axelerator Oct 16, 2009 3:10 PMHey,
we have
JBoss 4.2.3.GA
JGroups 2.6.10.merge
in development. We have a component, which is deployed on all cluster nodes and can be called externally and from within the server. If that component is called on the slave node, it redirects the call to the master (callMethodOnCoordinator). If called on the master node, it just continues.
What we see is that if we call that component from the outside on the master, the response time is very constant around 90ms. If called on the slave node, where this is relayed to the master, the response time is somewhere between 200ms to 10000ms.
I don't really get, why there is such a huge difference (besides 10sec to be ridiculous).
Here is the cluster service config we use:
<?xml version="1.0" encoding="UTF-8"?> <server> <mbean code="org.jboss.ha.framework.server.ClusterPartition" name="jboss:service=${jboss.partition.name:DefaultPartition}"> <attribute name="PartitionName">${jboss.partition.name:DefaultPartition}</attribute> <attribute name="NodeAddress">${jboss.bind.address}</attribute> <attribute name="DeadlockDetection">False</attribute> <attribute name="StateTransferTimeout">30000</attribute> <attribute name="MethodCallTimeout">300000</attribute> <attribute name="PartitionConfig"> <Config> <UDP mcast_addr="${jboss.partition.udpGroup:228.1.2.3}" mcast_port="${jboss.hapartition.mcast_port:45566}" tos="8" ucast_recv_buf_size="20000000" ucast_send_buf_size="640000" mcast_recv_buf_size="25000000" mcast_send_buf_size="640000" loopback="false" discard_incompatible_packets="true" use_incoming_packet_handler="true" max_bundle_size="60000" max_bundle_timeout="30" ip_ttl="${jgroups.udp.ip_ttl:8}" enable_bundling="false" receive_on_all_interfaces="true" send_on_all_interfaces="true" use_concurrent_stack="true" thread_pool.enabled="true" thread_pool.min_threads="80" thread_pool.max_threads="100" thread_ss.ha.framework.interfaces.RoundRobin</attribute> </mbean> <mbean code="org.jboss.invocation.unified.server.UnifiedInvokerHA" name="jboss:service=invoker,type=unifiedha"> <depends>jboss:service=TransactionManager</depends> <depends optional-attribute-name="Connector" proxy-type="attribute">jboss.remoting:service=Connector,transport=socket</depends> <depends>jboss:service=${jboss.partition.name:DefaultPartition}</depends> </mbean> <mbean code="org.jboss.invocation.jrmp.server.JRMPInvokerHA" name="jboss:service=invoker,type=jrmpha"> <attribute name="ServerAddress">${jboss.bind.address}</attribute> <attribute name="RMIObjectPort">4447</attribute> <depends>jboss:service=Naming</depends> </mbean> <mbean code="org.jboss.invocation.pooled.server.PooledInvokerHA" name="jboss:service=invoker,type=pooledha"> <attribute name="NumAcceptThreads">1</attribute> <attribute name="MaxPoolSize">300</attribute> <attribute name="ClientMaxPoolSize">300</attribute> <attribute name="SocketTimeout">60000</attribute> <attribute name="ServerBindAddress">${jboss.bind.address}</attribute> <attribute name="ServerBindPort">4448</attribute> <attribute name="ClientConnectAddress">${jboss.bind.address}</attribute> <attribute name="ClientConnectPort">0</attribute> <attribute name="EnableTcpNoDelay">false</attribute> <depends optional-attribute-name="TransactionManagerService">jboss:service=TransactionManager</depends> <depends>jboss:service=Naming</depends> </mbean> <mbean code="org.jboss.cache.invalidation.bridges.JGCacheInvalidationBridge" name="jboss.cache:service=InvalidationBridge,type=JavaGroups"> <depends optional-attribute-name="ClusterPartition" proxy-type="attribute">jboss:service=${jboss.partition.name:DefaultPartition}</depends> <depends>jboss.cache:service=InvalidationManager</depends> <attribute name="InvalidationManager">jboss.cache:service=InvalidationManager</attribute> <attribute name="BridgeName">DefaultJGBridge</attribute> </mbean> </server>
Any idea how this can happen? Something with the slave-master communication seems to go wrong.
We tested the following constellation:
Master only -> everything fine,
Master+Slave, Master queried -> everyhting fine,
Master+Slave, Slave queried -> Delays and timeouts
Master+Slave, Both qeueried -> Delays and timeouts
So I conclude that JGroups must be involved here ...
Any help is greatly appreciated !!!
Best regards and cheers,
Axelerator