How are these machines communicating? Via udp or tcp?
If using udp, can you switch to tcp and see if you have the same issues?
The easiest way to switch to tcp is to use a transport config like this:
<transport clusterName="infinispan-cluster"> <properties> <property name="configurationFile" value="tcp.xml"/> </properties> </transport>
And then pass the initial hosts to each node via system property, exampple:
Are the 2 nodes that work fine started in the same machine or different machines?
And the 3rd node? Is it a different machine?
Those machines communicate via default protocol (I believe it is udp).
When I changed that to tcp there was no improvement.
It doesn't matter if those two nodes that work fine are on the same machine or two different machines - it works ok. Problems start when I add more nodes, and here again it doesn't matter if those new nodes are created on one of those 2 machines that are already running or if I run it on 3rd machine. Anyway, after I add 3rd node, my cluster works a lot worse.
It is weird, because when I have 3 nodes or more, some of them still work fine, but some don't, and I can't find any dependencies which work better and which worse. It just seems to be quite random.
Could you post your full configuration file?
What's the Infinispan version you're using? Beta1?
I'd suggest enabling TRACE logging (see http://www.jboss.org/community/wiki/infinispantechnicalfaqs#How_can_I_enable_logging) in each of the 3 nodes and try to replicate the issue where you get those timeouts.
well I've been trying lots of configuration files, but as for now I'm using this one:
<?xml version="1.0" encoding="UTF-8"?> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0"> <global> <transport clusterName="infinispan-cluster"> <properties> <property name="configurationFile" value="udp.xml"/> </properties> </transport> </global> <default> <locking isolationLevel="REPEATABLE_READ" lockAcquisitionTimeout="20000" writeSkewCheck="false" concurrencyLevel="500"/> <!-- Used to register a transaction manager and participate in ongoing transactions. --> <transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup" syncRollbackPhase="false" syncCommitPhase="false" useEagerLocking="true"/> <clustering mode="distribution"> <l1 enabled="true" lifespan="10000"/> <hash numOwners="2" rehashWait="12000" rehashRpcTimeout="60000"/> <sync replTimeout="200000"/> </clustering> </default> </infinispan>
I know that here I use UDP, but I also tried TCP. And there were versions without transaction and locking part.
So that's it. Now I'm going to enable trace logging and we'll see ;)
I forgot. I'm using Beta1. Sorry for posting two replyes in a row, but I couldn't find "edit" button.
Hi, I don't think you should be using useEagerLocking unless you have specific case for it, i.e. cache a shared counter that can be modified by different nodes at the same time.
Please make sure that apart from noting anything specific of the log files, you actually send me the full TRACE log files to my email address which is my username.