3 nodes problem

[mcmati]

Hi,

I've got another problem. When I use 2 nodes over Lan, everything seems to be ok. I can add data relativelly fast. But when I start another node then things aren't so great. I have to wait reeaaaallllly long before some information is added, and sometimes it isn't added at all (replication timeout error even when I set: sync replTimeout="200000"). I use distribution mode, with hash numOwners="2" and I use GuiDemo provided with infinispan. Do you have any idea what may be causing that?

Best regards,
Martin

[galder.zamarreno]

How are these machines communicating? Via udp or tcp?
If using udp, can you switch to tcp and see if you have the same issues?
The easiest way to switch to tcp is to use a transport config like this:

<transport clusterName="infinispan-cluster">
 <properties>
 <property name="configurationFile" value="tcp.xml"/>
 </properties>
</transport>

And then pass the initial hosts to each node via system property, exampple:

-Djgroups.tcpping.initial_hosts=host1[7800],host2[7800],host3[7800]

Are the 2 nodes that work fine started in the same machine or different machines?
And the 3rd node? Is it a different machine?

[mcmati]

Those machines communicate via default protocol (I believe it is udp).
When I changed that to tcp there was no improvement.

It doesn't matter if those two nodes that work fine are on the same machine or two different machines - it works ok. Problems start when I add more nodes, and here again it doesn't matter if those new nodes are created on one of those 2 machines that are already running or if I run it on 3rd machine. Anyway, after I add 3rd node, my cluster works a lot worse.

It is weird, because when I have 3 nodes or more, some of them still work fine, but some don't, and I can't find any dependencies which work better and which worse. It just seems to be quite random.

[galder.zamarreno]

Could you post your full configuration file?
What's the Infinispan version you're using? Beta1?
I'd suggest enabling TRACE logging (see http://www.jboss.org/community/wiki/infinispantechnicalfaqs#How_can_I_enable_logging) in each of the 3 nodes and try to replicate the issue where you get those timeouts.

[mcmati]

well I've been trying lots of configuration files, but as for now I'm using this one:

<?xml version="1.0" encoding="UTF-8"?>

<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0">
 <global>
 <transport clusterName="infinispan-cluster">
 <properties>
 <property name="configurationFile" value="udp.xml"/>
 </properties>
 </transport>
 </global>

 <default>

 <locking
 isolationLevel="REPEATABLE_READ"
 lockAcquisitionTimeout="20000"
 writeSkewCheck="false"
 concurrencyLevel="500"/>

 <!--
 Used to register a transaction manager and participate in ongoing transactions.
 -->
 <transaction
 transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
 syncRollbackPhase="false"
 syncCommitPhase="false"
 useEagerLocking="true"/>

 <clustering mode="distribution">
 <l1 enabled="true" lifespan="10000"/>
 <hash numOwners="2" rehashWait="12000" rehashRpcTimeout="60000"/>
 <sync replTimeout="200000"/>
 </clustering>
 </default>
</infinispan>

I know that here I use UDP, but I also tried TCP. And there were versions without transaction and locking part.

So that's it. Now I'm going to enable trace logging and we'll see ;)


Best regards,
Martin

[mcmati]

I forgot. I'm using Beta1. Sorry for posting two replyes in a row, but I couldn't find "edit" button.

[galder.zamarreno]

Hi, I don't think you should be using useEagerLocking unless you have specific case for it, i.e. cache a shared counter that can be modified by different nodes at the same time.

Please make sure that apart from noting anything specific of the log files, you actually send me the full TRACE log files to my email address which is my username.

[galder.zamarreno]

mcmati, solution to the issue you're encountering with adding big maps to the cache is being discussed in http://lists.jboss.org/pipermail/infinispan-dev/2009-September/001457.html