7 Replies Latest reply on Sep 24, 2009 12:19 PM by Galder Zamarreño

    3 nodes problem

    Marcin Matuszak Newbie

      Hi,

      I've got another problem. When I use 2 nodes over Lan, everything seems to be ok. I can add data relativelly fast. But when I start another node then things aren't so great. I have to wait reeaaaallllly long before some information is added, and sometimes it isn't added at all (replication timeout error even when I set: sync replTimeout="200000"). I use distribution mode, with hash numOwners="2" and I use GuiDemo provided with infinispan. Do you have any idea what may be causing that?

      Best regards,
      Martin

        • 1. Re: 3 nodes problem
          Galder Zamarreño Master

          How are these machines communicating? Via udp or tcp?
          If using udp, can you switch to tcp and see if you have the same issues?
          The easiest way to switch to tcp is to use a transport config like this:

          <transport clusterName="infinispan-cluster">
           <properties>
           <property name="configurationFile" value="tcp.xml"/>
           </properties>
          </transport>


          And then pass the initial hosts to each node via system property, exampple:
          -Djgroups.tcpping.initial_hosts=host1[7800],host2[7800],host3[7800]


          Are the 2 nodes that work fine started in the same machine or different machines?
          And the 3rd node? Is it a different machine?

          • 2. Re: 3 nodes problem
            Marcin Matuszak Newbie

            Those machines communicate via default protocol (I believe it is udp).
            When I changed that to tcp there was no improvement.

            It doesn't matter if those two nodes that work fine are on the same machine or two different machines - it works ok. Problems start when I add more nodes, and here again it doesn't matter if those new nodes are created on one of those 2 machines that are already running or if I run it on 3rd machine. Anyway, after I add 3rd node, my cluster works a lot worse.

            It is weird, because when I have 3 nodes or more, some of them still work fine, but some don't, and I can't find any dependencies which work better and which worse. It just seems to be quite random.

            • 3. Re: 3 nodes problem
              Galder Zamarreño Master

              Could you post your full configuration file?
              What's the Infinispan version you're using? Beta1?
              I'd suggest enabling TRACE logging (see http://www.jboss.org/community/wiki/infinispantechnicalfaqs#How_can_I_enable_logging) in each of the 3 nodes and try to replicate the issue where you get those timeouts.

              • 4. Re: 3 nodes problem
                Marcin Matuszak Newbie

                well I've been trying lots of configuration files, but as for now I'm using this one:

                <?xml version="1.0" encoding="UTF-8"?>
                
                <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:4.0">
                 <global>
                 <transport clusterName="infinispan-cluster">
                 <properties>
                 <property name="configurationFile" value="udp.xml"/>
                 </properties>
                 </transport>
                 </global>
                
                 <default>
                
                 <locking
                 isolationLevel="REPEATABLE_READ"
                 lockAcquisitionTimeout="20000"
                 writeSkewCheck="false"
                 concurrencyLevel="500"/>
                
                 <!--
                 Used to register a transaction manager and participate in ongoing transactions.
                 -->
                 <transaction
                 transactionManagerLookupClass="org.infinispan.transaction.lookup.GenericTransactionManagerLookup"
                 syncRollbackPhase="false"
                 syncCommitPhase="false"
                 useEagerLocking="true"/>
                
                 <clustering mode="distribution">
                 <l1 enabled="true" lifespan="10000"/>
                 <hash numOwners="2" rehashWait="12000" rehashRpcTimeout="60000"/>
                 <sync replTimeout="200000"/>
                 </clustering>
                 </default>
                </infinispan>
                


                I know that here I use UDP, but I also tried TCP. And there were versions without transaction and locking part.

                So that's it. Now I'm going to enable trace logging and we'll see ;)


                Best regards,
                Martin

                • 5. Re: 3 nodes problem
                  Marcin Matuszak Newbie

                  I forgot. I'm using Beta1. Sorry for posting two replyes in a row, but I couldn't find "edit" button.

                  • 6. Re: 3 nodes problem
                    Galder Zamarreño Master

                    Hi, I don't think you should be using useEagerLocking unless you have specific case for it, i.e. cache a shared counter that can be modified by different nodes at the same time.

                    Please make sure that apart from noting anything specific of the log files, you actually send me the full TRACE log files to my email address which is my username.

                    • 7. Re: 3 nodes problem
                      Galder Zamarreño Master

                      mcmati, solution to the issue you're encountering with adding big maps to the cache is being discussed in http://lists.jboss.org/pipermail/infinispan-dev/2009-September/001457.html