4 Replies Latest reply on Feb 27, 2014 4:51 AM by tromper11

    Infinispan/JGroups TCPPING initial_hosts members

    tromper11

      Hi,

       

      We've a cluster with 8 machines and using TCPPING.

       

      Can we specify initial_hosts to contain all servers and be the same value on every node e.g:

       

      nodeA:

      initial_hosts=nodeA,nodeB

       

      nodeB:

      initial_hosts=nodeA,nodeB

       

      or we have to exclude the actual server the configuration is on, e,.g:

       

      nodeA:

      initial_hosts=nodeB

       

      nodeB:

      initial_hosts=nodeA


      Thanks,


        • 1. Re: Infinispan/JGroups TCPPING initial_hosts members
          wdfink

          Both will work.

          But we recommend to use the first approach with all members, it keep the configuration simple and did not have a negative effect.

          1 of 1 people found this helpful
          • 2. Re: Infinispan/JGroups TCPPING initial_hosts members
            tromper11

            Thanks,

             

            Do you happen to know if there's any relation between TCPPING initial_hosts and TCP loopback?

             

            There's  a strange situation I see...

             

            We have multiple servers, but every server has:

            • an infinispan cache cluster (TCP loopback: true)
            • and a JGroups pub-sub cluster (TCP loopback: false)

             

            I see the following warning even if there's only one server running (2 JGroups channels; cache/pubsub):

             

            WARN  org.jgroups.protocols.TCP - [JGRP00012] discarded message from different cluster infinispan_1  (our cluster is pubsub_1).


            So it seems that the pubsub cluster receives the notifications from the JGroups cluster of Infinispan -- but they are configured to use separate addresses and ports.  But when I  exclude the actual server from initial_hosts and list only the others, the occurrence of JGRP00012 drops significantly.

             

            I checked the configuration a zillion times and still could not find the cause.

             

            Any ideas?

             

            Thanks,

            • 3. Re: Infinispan/JGroups TCPPING initial_hosts members
              belaban

              This has nothing to do with loopback. You should rather separate your clusters cleanly, e.g. assuming you have nodes A,B,C in cluster one and X,Y,Z in cluster two.

               

              one:

              A: TCP.bind_port=2000, TCPPING.initial_hosts=A[2000],B[2000],C[2000]

              B: TCP.bind_port=2000, TCPPING.initial_hosts=A[2000],B[2000],C[2000]

              C: TCP.bind_port=2000, TCPPING.initial_hosts=A[2000],B[2000],C[2000]

               

              two:

              X: TCP.bind_port=3000, TCPPING.initial_hosts=X[3000],Y[3000],Z[3000]

              Y: TCP.bind_port=3000, TCPPING.initial_hosts=X[3000],Y[3000],Z[3000]

              Z: TCP.bind_port=3000, TCPPING.initial_hosts=X[3000],Y[3000],Z[3000]

              • 4. Re: Infinispan/JGroups TCPPING initial_hosts members
                tromper11

                Thanks Bela,

                 

                I've checked the configuration a million times and it seems to be fine in regards the channel separation.

                • There are 4 servers 3 running on the same box;
                • Every server has 2 JGroups channels configured (one for Infinispan, one for pub-sub);
                • There's a cross-talk as the JGroups channel seems to receive all trafic;
                • evenif only one server is running there's still a cross-talk between the different JGroups channels on the same box within the same server process;

                 

                Here's the configuration (essentially the stock configs modified for custom addresses) for one of the servers experiencing this behavior:

                 

                Infinispan config:

                 

                    <TCP bind_addr="192.168.128.143"

                         bind_port="8048"

                         loopback="true"

                         port_range="0"

                         recv_buf_size="20m"

                         send_buf_size="640k"

                         max_bundle_size="64k"

                         use_send_queues="true"

                         enable_diagnostics="true"

                         bundler_type="new"

                 

                         thread_naming_pattern="pl"

                 

                         thread_pool.enabled="true"

                         thread_pool.min_threads="2"

                         thread_pool.max_threads="30"

                         thread_pool.keep_alive_time="60000"

                         thread_pool.queue_enabled="true"

                         thread_pool.queue_max_size="100"

                         thread_pool.rejection_policy="Discard"

                 

                         oob_thread_pool.enabled="true"

                         oob_thread_pool.min_threads="2"

                         oob_thread_pool.max_threads="30"

                         oob_thread_pool.keep_alive_time="60000"

                         oob_thread_pool.queue_enabled="false"

                         oob_thread_pool.queue_max_size="100"

                         oob_thread_pool.rejection_policy="Discard"/>

                 

                    <TCPPING timeout="3000"

                             initial_hosts="192.168.128.143[8048], 192.168.128.6[8038], 192.168.128.143[8028], 192.168.128.143[8008]"

                             port_range="0"

                             num_initial_members="4"

                             ergonomics="false"/>

                 

                JGroups config:

                 

                    <TCP bind_addr="192.168.128.143"

                         bind_port="8049"
                         loopback="false"

                         port_range="0"

                         recv_buf_size="5m"

                         send_buf_size="640k"

                         max_bundle_size="64k"

                         max_bundle_timeout="30"

                         use_send_queues="true"

                         sock_conn_timeout="300"

                 

                         timer_type="new3"

                         timer.min_threads="4"

                         timer.max_threads="10"

                         timer.keep_alive_time="3000"

                         timer.queue_max_size="500"

                 

                         thread_pool.enabled="true"

                         thread_pool.min_threads="1"

                         thread_pool.max_threads="10"

                         thread_pool.keep_alive_time="5000"

                         thread_pool.queue_enabled="false"

                         thread_pool.queue_max_size="100"

                         thread_pool.rejection_policy="discard"

                 

                         oob_thread_pool.enabled="true"

                         oob_thread_pool.min_threads="1"

                         oob_thread_pool.max_threads="8"

                         oob_thread_pool.keep_alive_time="5000"

                         oob_thread_pool.queue_enabled="false"

                         oob_thread_pool.queue_max_size="100"

                         oob_thread_pool.rejection_policy="discard"/>

                 

                    <TCPPING timeout="3000"

                             initial_hosts="192.168.128.143[8049],192.168.128.6[8039],192.168.128.143[8029],192.168.128.143[8009]"

                             port_range="0"

                             num_initial_members="4"/>

                 

                As you can see the ports for different channels always end with either 09 or 08.

                 

                Still there's something wrong as there are a lot of 'JGRP00012] discarded message from different' warnings.

                 

                Any ideas hot to resolve it?

                 

                Thanks,