4 Replies Latest reply on Oct 1, 2006 8:59 PM by jackerm

    Problem with shunning in jgroups

    jboss.hogstrom.se

      I have a cluster using jgroups(2.2.8) with members {A, B, C}. When I unplug the network on
      {C} and leave it disconnected for about 20 seconds before reconnecting the network the
      following occurs:

      {A, B} consider themselves as a cluster and {C} considers itself as an other cluster
      and any messages being sent between {A, B} and {C} will be thrown away with the following
      log:
      2005-aug-29 10:31:36 org.jgroups.protocols.pbcast.NAKACK handleMessage
      VARNING: 10.0.1.187:2147] discarded message from non-member 10.0.1.196:3942


      What I want to happen is that {A, B} should shun {C} and that the ChannelListener
      on member {C} should invoke the channelShunned() method.

      How can I accomplish this?


      My setup is JGroups 2.2.8, j2sdk1.4.2_08, windows XP and the following protocol stack:

      String props = "UDP(mcast_addr=228.1.2.53;mcast_port=45566;" +
      "ip_ttl=64;ip_mcast=true;mcast_send_buf_size=150000;mcast_recv_buf_size=80000;" +
      "ucast_send_buf_size=150000;ucast_recv_buf_size=80000;" +
      "loopback=true):" +
      "PING(timeout=2000;num_initial_members=3;up_thread=true;down_thread=true):" +
      "FD(shun=true;up_thread=true;down_thread=true):" +
      "VERIFY_SUSPECT(timeout=1500;up_thread=true;down_thread=true):" +
      "pbcast.NAKACK(gc_lag=50;retransmit_timeout=300,600,1200,2400,4800;up_thread=true;down_thread=true):" +
      "UNICAST(timeout=5000;window_size=100;min_threshold=10;down_thread=true):" +
      "pbcast.STABLE(desired_avg_gossip=20000;up_thread=true;down_thread=true):" +
      "FRAG(frag_size=8192;down_thread=true;up_thread=true):" +
      "pbcast.GMS(join_timeout=5000;join_retry_timeout=2000;shun=true;print_local_addr=true):" +
      "pbcast.STATE_TRANSFER(up_thread=true;down_thread=true)";


      org.jgroup.JChannel m_dispChannel = new org.jgroup.JChannel(props);
      m_dispChannel.setOpt(JChannel.GET_STATE_EVENTS , Boolean.TRUE);
      m_dispChannel.setOpt(JChannel.AUTO_GETSTATE , Boolean.TRUE);
      m_dispChannel.setChannelListener(this);
      m_dispChannel.connect( "JTEST" );


      Thanks in advance!
      /Andreas

        • 1. Re: Problem with shunning in jgroups
          belaban

          This is not how shunning works: a member P is only shunned if P is excluded from the group and then (at some later time) comes back and claims to still be in the same group.
          So in you case, this happens when you put the plug back in.
          More info at: http://wiki.jboss.org/wiki/Wiki.jsp?page=Shunning

          • 2. Re: Problem with shunning in jgroups
            milowe

            Thanks for the pointer Bela :)

            I am running this test with Andreas. Our test cases always simulated hardware failure and produced subgroups, thats why we never got any shunning. We have changed the test case and can now see that a node is being shunned.

            We have some problem with the ChannelListener though. We get the intial channelConnected() notification but no channelShunned() or channelClosed().

            62293 [UpHandler (FD)] DEBUG org.jgroups.protocols.FD - [NOT_MEMBER] I'm being shunned; exiting
            62293 [UpHandler (STATE_TRANSFER)] DEBUG org.jgroups.blocks.PullPushAdapter - channel is shunned
            62293 [UpHandler (STATE_TRANSFER)] INFO org.jgroups.JChannel - received an EXIT event, will leave the channel
            62293 [CloserThread] INFO org.jgroups.JChannel - closing the channel
            62293 [DownHandler (UDP)] DEBUG org.jgroups.protocols.UDP - closing sockets and stopping threads
            62293 [UDP mcast receiver] DEBUG org.jgroups.protocols.UDP - multicast socket is closed, exception=java.net.SocketException: socket closed
            62293 [UDP mcast receiver] DEBUG org.jgroups.protocols.UDP - multicast thread terminated
            62293 [DownHandler (UDP)] DEBUG org.jgroups.protocols.UDP - multicast receive socket closed
            62293 [DownHandler (UDP)] DEBUG org.jgroups.protocols.UDP - multicast send socket closed
            62293 [UDP.UcastReceiverThread] DEBUG org.jgroups.protocols.UDP - unicast receiver socket is closed, exception=java.net.SocketException: socket closed
            62293 [DownHandler (UDP)] DEBUG org.jgroups.protocols.UDP - socket closed
            62293 [UDP.UcastReceiverThread] DEBUG org.jgroups.protocols.UDP - unicast receiver thread terminated
            62293 [PullPushAdapterThread] WARN org.jgroups.blocks.PullPushAdapter - [<null>] channel closed, exception is ChannelClosedException
            


            Any ideas?



            • 3. Re: Problem with shunning in jgroups
              milowe

              Nevermind, problem solved.

              Thanks anyway

              • 4. Re: Problem with shunning in jgroups
                jackerm

                 

                "milowe" wrote:
                Nevermind, problem solved.

                Thanks anyway


                Hey milowe

                I'm getting the same issue as you had in teh stack trace. What was the fix ?

                Jonathan