Infinispan - jgroups protocol stack, order of protocols

[gregosby]

Infinispan - jgroups protocol stack, order of protocols

Hi I see that in Infinispan documentation protocol stack is documented basing on jgroups stack.

For example UDP:

UDP uses IP multicast for sending messages to all members of a cluster, and UDP datagrams for unicast messages (sent to a single member). When started, it opens a unicast and multicast socket: the unicast socket is used to send/receive unicast messages, while the multicast socket sends/receives multicast messages. The physical address of the channel will be the address and port number of the unicast socket.

Thats ok but cache configuration (sample infinispan jgroups files) have list of protocols in specific order.

Where can I find which protocol is doing what and what is the order, there is some sequence of processing I think... Looking into jgroups sources there are some UpHandlers and DownHandlers that handle different events for each protocol. Is there any documentation of this area - the reason and order of protocols??

public static final int MSG                       =  1;  // arg = Message
    public static final int CONNECT                   =  2;  // arg = group address (string)
    public static final int CONNECT_OK                =  3;  // arg = null (or exception if JOIN failed)
    public static final int DISCONNECT                =  4;  // arg = member address (Address)
    public static final int DISCONNECT_OK             =  5;
    public static final int VIEW_CHANGE               =  6;  // arg = View (or MergeView in case of merge)
    public static final int GET_LOCAL_ADDRESS         =  7;
    public static final int SET_LOCAL_ADDRESS         =  8;
    public static final int SUSPECT                   =  9;  // arg = Address of suspected member
    public static final int BLOCK                     = 10;
    public static final int BLOCK_OK                  = 11;
    public static final int FIND_INITIAL_MBRS         = 12;
    public static final int FIND_INITIAL_MBRS_OK      = 13;  // arg = Vector of PingRsps
    public static final int MERGE                     = 14;  // arg = Vector of Objects
    public static final int TMP_VIEW                  = 15;  // arg = View
    public static final int BECOME_SERVER             = 16;  // sent when client has joined group
    public static final int GET_APPLSTATE             = 17;  // get state from appl (arg=StateTransferInfo)
    public static final int GET_APPLSTATE_OK          = 18;  // arg = StateTranferInfo (includes state)
    public static final int GET_STATE                 = 19;  // arg = StateTransferInfo
    public static final int GET_STATE_OK              = 20;  // arg = StateTransferInfo
    public static final int STATE_RECEIVED            = 21;  // arg = StateTransferInfo (with state and state_id)
    public static final int START_QUEUEING            = 22;
    public static final int STOP_QUEUEING             = 23;  // arg = Vector (event-list)
    public static final int SWITCH_NAK                = 24;
    public static final int SWITCH_NAK_ACK            = 25;
    public static final int SWITCH_OUT_OF_BAND        = 26;
    public static final int FLUSH                     = 27;  // arg = Vector (destinatinon for FLUSH)
    public static final int FLUSH_OK                  = 28;  // arg = FlushRsp
    public static final int DROP_NEXT_MSG             = 29;
    public static final int STABLE                    = 30;  // arg = long[] (stable seqnos for mbrs)
    public static final int GET_MSG_DIGEST            = 31;  // arg = long[] (highest seqnos from mbrs)
    public static final int GET_MSG_DIGEST_OK         = 32;  // arg = Digest
    public static final int REBROADCAST_MSGS          = 33;  // arg = Vector (msgs with NakAckHeader)
    public static final int REBROADCAST_MSGS_OK       = 34;
    public static final int GET_MSGS_RECEIVED         = 35;
    public static final int GET_MSGS_RECEIVED_OK      = 36;  // arg = long[] (highest deliverable seqnos)
    public static final int GET_MSGS                  = 37;  // arg = long[][] (range of seqnos for each m.)
    public static final int GET_MSGS_OK               = 38;  // arg = List
    public static final int GET_DIGEST                = 39;  //
    public static final int GET_DIGEST_OK             = 40;  // arg = Digest (response to GET_DIGEST)
    public static final int SET_DIGEST                = 41;  // arg = Digest
    public static final int GET_DIGEST_STATE          = 42;  // see ./JavaStack/Protocols/pbcast/DESIGN for explanantion
    public static final int GET_DIGEST_STATE_OK       = 43;  // see ./JavaStack/Protocols/pbcast/DESIGN for explanantion
    public static final int SET_PARTITIONS            = 44;  // arg = Hashtable of addresses and numbers
    public static final int MERGE_DENIED              = 45;  // Passed down from gms when a merge attempt fails
    public static final int EXIT                      = 46;  // received when member was forced out of the group
    public static final int PERF                      = 47;  // for performance measurements
    public static final int SUBVIEW_MERGE             = 48;  // arg = vector of addresses; see JGroups/EVS/Readme.txt
    public static final int SUBVIEWSET_MERGE          = 49;  // arg = vector of addresses; see JGroups/EVS/Readme.txt
    public static final int HEARD_FROM                = 50;  // arg = Vector (list of Addresses)
    public static final int UNSUSPECT                 = 51;  // arg = Address (of unsuspected member)
    public static final int SET_PID                   = 52;  // arg = Integer (process id)
    public static final int MERGE_DIGEST              = 53;  // arg = Digest
    public static final int CONFIG                    = 56;  // arg = HashMap (config properties)
    public static final int GET_DIGEST_STABLE         = 57;
    public static final int GET_DIGEST_STABLE_OK      = 58;  // response to GET_DIGEST_STABLE
    // public static final int ACK                       = 59;  // used to flush down events
    // public static final int ACK_OK                    = 60;  // response to ACK
    public static final int START                     = 61;  // triggers start() - internal event, handled by Protocol
    public static final int START_OK                  = 62;  // arg = exception of null - internal event, handled by Protocol
    public static final int STOP                      = 63;  // triggers stop() - internal event, handled by Protocol
    public static final int STOP_OK                   = 64;  // arg = exception or null - internal event, handled by Protocol
    public static final int SUSPEND_STABLE            = 65;  // arg = Long (max_suspend_time)
    public static final int RESUME_STABLE             = 66;  // arg = null
    public static final int ENABLE_UNICASTS_TO        = 67;  // arg = Address (member)
    public static final int SUSPEND                      = 68;  // arg = View|null
    public static final int SUSPEND_OK                  = 69;     // arg = null
    public static final int RESUME                      = 70;  // arg = null
    public static final int STATE_TRANSFER_INPUTSTREAM= 71;     // arg=java.io.InputStream subclass
    public static final int STATE_TRANSFER_OUTPUTSTREAM= 72; // arg=java.io.OutputStream subclass
    public static final int STATE_TRANSFER_INPUTSTREAM_CLOSED= 73;//arg=null
    public static final int STATE_TRANSFER_OUTPUTSTREAM_CLOSED= 74;//arg=null
    public static final int UNBLOCK                   =75;  //arg=null (indicate end of flush round)
    public static final int SUSPEND_FAILED            =76;  // arg = null

[rvansa]

What each protocol does: Reliable group communication with JGroups

The sequence of processing starts with receiving the data and unmarshalling the message in UDP/TCP protocol and sending that up the stack (counter-intuitively, it means down in the order as jgroups config is written). When a message/RPC is sent from Infinispan, it starts in the topmost protocol (FRAG2 I think in the default config) and goes down (up in the config).

The order is important, but not always. E.g. FD* protocols don't require reliability of messages, so they are below the UNICAST3 or NAKACK2 protocols that guarantee realiability and/or ordering. But it does not matter whether they are below or over *PING protocols.