6 Replies Latest reply on Jun 28, 2004 6:09 PM by julien.dubois

    3.2.4 cluster problem

    amdonov

      I had two solaris machines clustered with 3.2.3. Moved to 3.2.4 and started an all instance on each box. They don't see each other. I can start two instances on the same machine and they cluster fine. The boxes both only have a single network card (That seemed to be causing others problems). Any ideas?

        • 1. Re: 3.2.4 cluster problem
          amdonov

          Just as a follow up. I started 3.2.4 on one machine and 3.2.3 on the other. They see each other but complain about a version mismatch. So I'm guessing 3.2.4 is receiving requests fine but not broadcasting correctly.

          • 2. Re: 3.2.4 cluster problem
            amdonov

            If I run the org.jgroups.demos.Chat program from the jgroups.jar 2.2.4 that comes with JBoss 3.2.4 I get the same behavior. Clients on the same machine see each other, but remote clients are not visible. Same demo with jgroups.jar 2.2.0 from JBoss 3.2.3 works. Can I use the old jar or will that cause me other problems?

            • 3. Re: 3.2.4 cluster problem
              belaban

              So you're saying the *same* program with the *same* config (cluster-service.xml) worked under 3.2.3, but *not* under 3.2.4 ?
              Can you
              - post your JGroups configuration
              - enable tracing

              Let's do Draw first so we can determine whether it is a JGroups problem.

              Copy your attribute from cluster-service.xml into a file, e.g. test.xml, and start Draw like this:
              java -Dlog4j.configuration=file:c:\log4j.properties org.jgroups.demos.Draw -props c:\\test.xml

              c:\log4j.properties needs to have an entry for org.jgroups set to DEBUG.

              Bela

              • 4. Re: 3.2.4 cluster problem
                amdonov

                Bela,

                Thanks for the response.

                The cluster-service.xml I am using is the default for 3.2.3

                <?xml version="1.0"?>

                <!-- UDP: if you have a multihomed machine,
                set the bind_addr attribute to the appropriate NIC IP address -->
                <!-- UDP: On Windows machines, because of the media sense feature
                being broken with multicast (even after disabling media sense)
                set the loopback attribute to true -->
                <UDP mcast_addr="228.1.2.3" mcast_port="45566"
                ip_ttl="64" ip_mcast="true"
                mcast_send_buf_size="150000" mcast_recv_buf_size="80000"
                ucast_send_buf_size="150000" ucast_recv_buf_size="80000"
                loopback="false" />
                <PING timeout="2000" num_initial_members="3"
                up_thread="true" down_thread="true" />
                <MERGE2 min_interval="5000" max_interval="10000" />
                <FD shun="true" up_thread="true" down_thread="true"
                timeout="2500" max_tries="5" />
                <VERIFY_SUSPECT timeout="3000" num_msgs="3"
                up_thread="true" down_thread="true" />
                <pbcast.NAKACK gc_lag="50" retransmit_timeout="300,600,1200,2400,4800"
                up_thread="true" down_thread="true" />
                <pbcast.STABLE desired_avg_gossip="20000"
                up_thread="true" down_thread="true" />
                <UNICAST timeout="5000" window_size="100" min_threshold="10"
                down_thread="true" />
                <FRAG frag_size="8192"
                down_thread="true" up_thread="true" />
                <pbcast.GMS join_timeout="5000" join_retry_timeout="2000"
                shun="true" print_local_addr="true" />
                <pbcast.STATE_TRANSFER up_thread="true" down_thread="true" />




                I saved it as test.xml as suggested.



                First machine output:

                pegasus:/home/frdonam/software/jboss/server/all/lib% java -cp jgroups.jar:commons-logging.jar:./ org.jgroups.Version

                Version: 2.2.4
                CVS: $Id: Version.java,v 1.5 2004/04/28 18:44:58 belaban Exp $
                History: (see doc/history.txt for details)

                pegasus:/home/frdonam/software/jboss/server/all/lib% java -cp jgroups.jar:commons-logging.jar:./ org.jgroups.demos.Draw -props test.xml
                19:05:56,267 [DEBUG] org.jgroups.conf.ClassConfigurator: mapping is:
                1: class org.jgroups.stack.IpAddress
                2: class org.jgroups.protocols.CAUSAL$CausalHeader
                3: class org.jgroups.protocols.FD$FdHeader
                4: class org.jgroups.protocols.FD_PID$FdHeader
                5: class org.jgroups.protocols.FD_PROB$FdHeader
                6: class org.jgroups.protocols.FD_SOCK$FdHeader
                7: class org.jgroups.protocols.FRAG$FragHeader
                8: class org.jgroups.protocols.MERGE$MergeHeader
                9: class org.jgroups.protocols.NakAckHeader
                10: class org.jgroups.protocols.PARTITIONER$PartitionerHeader
                11: class org.jgroups.protocols.PerfHeader
                12: class org.jgroups.protocols.PIGGYBACK$PiggybackHeader
                13: class org.jgroups.protocols.PingHeader
                14: class org.jgroups.protocols.TcpHeader
                15: class org.jgroups.protocols.TOTAL$Header
                16: class org.jgroups.protocols.TOTAL_OLD$TotalHeader
                17: class org.jgroups.protocols.TOTAL_TOKEN$TotalTokenHeader
                18: class org.jgroups.protocols.TOTAL_TOKEN$RingTokenHeader
                19: class org.jgroups.protocols.TunnelHeader
                20: class org.jgroups.protocols.UdpHeader
                21: class org.jgroups.protocols.UNICAST$UnicastHeader
                22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
                23: class org.jgroups.protocols.WANPIPE$WanPipeHeader
                24: class org.jgroups.protocols.pbcast.GMS$GmsHeader
                25: class org.jgroups.protocols.pbcast.NakAckHeader
                26: class org.jgroups.protocols.pbcast.PbcastHeader
                27: class org.jgroups.protocols.pbcast.STABLE$StableHeader
                28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
                29: class org.jgroups.protocols.SMACK$SmackHeader
                50: class org.jgroups.service.lease.LeaseRequestHeader
                51: class org.jgroups.service.lease.LeaseResponseHeader
                52: class org.jgroups.service.lease.DenyResponseHeader
                53: class org.jgroups.protocols.COMPRESS$CompressHeader

                19:09:18,655 [DEBUG] org.jgroups.protocols.AUTOCONF: frag_size=64000
                19:09:18,861 [DEBUG] org.jgroups.protocols.pbcast.GMS: changed role to org.jgroups.protocols.pbcast.ClientGmsImpl
                19:09:18,862 [DEBUG] org.jgroups.protocols.pbcast.GMS: null became client
                19:09:18,936 [DEBUG] org.jgroups.protocols.UDP: creating sockets and starting threads
                19:09:18,942 [INFO] org.jgroups.protocols.UDP: unicast sockets will use interface 136.195.116.11
                19:09:18,956 [INFO] org.jgroups.protocols.UDP: socket information:
                local_addr=pegasus:59325, mcast_addr=228.1.2.3:45566, bind_addr=/136.195.116.11, ttl=64
                socket: bound to 136.195.116.11:59325, receive buffer size=80000, send buffer size=150000
                multicast socket: bound to 136.195.116.11:45566, send buffer size=150000, receive buffer size=80000
                19:09:18,963 [DEBUG] org.jgroups.protocols.UDP: created unicast receiver thread

                -------------------------------------------------------
                GMS: address is pegasus:59325
                -------------------------------------------------------
                19:09:18,977 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:18,994 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:18,996 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:19,011 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.116.11:59325 (size=101 bytes)
                19:09:19,025 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:21,000 [DEBUG] org.jgroups.protocols.PING: initial mbrs are []
                19:09:21,002 [DEBUG] org.jgroups.protocols.pbcast.ClientGmsImpl: initial_mbrs are []
                19:09:21,003 [DEBUG] org.jgroups.protocols.pbcast.ClientGmsImpl: no initial members discovered: creating group as first member
                19:09:21,005 [DEBUG] org.jgroups.protocols.pbcast.GMS: [local_addr=pegasus:59325] view is [pegasus:59325|0] [pegasus:59325]
                19:09:21,024 [DEBUG] org.jgroups.protocols.pbcast.GMS: changed role to org.jgroups.protocols.pbcast.CoordGmsImpl
                19:09:21,025 [DEBUG] org.jgroups.protocols.pbcast.GMS: pegasus:59325 became coordinator
                19:09:21,026 [DEBUG] org.jgroups.protocols.pbcast.ClientGmsImpl: created group (first member). My view is [pegasus:59325|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl
                19:09:21,067 [DEBUG] org.jgroups.protocols.pbcast.STABLE: stable task started; num_gossip_runs=3, max_gossip_runs=3
                19:09:21,091 [DEBUG] org.jgroups.protocols.FD: suspected_mbrs: [], after adjustment: [], stopped: true
                19:09:21,098 [DEBUG] org.jgroups.protocols.MERGE2: merge task started
                ** View=[pegasus:59325|0] [pegasus:59325]
                19:09:25,644 [DEBUG] org.jgroups.protocols.pbcast.STABLE: mcasting digest [pegasus:59325: [0 : 0] (num_gossip_runs=3, max_gossip_runs=3)
                19:09:25,646 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #0
                19:09:25,648 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 0]
                19:09:25,649 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=0, range=null], STABLE=[STABLE_GOSSIP]: digest is [pegasus:59325: [0 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:25,688 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 271 bytes from /136.195.116.11:59325 (size=271 bytes)
                19:09:25,697 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (3 headers), size = 0 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=0, range=null], STABLE=[STABLE_GOSSIP]: digest is [pegasus:59325: [0 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:25,699 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#0>
                19:09:25,701 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [0 - 0 (size=1, missing=0)], delivered_msgs: [null - null]
                19:09:25,702 [DEBUG] org.jgroups.protocols.pbcast.STABLE: received digest pegasus:59325#0 (-1) from pegasus:59325
                19:09:25,702 [DEBUG] org.jgroups.protocols.pbcast.STABLE: sending stability msg pegasus:59325#0 (-1)
                19:09:25,703 [DEBUG] org.jgroups.protocols.pbcast.STABLE: stability_task=null, delay is 2654
                19:09:27,250 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:27,251 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:27,252 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:27,253 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.116.11:59325 (size=101 bytes)
                19:09:27,254 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:27,258 [DEBUG] org.jgroups.protocols.PING: received GET_MBRS_REQ from pegasus:59325, returning [PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:27,259 [DEBUG] org.jgroups.protocols.UDP: sending message to pegasus:59325 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:27,267 [DEBUG] org.jgroups.protocols.UDP: received (ucast) 263 bytes from /136.195.116.11:59325
                19:09:27,270 [DEBUG] org.jgroups.protocols.UDP: message is [dst: pegasus:59325, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:27,271 [DEBUG] org.jgroups.protocols.PING: received FIND_INITAL_MBRS_RSP, rsp=[own_addr=pegasus:59325, coord_addr=pegasus:59325]
                19:09:27,271 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=1980, got 1 rsps
                19:09:28,370 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #1
                19:09:28,371 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 0]
                19:09:29,261 [DEBUG] org.jgroups.protocols.PING: initial mbrs are [[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:29,262 [DEBUG] org.jgroups.protocols.MERGE2: initial_mbrs=[[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:29,263 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=1, range=null], STABLE=[STABILITY]: digest is [pegasus:59325: [-1 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:29,264 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 271 bytes from /136.195.116.11:59325 (size=271 bytes)
                19:09:29,266 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (3 headers), size = 0 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=1, range=null], STABLE=[STABILITY]: digest is [pegasus:59325: [-1 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:29,268 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#1>
                19:09:29,269 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [1 - 1 (size=1, missing=0)], delivered_msgs: [0 - 0]
                19:09:29,272 [DEBUG] org.jgroups.protocols.pbcast.STABLE: stability vector is [pegasus:59325#0]
                19:09:29,274 [DEBUG] org.jgroups.protocols.pbcast.STABLE: cancelling stability task (running=false)
                19:09:29,277 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: received digest [pegasus:59325: [-1 : 0]
                19:09:33,350 [DEBUG] org.jgroups.protocols.pbcast.STABLE: mcasting digest [pegasus:59325: [0 : 1 (1)]] (num_gossip_runs=2, max_gossip_runs=3)
                19:09:33,351 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #2
                19:09:33,352 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 1]
                19:09:33,353 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=2, range=null], STABLE=[STABLE_GOSSIP]: digest is [pegasus:59325: [0 : 1 (1)]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:33,354 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 271 bytes from /136.195.116.11:59325 (size=271 bytes)
                19:09:33,358 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (3 headers), size = 0 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=2, range=null], STABLE=[STABLE_GOSSIP]: digest is [pegasus:59325: [0 : 1 (1)]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:33,359 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#2>
                19:09:33,360 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [2 - 2 (size=1, missing=0)], delivered_msgs: [0 - 1]
                19:09:33,361 [DEBUG] org.jgroups.protocols.pbcast.STABLE: received digest pegasus:59325#1 (1) from pegasus:59325
                19:09:33,361 [DEBUG] org.jgroups.protocols.pbcast.STABLE: sending stability msg pegasus:59325#1 (1)
                19:09:33,361 [DEBUG] org.jgroups.protocols.pbcast.STABLE: stability_task=null, delay is 4053
                19:09:35,430 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:35,431 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:35,431 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:35,432 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.116.11:59325 (size=101 bytes)
                19:09:35,433 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:35,434 [DEBUG] org.jgroups.protocols.PING: received GET_MBRS_REQ from pegasus:59325, returning [PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:35,435 [DEBUG] org.jgroups.protocols.UDP: sending message to pegasus:59325 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:35,436 [DEBUG] org.jgroups.protocols.UDP: received (ucast) 263 bytes from /136.195.116.11:59325
                19:09:35,438 [DEBUG] org.jgroups.protocols.UDP: message is [dst: pegasus:59325, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:35,439 [DEBUG] org.jgroups.protocols.PING: received FIND_INITAL_MBRS_RSP, rsp=[own_addr=pegasus:59325, coord_addr=pegasus:59325]
                19:09:35,439 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=1991, got 1 rsps
                19:09:37,420 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #3
                19:09:37,421 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 2]
                19:09:37,440 [DEBUG] org.jgroups.protocols.PING: initial mbrs are [[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:37,440 [DEBUG] org.jgroups.protocols.MERGE2: initial_mbrs=[[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:37,441 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=3, range=null], STABLE=[STABILITY]: digest is [pegasus:59325: [-1 : 1 (1)]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:37,443 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 271 bytes from /136.195.116.11:59325 (size=271 bytes)
                19:09:37,445 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (3 headers), size = 0 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=3, range=null], STABLE=[STABILITY]: digest is [pegasus:59325: [-1 : 1 (1)]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:37,446 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#3>
                19:09:37,447 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [3 - 3 (size=1, missing=0)], delivered_msgs: [0 - 2]
                19:09:37,447 [DEBUG] org.jgroups.protocols.pbcast.STABLE: stability vector is [pegasus:59325#1]
                19:09:37,448 [DEBUG] org.jgroups.protocols.pbcast.STABLE: cancelling stability task (running=false)
                19:09:37,448 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: received digest [pegasus:59325: [-1 : 1 (1)]]
                19:09:44,880 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:44,881 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:44,881 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:44,882 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.116.11:59325 (size=101 bytes)
                19:09:44,883 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:44,884 [DEBUG] org.jgroups.protocols.PING: received GET_MBRS_REQ from pegasus:59325, returning [PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:44,885 [DEBUG] org.jgroups.protocols.UDP: sending message to pegasus:59325 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:44,889 [DEBUG] org.jgroups.protocols.UDP: received (ucast) 263 bytes from /136.195.116.11:59325
                19:09:44,891 [DEBUG] org.jgroups.protocols.UDP: message is [dst: pegasus:59325, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:44,892 [DEBUG] org.jgroups.protocols.PING: received FIND_INITAL_MBRS_RSP, rsp=[own_addr=pegasus:59325, coord_addr=pegasus:59325]
                19:09:44,893 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=1987, got 1 rsps
                19:09:46,890 [DEBUG] org.jgroups.protocols.PING: initial mbrs are [[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:46,891 [DEBUG] org.jgroups.protocols.MERGE2: initial_mbrs=[[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:49,065 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #4
                19:09:49,066 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 3]
                19:09:49,066 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=4, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,068 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,072 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=4, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,073 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#4>
                19:09:49,074 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [4 - 4 (size=1, missing=0)], delivered_msgs: [0 - 3]
                19:09:49,087 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #5
                19:09:49,091 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 4]
                19:09:49,092 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #6
                19:09:49,093 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 5]
                19:09:49,093 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=5, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,095 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,096 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=5, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,096 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=6, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,097 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,098 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=6, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,099 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#5>
                19:09:49,100 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [5 - 5 (size=1, missing=0)], delivered_msgs: [0 - 4]
                19:09:49,101 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#6>
                19:09:49,101 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [6 - 6 (size=1, missing=0)], delivered_msgs: [0 - 5]
                19:09:49,107 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #7
                19:09:49,118 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 6]
                19:09:49,119 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #8
                19:09:49,119 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 7]
                19:09:49,120 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=7, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,122 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,124 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=7, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,125 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=8, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,127 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,134 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=8, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,138 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#7>
                19:09:49,140 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [7 - 7 (size=1, missing=0)], delivered_msgs: [0 - 6]
                19:09:49,141 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#8>
                19:09:49,142 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [8 - 8 (size=1, missing=0)], delivered_msgs: [0 - 7]
                19:09:49,302 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #9
                19:09:49,303 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 8]
                19:09:49,304 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #10
                19:09:49,304 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 9]
                19:09:49,305 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=9, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,306 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,307 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=9, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,308 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=10, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,311 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,320 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#9>
                19:09:49,321 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [9 - 9 (size=1, missing=0)], delivered_msgs: [0 - 8]
                19:09:49,328 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #11
                19:09:49,328 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 10]
                19:09:49,329 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #12
                19:09:49,329 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 11]
                19:09:49,330 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=11, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,331 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=12, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,335 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #13
                19:09:49,336 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 12]
                19:09:49,337 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=13, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,344 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=10, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,348 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,350 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=11, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,350 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,351 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=12, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,358 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,359 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=13, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,361 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#10>
                19:09:49,362 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [10 - 10 (size=1, missing=0)], delivered_msgs: [0 - 9]
                19:09:49,363 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#11>
                19:09:49,364 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [11 - 11 (size=1, missing=0)], delivered_msgs: [0 - 10]
                19:09:49,372 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#12>
                19:09:49,373 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [12 - 12 (size=1, missing=0)], delivered_msgs: [0 - 11]
                19:09:49,374 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#13>
                19:09:49,374 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [13 - 13 (size=1, missing=0)], delivered_msgs: [0 - 12]
                19:09:49,381 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #14
                19:09:49,381 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 13]
                19:09:49,382 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #15
                19:09:49,382 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 14]
                19:09:49,384 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=14, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,385 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,388 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=14, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,389 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {NAKACK=[NAKACK: MSG, seqno=15, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,390 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 210 bytes from /136.195.116.11:59325 (size=210 bytes)
                19:09:49,393 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 101 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=15, range=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:49,394 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#14>
                19:09:49,397 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [14 - 14 (size=1, missing=0)], delivered_msgs: [0 - 13]
                19:09:49,398 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [pegasus:59325] received <pegasus:59325#15>
                19:09:49,399 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for pegasus:59325 is: received_msgs: [15 - 15 (size=1, missing=0)], delivered_msgs: [0 - 14]
                19:09:53,850 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:53,851 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:53,851 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:53,852 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.116.11:59325 (size=101 bytes)
                19:09:53,853 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:53,854 [DEBUG] org.jgroups.protocols.PING: received GET_MBRS_REQ from pegasus:59325, returning [PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:53,854 [DEBUG] org.jgroups.protocols.UDP: sending message to pegasus:59325 (src=pegasus:59325), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:53,856 [DEBUG] org.jgroups.protocols.UDP: received (ucast) 263 bytes from /136.195.116.11:59325
                19:09:53,857 [DEBUG] org.jgroups.protocols.UDP: message is [dst: pegasus:59325, src: pegasus:59325 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=pegasus:59325, coord_addr=pegasus:59325]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:53,858 [DEBUG] org.jgroups.protocols.PING: received FIND_INITAL_MBRS_RSP, rsp=[own_addr=pegasus:59325, coord_addr=pegasus:59325]
                19:09:53,858 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=1992, got 1 rsps
                19:09:55,860 [DEBUG] org.jgroups.protocols.PING: initial mbrs are [[own_addr=pegasus:59325, coord_addr=pegasus:59325]]
                19:09:55,861 [DEBUG] org.jgroups.protocols.MERGE2: initial_mbrs=[[own_addr=pegasus:59325, coord_addr=pegasus:59325]]


                Second machine output:

                ngic01juweb01% java -cp jgroups.jar:commons-logging.jar:./ org.jgroups.Version

                Version: 2.2.4
                CVS: $Id: Version.java,v 1.5 2004/04/28 18:44:58 belaban Exp $
                History: (see doc/history.txt for details)

                ngic01juweb01% java -cp jgroups.jar:commons-logging.jar:./ org.jgroups.demos.Draw -props test.xml
                19:09:28,375 [DEBUG] org.jgroups.conf.ClassConfigurator: mapping is:
                1: class org.jgroups.stack.IpAddress
                2: class org.jgroups.protocols.CAUSAL$CausalHeader
                3: class org.jgroups.protocols.FD$FdHeader
                4: class org.jgroups.protocols.FD_PID$FdHeader
                5: class org.jgroups.protocols.FD_PROB$FdHeader
                6: class org.jgroups.protocols.FD_SOCK$FdHeader
                7: class org.jgroups.protocols.FRAG$FragHeader
                8: class org.jgroups.protocols.MERGE$MergeHeader
                9: class org.jgroups.protocols.NakAckHeader
                10: class org.jgroups.protocols.PARTITIONER$PartitionerHeader
                11: class org.jgroups.protocols.PerfHeader
                12: class org.jgroups.protocols.PIGGYBACK$PiggybackHeader
                13: class org.jgroups.protocols.PingHeader
                14: class org.jgroups.protocols.TcpHeader
                15: class org.jgroups.protocols.TOTAL$Header
                16: class org.jgroups.protocols.TOTAL_OLD$TotalHeader
                17: class org.jgroups.protocols.TOTAL_TOKEN$TotalTokenHeader
                18: class org.jgroups.protocols.TOTAL_TOKEN$RingTokenHeader
                19: class org.jgroups.protocols.TunnelHeader
                20: class org.jgroups.protocols.UdpHeader
                21: class org.jgroups.protocols.UNICAST$UnicastHeader
                22: class org.jgroups.protocols.VERIFY_SUSPECT$VerifyHeader
                23: class org.jgroups.protocols.WANPIPE$WanPipeHeader
                24: class org.jgroups.protocols.pbcast.GMS$GmsHeader
                25: class org.jgroups.protocols.pbcast.NakAckHeader
                26: class org.jgroups.protocols.pbcast.PbcastHeader
                27: class org.jgroups.protocols.pbcast.STABLE$StableHeader
                28: class org.jgroups.protocols.pbcast.STATE_TRANSFER$StateHeader
                29: class org.jgroups.protocols.SMACK$SmackHeader
                50: class org.jgroups.service.lease.LeaseRequestHeader
                51: class org.jgroups.service.lease.LeaseResponseHeader
                52: class org.jgroups.service.lease.DenyResponseHeader
                53: class org.jgroups.protocols.COMPRESS$CompressHeader

                19:09:28,441 [DEBUG] org.jgroups.protocols.AUTOCONF: frag_size=64000
                19:09:28,559 [DEBUG] org.jgroups.protocols.pbcast.GMS: changed role to org.jgroups.protocols.pbcast.ClientGmsImpl
                19:09:28,559 [DEBUG] org.jgroups.protocols.pbcast.GMS: null became client
                19:09:28,602 [DEBUG] org.jgroups.protocols.UDP: creating sockets and starting threads
                19:09:28,604 [INFO] org.jgroups.protocols.UDP: unicast sockets will use interface 136.195.214.148
                19:09:28,611 [INFO] org.jgroups.protocols.UDP: socket information:
                local_addr=ngic01juweb01:34373, mcast_addr=228.1.2.3:45566, bind_addr=/136.195.214.148, ttl=64
                socket: bound to 136.195.214.148:34373, receive buffer size=80000, send buffer size=150000
                multicast socket: bound to 136.195.214.148:45566, send buffer size=150000, receive buffer size=80000
                19:09:28,615 [DEBUG] org.jgroups.protocols.UDP: created unicast receiver thread

                -------------------------------------------------------
                GMS: address is ngic01juweb01:34373
                -------------------------------------------------------
                19:09:28,622 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:28,630 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:28,630 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=ngic01juweb01:34373), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:28,638 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.214.148:34373 (size=101 bytes)
                19:09:28,646 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: ngic01juweb01:34373 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:30,640 [DEBUG] org.jgroups.protocols.PING: initial mbrs are []
                19:09:30,641 [DEBUG] org.jgroups.protocols.pbcast.ClientGmsImpl: initial_mbrs are []
                19:09:30,641 [DEBUG] org.jgroups.protocols.pbcast.ClientGmsImpl: no initial members discovered: creating group as first member
                19:09:30,642 [DEBUG] org.jgroups.protocols.pbcast.GMS: [local_addr=ngic01juweb01:34373] view is [ngic01juweb01:34373|0] [ngic01juweb01:34373]
                19:09:30,658 [DEBUG] org.jgroups.protocols.pbcast.GMS: changed role to org.jgroups.protocols.pbcast.CoordGmsImpl
                19:09:30,658 [DEBUG] org.jgroups.protocols.pbcast.GMS: ngic01juweb01:34373 became coordinator
                19:09:30,659 [DEBUG] org.jgroups.protocols.pbcast.ClientGmsImpl: created group (first member). My view is [ngic01juweb01:34373|0], impl is org.jgroups.protocols.pbcast.CoordGmsImpl
                19:09:30,673 [DEBUG] org.jgroups.protocols.pbcast.STABLE: stable task started; num_gossip_runs=3, max_gossip_runs=3
                19:09:30,682 [DEBUG] org.jgroups.protocols.FD: suspected_mbrs: [], after adjustment: [], stopped: true
                19:09:30,686 [DEBUG] org.jgroups.protocols.MERGE2: merge task started
                ** View=[ngic01juweb01:34373|0] [ngic01juweb01:34373]
                19:09:36,690 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:36,691 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:36,691 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=ngic01juweb01:34373), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:36,692 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.214.148:34373 (size=101 bytes)
                19:09:36,693 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: ngic01juweb01:34373 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:36,695 [DEBUG] org.jgroups.protocols.PING: received GET_MBRS_REQ from ngic01juweb01:34373, returning [PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]]
                19:09:36,696 [DEBUG] org.jgroups.protocols.UDP: sending message to ngic01juweb01:34373 (src=ngic01juweb01:34373), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:36,721 [DEBUG] org.jgroups.protocols.UDP: received (ucast) 263 bytes from /136.195.214.148:34373
                19:09:36,724 [DEBUG] org.jgroups.protocols.UDP: message is [dst: ngic01juweb01:34373, src: ngic01juweb01:34373 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:36,724 [DEBUG] org.jgroups.protocols.PING: received FIND_INITAL_MBRS_RSP, rsp=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]
                19:09:36,725 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=1965, got 1 rsps
                19:09:37,461 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 124 bytes from /136.195.214.148:34213 (size=124 bytes)
                19:09:37,461 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: ngic01juweb01:34213 (additional data: 20 bytes) (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DefaultPartition]}
                19:09:37,462 [WARN] org.jgroups.protocols.UDP: discarded message from different group (DefaultPartition). Sender was ngic01juweb01:34213 (additional data: 20 bytes)
                19:09:38,700 [DEBUG] org.jgroups.protocols.PING: initial mbrs are [[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]]
                19:09:38,701 [DEBUG] org.jgroups.protocols.MERGE2: initial_mbrs=[[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]]
                19:09:47,800 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:47,800 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:47,801 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=ngic01juweb01:34373), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:47,802 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.214.148:34373 (size=101 bytes)
                19:09:47,803 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: ngic01juweb01:34373 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:47,803 [DEBUG] org.jgroups.protocols.PING: received GET_MBRS_REQ from ngic01juweb01:34373, returning [PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]]
                19:09:47,804 [DEBUG] org.jgroups.protocols.UDP: sending message to ngic01juweb01:34373 (src=ngic01juweb01:34373), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:47,805 [DEBUG] org.jgroups.protocols.UDP: received (ucast) 263 bytes from /136.195.214.148:34373
                19:09:47,806 [DEBUG] org.jgroups.protocols.UDP: message is [dst: ngic01juweb01:34373, src: ngic01juweb01:34373 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:47,807 [DEBUG] org.jgroups.protocols.PING: received FIND_INITAL_MBRS_RSP, rsp=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]
                19:09:47,807 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=1993, got 1 rsps
                19:09:49,810 [DEBUG] org.jgroups.protocols.PING: initial mbrs are [[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]]
                19:09:49,810 [DEBUG] org.jgroups.protocols.MERGE2: initial_mbrs=[[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]]
                19:09:54,441 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 124 bytes from /136.195.214.148:34213 (size=124 bytes)
                19:09:54,443 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: ngic01juweb01:34213 (additional data: 20 bytes) (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DefaultPartition]}
                19:09:54,445 [WARN] org.jgroups.protocols.UDP: discarded message from different group (DefaultPartition). Sender was ngic01juweb01:34213 (additional data: 20 bytes)
                19:09:54,773 [DEBUG] org.jgroups.protocols.pbcast.STABLE: mcasting digest [ngic01juweb01:34373: [0 : 0] (num_gossip_runs=3, max_gossip_runs=3)
                19:09:54,774 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #0
                19:09:54,775 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 0]
                19:09:54,776 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=ngic01juweb01:34373), headers are {NAKACK=[NAKACK: MSG, seqno=0, range=null], STABLE=[STABLE_GOSSIP]: digest is [ngic01juweb01:34373: [0 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:54,780 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 271 bytes from /136.195.214.148:34373 (size=271 bytes)
                19:09:54,781 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: ngic01juweb01:34373 (3 headers), size = 0 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=0, range=null], STABLE=[STABLE_GOSSIP]: digest is [ngic01juweb01:34373: [0 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:54,782 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [ngic01juweb01:34373] received <ngic01juweb01:34373#0>
                19:09:54,783 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for ngic01juweb01:34373 is: received_msgs: [0 - 0 (size=1, missing=0)], delivered_msgs: [null - null]
                19:09:54,784 [DEBUG] org.jgroups.protocols.pbcast.STABLE: received digest ngic01juweb01:34373#0 (-1) from ngic01juweb01:34373
                19:09:54,784 [DEBUG] org.jgroups.protocols.pbcast.STABLE: sending stability msg ngic01juweb01:34373#0 (-1)
                19:09:54,785 [DEBUG] org.jgroups.protocols.pbcast.STABLE: stability_task=null, delay is 1493
                19:09:56,290 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sending msg #1
                19:09:56,291 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: sent_msgs: [0 - 0]
                19:09:56,291 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=ngic01juweb01:34373), headers are {NAKACK=[NAKACK: MSG, seqno=1, range=null], STABLE=[STABILITY]: digest is [ngic01juweb01:34373: [-1 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                (src=ngic01juweb01:34373), headers are {NAKACK=[NAKACK: MSG, seqno=1, range=null], STABLE=[STABILITY]: digest is [ngic01juweb01:34373: [-1 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:56,292 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 271 bytes from /136.195.214.148:34373 (size=271 bytes)
                19:09:56,294 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: ngic01juweb01:34373 (3 headers), size = 0 bytes], headers are {NAKACK=[NAKACK: MSG, seqno=1, range=null], STABLE=[STABILITY]: digest is [ngic01juweb01:34373: [-1 : 0], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:56,294 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: [ngic01juweb01:34373] received <ngic01juweb01:34373#1>
                19:09:56,295 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: receiver window for ngic01juweb01:34373 is: received_msgs: [1 - 1 (size=1, missing=0)], delivered_msgs: [0 - 0]
                19:09:56,295 [DEBUG] org.jgroups.protocols.pbcast.STABLE: stability vector is [ngic01juweb01:34373#0]
                19:09:56,296 [DEBUG] org.jgroups.protocols.pbcast.STABLE: cancelling stability task (running=false)
                19:09:56,296 [DEBUG] org.jgroups.protocols.pbcast.NAKACK: received digest [ngic01juweb01:34373: [-1 : 0]
                19:09:57,370 [DEBUG] org.jgroups.protocols.PING: FIND_INITIAL_MBRS
                19:09:57,370 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=2000, got 0 rsps
                19:09:57,370 [DEBUG] org.jgroups.protocols.UDP: sending message to 228.1.2.3:45566 (src=ngic01juweb01:34373), headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:57,372 [DEBUG] org.jgroups.protocols.UDP: received (mcast) 101 bytes from /136.195.214.148:34373 (size=101 bytes)
                19:09:57,372 [DEBUG] org.jgroups.protocols.UDP: message is [dst: 228.1.2.3:45566, src: ngic01juweb01:34373 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_REQ, arg=null], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:57,373 [DEBUG] org.jgroups.protocols.PING: received GET_MBRS_REQ from ngic01juweb01:34373, returning [PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]]
                19:09:57,373 [DEBUG] org.jgroups.protocols.UDP: sending message to ngic01juweb01:34373 (src=ngic01juweb01:34373), headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:57,374 [DEBUG] org.jgroups.protocols.UDP: received (ucast) 263 bytes from /136.195.214.148:34373
                19:09:57,376 [DEBUG] org.jgroups.protocols.UDP: message is [dst: ngic01juweb01:34373, src: ngic01juweb01:34373 (2 headers), size = 0 bytes], headers are {PING=[PING: type=GET_MBRS_RSP, arg=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]], UDP=[UDP:group_addr=DrawGroupDemo]}
                19:09:57,376 [DEBUG] org.jgroups.protocols.PING: received FIND_INITAL_MBRS_RSP, rsp=[own_addr=ngic01juweb01:34373, coord_addr=ngic01juweb01:34373]
                19:09:57,376 [DEBUG] org.jgroups.protocols.PING: waiting for initial members: time_to_wait=1994, got 1 rsps
                ^Cngic01juweb01%

                Unfortunately, when I try to run the same config file with the older jgroups.jar I get the following error:

                pegasus:/home/frdonam/software/jboss-3.2.3/server/all/lib% java -cp jgroups.jar org.jgroups.Version

                Version: 2.2.0
                CVS: $Id: Version.java,v 1.1.1.1 2003/09/09 01:24:08 belaban Exp $
                History: (see doc/history.txt for details)

                pegasus:/home/frdonam/software/jboss-3.2.3/server/all/lib% java -cp jgroups.jar org.jgroups.demos.Draw -props test.xml
                ChannelException: JChannel(): java.lang.Exception: Configurator.ProtocolConfiguration.createLayer(): unable to load class for protocol test.xml (either as an absolute - test.xml - or relative - org.jgroups.protocols.test.xml - package name)!
                at org.jgroups.JChannel.(JChannel.java:151)
                at org.jgroups.demos.Draw.(Draw.java:53)
                at org.jgroups.demos.Draw.main(Draw.java:113)
                pegasus:/home/frdonam/software/jboss-3.2.3/server/all/lib% java -cp jgroups.jar:./ org.jgroups.demos.Draw -props test.xml
                java.lang.NullPointerException
                at org.jgroups.conf.XmlConfigurator.parse(XmlConfigurator.java:130)
                at org.jgroups.conf.XmlConfigurator.getInstance(XmlConfigurator.java:70)
                at org.jgroups.conf.ConfiguratorFactory.getStackConfigurator(ConfiguratorFactory.java:67)
                at org.jgroups.JChannel.(JChannel.java:133)
                at org.jgroups.demos.Draw.(Draw.java:53)
                at org.jgroups.demos.Draw.main(Draw.java:113)
                [19:28:52[395]] [ERROR] XmlConfigurator: java.lang.NullPointerException
                at org.jgroups.conf.XmlConfigurator.parse(XmlConfigurator.java:130)
                at org.jgroups.conf.XmlConfigurator.getInstance(XmlConfigurator.java:70)
                at org.jgroups.conf.ConfiguratorFactory.getStackConfigurator(ConfiguratorFactory.java:67)
                at org.jgroups.JChannel.(JChannel.java:133)
                at org.jgroups.demos.Draw.(Draw.java:53)
                at org.jgroups.demos.Draw.main(Draw.java:113)

                ChannelException: JChannel(): java.lang.Exception: Configurator.ProtocolConfiguration.createLayer(): unable to load class for protocol test.xml (either as an absolute - test.xml - or relative - org.jgroups.protocols.test.xml - package name)!
                at org.jgroups.JChannel.(JChannel.java:151)
                at org.jgroups.demos.Draw.(Draw.java:53)
                at org.jgroups.demos.Draw.main(Draw.java:113)


                However, I assure you clustering works with that configuration under 3.2.3, and if I run Draw without properties I get the following:

                log4j/commons-logging not used? how do I get more output?

                First machine output:

                pegasus:/home/frdonam/software/jboss-3.2.3/server/all/lib% java -cp ./:jgroups.jar org.jgroups.Version

                Version: 2.2.0
                CVS: $Id: Version.java,v 1.1.1.1 2003/09/09 01:24:08 belaban Exp $
                History: (see doc/history.txt for details)

                pegasus:/home/frdonam/software/jboss-3.2.3/server/all/lib% java -cp jgroups.jar:./ org.jgroups.demos.Draw

                -------------------------------------------------------
                GMS: address is pegasus:59352
                -------------------------------------------------------
                ** View=[pegasus:59352|0] [pegasus:59352]
                [19:39:16[896]] [WARN] AckReceiverWindow.add(): discarded msg with seqno=66 (next msg to receive is 67)
                ** View=[pegasus:59352|1] [pegasus:59352, ngic01juweb01:34386]


                Second machine output:

                ngic01juweb01% java -cp ./:jgroups.jar org.jgroups.Version

                Version: 2.2.0
                CVS: $Id: Version.java,v 1.1.1.1 2003/09/09 01:24:08 belaban Exp $
                History: (see doc/history.txt for details)

                ngic01juweb01% java -cp jgroups.jar:./ org.jgroups.demos.Draw
                Trace.init() java.io.FileNotFoundException: /opt/jboss/jgroups.properties (No such file or directory)

                -------------------------------------------------------
                GMS: address is ngic01juweb01:34386
                -------------------------------------------------------
                ** View=[pegasus:59352|1] [pegasus:59352, ngic01juweb01:34386]


                If I run the 2.2.4 Draw program with no properties the machines never see each other, but multiple copies on the same machine show up.
                I can't get clustering to work under 3.2.4 with the cluster-service.xml that comes with it or the one from 3.2.3.

                Thanks for any help you can provide.
                I'll happily give you any more information you might need.

                Aaron

                • 5. Re: 3.2.4 cluster problem
                  amdonov

                  Upgrading to JGroups 2.2.5 fixed this problem.

                  • 6. Re: 3.2.4 cluster problem
                    julien.dubois

                    I think I'm having the same issue.... Oups.... I shouted after a sysadmin because I thought he misconfigured the network...
                    Anyway, I've another solution:
                    http://jroller.com/comments/julien.dubois?anchor=unicast_clustering_with_jboss

                    It's not as good as upgrading JavaGroups IMHO, but it solves the problem, so I thought it might interest people in this thread.

                    HTH.