4 Replies Latest reply on Aug 14, 2014 6:32 PM by Gerard Fulton

    Hot Rod not starting

    Gerard Fulton Newbie

      We have a four node cluster with each running a Hot Rod server instance.

       

        Below is a pasted log of an infinispan service starting but where the hot rod client port does not start…

       

        Here are some interesting lines:

       

      13:26:33,737 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010000: HotRodServer starting

      13:26:33,738 INFO  [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010001: HotRodServer listening on 192.168.136.129:11222

       

      13:27:34,880 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC00001: Failed to start service jboss.endpoint.hotrod.hotrod-connector: org.jboss.msc.service.StartException in service jboss.endpoint.hotrod.hotrod-connector: JDGS010004: Failed to start HotRodServer

       

      Is there anything in this log that indicates why the hot rod service fails to start?

       

      This machine(the cluster) is now in a state where repeated attempts of restarting the infinispan service will not result in a listener on 11222…

       

      On a working node (listener was active on 11222), a restart of the infinispan service fails to result in a running hot rod listener…

       

      If even just one of the nodes in the “existing/bad” cluster remains active, any new node starting will not start a listener on 11222…

       

      Only when all nodes in the cluster were stopped and then all were restarted did all new instances start a listener on 11222…

       

      Two logs attached, the second includes a shutdown sequence…

        • 1. Re: Hot Rod not starting
          Wolf-Dieter Fink Master

          Where do you run Infinispan and how do you configure it?

          An example and a bit information about the environment might be helpful

          • 2. Re: Hot Rod not starting
            Gerard Fulton Newbie

            Here is info on the configuration and how testing was done.

             

            The test environment is an Infinispan cluster on four virtual machines (RHEL 5.10, 2.6.18-371.9.1.el5). 

            The cluster is configured to communicate using TCP:

            cluster="flexnet" stack="tcp" />

            default-stack="${jboss.default.jgroups.stack:tcp}">

             

            The machines are statically configured to know about each other:

            <property name="initial_hosts">infinispan-test1a[7800],infinispan-test2a[7800],infinispan-test3a[7800],infinispan-test4a[7800],</property>

             

            On each server VM, there is a simple client which uses the hotrod interface. 

            The client contains a loop that 10 times a second just does a put.

            key = "12345";

            value = "a value";

            pprev = cache.put(key, value);

            All four clients use the same code (same key, same value).

             

            Each client is started with the IP address and TCP port of its local server and the cache name:

            For example:

            ./test1 192.168.136.131 11222 cacheTest

             

            Where cacheTest is defined as:

            <distributed-cache name="cacheTest" mode="SYNC"

            segments="20" owners="2" remote-timeout="30000" start="EAGER" />

            This test configuration will run forever with no reported errors if left alone…

             

            The following sequence of steps will eventually result in one or more nodes failing to start a hotrod listener on port 11222 after a restart:

             

            [Server1]#  service infinispan stop

            Wait 30 seconds

            [Server1]#  service infinispan start

            Wait 60 seconds while performing:  netstat -nat | grep 11222

            [Server2]#  service infinispan stop

            Wait 30 seconds

            [Server2]#  service infinispan start

            Wait 60 seconds while performing:  netstat -nat | grep 11222

            [Server3]#  service infinispan stop

            Wait 30 seconds

            [Server3]#  service infinispan start

            Wait 60 seconds while performing:  netstat -nat | grep 11222

            [Server4]#  service infinispan stop

            Wait 30 seconds

            [Server4]#  service infinispan start

            Wait 60 seconds while performing:  netstat -nat | grep 11222

            Repeat the above until the failure occurs…this may take up to 30 minutes…

             

             

            The pcap file has the capture starting where startProtocolServer(Configuration) is called in ProtocolServerService.java.

            • 3. Re: Hot Rod not starting
              Gerard Fulton Newbie

              We are running a four node cluster using hot rod. The issue that we are seeing happens when we restart a server and the server fails to start.  The issue cascades if we restart another server in the same cluster. There seems to be issues with state transfer.

               

              Below is the trace from when we see the node failing to start.

               

              5:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.statetransfer.StateConsumerImpl.start() on component org.infinispan.statetransfer.StateConsumer

              15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method private void org.infinispan.distribution.DistributionManagerImpl.start() throws java.lang.Exception on component org.infinispan.distribution.DistributionManager

              15:50:32,365 TRACE [org.infinispan.distribution.DistributionManagerImpl] (MSC service thread 1-1) starting distribution manager on infinispan-test2/flexnet

              15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.eviction.EvictionManagerImpl.start() on component org.infinispan.eviction.EvictionManager

              15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.persistence.manager.PersistenceManagerImpl.preload() on component org.infinispan.persistence.manager.PersistenceManager

              15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.statetransfer.StateProviderImpl.start() on component org.infinispan.statetransfer.StateProvider

              15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on component org.infinispan.statetransfer.StateTransferManager

              15:50:32,365 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (MSC service thread 1-1) Starting StateTransferManager of cache cacheTest on node infinispan-test2/flexnet

              15:50:32,367 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (MSC service thread 1-1) Node infinispan-test2/flexnet joining cache cacheTest

              15:50:32,368 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) dests=[infinispan-test3/flexnet], command=CacheTopologyControlCommand{cache=cacheTest, type=JOIN, sender=infinispan-test2/flexnet, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory@274744d4, hashFunction=MurmurHash3, numSegments=40, numOwners=2, timeout=60000, totalOrder=false, distributed=true}, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=49}, mode=SYNCHRONOUS, timeout=60000

              15:50:32,368 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Replication task sending CacheTopologyControlCommand{cache=cacheTest, type=JOIN, sender=infinispan-test2/flexnet, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory@274744d4, hashFunction=MurmurHash3, numSegments=40, numOwners=2, timeout=60000, totalOrder=false, distributed=true}, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=49} to single recipient infinispan-test3/flexnet with response mode GET_ALL

              15:50:32,370 TRACE [org.infinispan.commons.marshall.MarshallableTypeHints] (MSC service thread 1-1) Cache a buffer size predictor for 'org.infinispan.topology.CacheTopologyControlCommand' assuming its serializability is unknown

              15:50:32,370 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (MSC service thread 1-1) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 512

              15:50:32,382 TRACE [org.infinispan.marshall.core.VersionAwareMarshaller] (MSC service thread 1-1) Wrote version 510

              15:50:32,383 TRACE [org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller] (MSC service thread 1-1) Stop marshaller

              15:50:32,385 TRACE [org.jgroups.blocks.Request] (MSC service thread 1-1) sending request (id=2)

              15:50:32,386 TRACE [org.jgroups.protocols.RSVP] (MSC service thread 1-1) infinispan-test2/flexnet: REQ --> infinispan-test3/flexnet

              15:50:32,386 TRACE [org.jgroups.protocols.UNICAST3] (MSC service thread 1-1) infinispan-test2/flexnet --> DATA(infinispan-test3/flexnet: #3, conn_id=0)

              15:50:32,386 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) null: sending msg to infinispan-test3/flexnet, src=infinispan-test2/flexnet, headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]

              15:50:32,387 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) dest=192.168.135.135:7800 (264 bytes)

              15:50:32,396 TRACE [org.jgroups.protocols.TCP] (Incoming-2,shared=tcp) null: received [dst: infinispan-test2/flexnet, src: infinispan-test3/flexnet (3 headers), size=0 bytes, flags=DONT_BUNDLE|RSVP|INTERNAL], headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=56, TCP: [channel_name=clustered]

              15:50:32,397 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-2,shared=tcp) infinispan-test2/flexnet <-- DATA(infinispan-test3/flexnet: #2, conn_id=56)

              15:50:32,397 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-2,shared=tcp) infinispan-test2/flexnet: delivering #2 - #2 (1 messages)

              15:50:32,423 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: received [dst: infinispan-test2/flexnet, src: infinispan-test3/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=3, TCP: [channel_name=clustered]

              15:50:32,423 TRACE [org.jgroups.protocols.UNICAST3] (INT-1,shared=tcp) infinispan-test2/flexnet <-- ACK(infinispan-test3/flexnet: #3, conn-id=0)

              15:50:32,526 TRACE [org.jgroups.protocols.UNICAST3] (Timer-2,shared=tcp) infinispan-test2/flexnet --> ACK(infinispan-test3/flexnet: #2)

              15:50:32,527 TRACE [org.jgroups.protocols.TCP] (Timer-2,shared=tcp) null: sending msg to infinispan-test3/flexnet, src=infinispan-test2/flexnet, headers are UNICAST3: ACK, seqno=2, conn_id=56, TCP: [channel_name=clustered]

              15:50:32,527 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (123 bytes (0.19% of max_bundle_size) to 1 destination(s)

              15:50:32,527 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.135:7800 (126 bytes)

              15:50:32,812 DEBUG [org.jgroups.protocols.FD] (Timer-5,shared=tcp) infinispan-test2/flexnet: sending are-you-alive msg to infinispan-test3/flexnet

              15:50:32,812 TRACE [org.jgroups.protocols.TCP] (Timer-5,shared=tcp) null: sending msg to infinispan-test3/flexnet, src=infinispan-test2/flexnet, headers are FD: heartbeat, TCP: [channel_name=clustered]

              15:50:32,812 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (122 bytes (0.19% of max_bundle_size) to 1 destination(s)

              15:50:32,812 TRACE [org.jgroups.protocols.TCP] (TransferQueue

               

               

              15:50:32,025 TRACE [org.jgroups.protocols.TCP] (INT-2,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=1, conn_id=56, TCP: [channel_name=clustered]

              15:50:32,025 TRACE [org.jgroups.protocols.UNICAST3] (INT-2,shared=tcp) infinispan-test3/flexnet <-- ACK(infinispan-test2/flexnet: #1, conn-id=56)

              15:50:32,169 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are FD_SOCK: GET_CACHE, TCP: [channel_name=clustered]

              15:50:32,169 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: sending msg to infinispan-test2/flexnet, src=infinispan-test3/flexnet, headers are FD_SOCK: GET_CACHE_RSP, TCP: [channel_name=clustered]

              15:50:32,169 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (353 bytes (0.55% of max_bundle_size) to 1 destination(s)

              15:50:32,169 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.134:7800 (356 bytes)

              15:50:32,171 TRACE [org.jgroups.protocols.FD_SOCK] (FD_SOCK acceptor,clustered,infinispan-test3/flexnet) accepted connection from /192.168.135.134:39759

              15:50:32,171 TRACE [org.jgroups.protocols.FD_SOCK] (FD_SOCK acceptor,clustered,infinispan-test3/flexnet) waiting for client connections on bind-node/192.168.135.135:57600

              15:50:32,295 DEBUG [org.jgroups.protocols.FD] (Timer-4,shared=tcp) infinispan-test3/flexnet: sending are-you-alive msg to infinispan-test4/flexnet

              15:50:32,295 TRACE [org.jgroups.protocols.TCP] (Timer-4,shared=tcp) null: sending msg to infinispan-test4/flexnet, src=infinispan-test3/flexnet, headers are FD: heartbeat, TCP: [channel_name=clustered]

              15:50:32,295 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (122 bytes (0.19% of max_bundle_size) to 1 destination(s)

              15:50:32,295 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.136:7800 (125 bytes)

              15:50:32,296 TRACE [org.jgroups.protocols.TCP] (INT-2,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test4/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are FD: heartbeat ack, TCP: [channel_name=clustered]

              15:50:32,388 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (4 headers), size=116 bytes, flags=OOB|DONT_BUNDLE|NO_TOTAL_ORDER|RSVP], headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]

              15:50:32,388 TRACE [org.jgroups.protocols.UNICAST3] (INT-1,shared=tcp) infinispan-test3/flexnet <-- DATA(infinispan-test2/flexnet: #3, conn_id=0)

              15:50:32,388 TRACE [org.jgroups.protocols.UNICAST3] (INT-1,shared=tcp) infinispan-test3/flexnet: delivering infinispan-test2/flexnet#3

              15:50:32,388 TRACE [org.jgroups.protocols.UFC] (INT-1,shared=tcp) infinispan-test2/flexnet used 116 credits, 1999884 remaining

              15:50:32,388 TRACE [org.jgroups.protocols.RSVP] (INT-1,shared=tcp) infinispan-test3/flexnet: REQ <-- infinispan-test2/flexnet

              15:50:32,388 TRACE [org.jgroups.protocols.RSVP] (INT-1,shared=tcp) infinispan-test3/flexnet: RSP --> infinispan-test2/flexnet

              15:50:32,388 TRACE [org.jgroups.protocols.UNICAST3] (INT-1,shared=tcp) infinispan-test3/flexnet --> DATA(infinispan-test2/flexnet: #2, conn_id=56)

              15:50:32,388 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: sending msg to infinispan-test2/flexnet, src=infinispan-test3/flexnet, headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=56, TCP: [channel_name=clustered]

              15:50:32,388 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) dest=192.168.135.134:7800 (134 bytes)

              15:50:32,389 TRACE [org.jgroups.blocks.RequestCorrelator] (INT-1,shared=tcp) calling (org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher) with request 2

              15:50:32,423 TRACE [org.jgroups.protocols.UNICAST3] (Timer-3,shared=tcp) infinispan-test3/flexnet --> ACK(infinispan-test2/flexnet: #3)

              15:50:32,423 TRACE [org.jgroups.protocols.TCP] (Timer-3,shared=tcp) null: sending msg to infinispan-test2/flexnet, src=infinispan-test3/flexnet, headers are UNICAST3: ACK, seqno=3, TCP: [channel_name=clustered]

              15:50:32,423 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (123 bytes (0.19% of max_bundle_size) to 1 destination(s)

              15:50:32,423 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.134:7800 (126 bytes)

              15:50:32,528 TRACE [org.jgroups.protocols.TCP] (INT-2,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=2, conn_id=56, TCP: [channel_name=clustered]

              15:50:32,528 TRACE [org.jgroups.protocols.UNICAST3] (INT-2,shared=tcp) infinispan-test3/flexnet <-- ACK(infinispan-test2/flexnet: #2, conn-id=56)

              15:50:32,813 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are FD: heartbeat, TCP: [channel_name=clustered]

              15:50:32,813 TRACE [org.jgroups.protocols.FD] (INT-1,shared=tcp) infinispan-test3/flexnet: received are-you-alive from infinispan-test2/flexnet, sending response

              15:50:32,813 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: sending msg to infinispan-test2/flexnet, src=infinispan-test3/flexnet, headers are FD: heartbeat ack, TCP: [channel_name=clustered]

              15:50:32,813 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (168 bytes (0.26% of max_bundle_size) to 1 destination(s)

              15:50:32,813 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.134:7800 (171 bytes)

              15:50:33,296 DEBUG [org.jgroups.protocols.FD] (Timer-3,shared=tcp) infinispan-test3/flexnet:

              • 4. Re: Hot Rod not starting
                Gerard Fulton Newbie

                Starting Infinispan service fails to start HotRod listener…

                 

                Two traces, first one shows the failure… the second shows success…

                 

                infinispan-test1a 192.168.136.128

                infinispan-test4a 192.168.136.131

                 

                 

                Failure case:

                 

                08:44:37,191 TRACE [org.jgroups.blocks.Request] (MSC service thread 1-2) sending request (id=2)

                08:44:37,192 TRACE [org.jgroups.protocols.RSVP] (MSC service thread 1-2) infinispan-test4a/flexnet: REQ --> infinispan-test1a/flexnet

                08:44:37,193 TRACE [org.jgroups.protocols.UNICAST3] (MSC service thread 1-2) infinispan-test4a/flexnet --> DATA(infinispan-test1a/flexnet: #3, conn_id=0)

                08:44:37,193 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-2) null: sending msg to infinispan-test1a/flexnet, src=infinispan-test4a/flexnet, headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]

                08:44:37,193 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-2) dest=192.168.136.128:7800 (267 bytes)

                 

                08:44:37,195 TRACE [org.jgroups.protocols.TCP] (Incoming-4,shared=tcp) null: received [dst: infinispan-test4a/flexnet, src: infinispan-test1a/flexnet (3 headers), size=0 bytes, flags=DONT_BUNDLE|RSVP|INTERNAL], headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=25, TCP: [channel_name=clustered]

                08:44:37,195 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet <-- DATA(infinispan-test1a/flexnet: #2, conn_id=25)

                08:44:37,196 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet: delivering #2 - #2 (1 messages)

                 

                One minute later…

                 

                08:45:37,212 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC00001: Failed to start service jboss.endpoint.hotrod.hotrod-connector: org.jboss.msc.service.StartException in service jboss.endpoint.hotrod.hotrod-connector: JDGS010004: Failed to start HotRodServer

                at org.infinispan.server.endpoint.subsystem.ProtocolServerService.start(ProtocolServerService.java:113)

                at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

                at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]

                at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]

                Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

                 

                 

                /usr/sbin/tcpdump -ieth1 -s512 -vv -nn -XX -w /tmp/tcpdump tcp port 7800

                 

                (tcp handshake packets removed for clarity)

                 

                08:44:37.193444 IP (tos 0x0, ttl  64, id 47273, offset 0, flags [DF], proto: TCP (6), length: 323) 192.168.136.131.52964 > 192.168.136.128.7800: P, cksum 0x938a ( (-> 0x50e9), 936:1207(271) ack 1422 win 88 <nop,nop,timestamp 1463150373 1463372874>

                0x0000:  000c 29fc e9eb 000c 291e 7876 0800 4500  ..).....).xv..E.

                0x0010:  0143 b8a9 4000 4006 eeb6 c0a8 8883 c0a8  .C..@.@.........

                0x0020:  8880 cee4 1e78 bfeb 5ee0 1676 68d9 8018  .....x..^..vh...

                0x0030:  0058 938a 0000 0101 080a 5735 e725 5739  .X........W5.%W9

                0x0040:  4c4a 0000 010b 1901 0007 00a3 0000 5496  LJ............T.

                0x0050:  0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....

                0x0060:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x0070:  6973 7061 6e2d 7465 7374 3161 0000 5410  ispan-test1a..T.

                0x0080:  ab31 738c 8f58 f744 6752 7dd2 8cf4 c300  .1s..X.DgR}.....

                0x0090:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x00a0:  6973 7061 6e2d 7465 7374 3461 0000 0075  ispan-test4a...u

                0x00b0:  0301 fe03 3e00 0011 093e 0963 6163 6865 ....>....>.cache

                0x00c0:  5465 7374 4900 033c 0000 5410 ab31 738c  TestI..<..T..1s.

                0x00d0:  8f58 f744 6752 7dd2 8cf4 c300 0100 0000  .X.DgR}.........

                0x00e0:  0552 6f77 2d31 0100 0000 1169 6e66 696e  .Row-1.....infin

                0x00f0:  6973 7061 6e2d 7465 7374 3461 0357 035e  ispan-test4a.W.^

                0x0100:  0349 0000 0028 0000 0002 0000 0000 0000  .I...(..........

                0x0110:  ea60 0001 3f80 0000 4b00 0000 0001 0101  .`..?...K.......

                0x0120:  4b00 0000 1700 0400 c800 4700 0102 0100  K.........G.....

                0x0130:  c800 3700 5b01 0000 0040 006b 0001 0300  ..7.[....@.k....

                0x0140:  0000 0016 0038 0009 636c 7573 7465 7265  .....8..clustere

                0x0150: 64 d

                 

                08:44:37.194778 IP (tos 0x0, ttl  64, id 10641, offset 0, flags [DF], proto: TCP (6), length: 192) 192.168.136.128.7800 > 192.168.136.131.52964: P, cksum 0x64e9 (correct), 1422:1562(140) ack 1207 win 104 <nop,nop,timestamp 1463373385 1463150373>

                0x0000:  000c 291e 7876 000c 29fc e9eb 0800 4500  ..).xv..).....E.

                0x0010:  00c0 2991 4000 4006 7e52 c0a8 8880 c0a8  ..).@.@.~R......

                0x0020:  8883 1e78 cee4 1676 68d9 bfeb 5fef 8018  ...x...vh..._...

                0x0030:  0068 64e9 0000 0101 080a 5739 4e49 5735  .hd.......W9NIW5

                0x0040:  e725 0000 0088 1901 0003 0182 0000 5410  .%............T.

                0x0050:  ab31 738c 8f58 f744 6752 7dd2 8cf4 c300  .1s..X.DgR}.....

                0x0060:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x0070:  6973 7061 6e2d 7465 7374 3461 0000 5496  ispan-test4a..T.

                0x0080:  0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....

                0x0090:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x00a0:  6973 7061 6e2d 7465 7374 3161 0003 0037  ispan-test1a...7

                0x00b0:  005b 0300 0000 4000 6b00 0102 0019 0000  .[....@.k.......

                0x00c0:  1600 3800 0963 6c75 7374 6572 6564       ..8..clustered

                 

                08:44:37.407984 IP (tos 0x0, ttl  64, id 47275, offset 0, flags [DF], proto: TCP (6), length: 183) 192.168.136.131.52964 > 192.168.136.128.7800: P, cksum 0x92fe ( (-> 0xf1ec), 1207:1338(131) ack 1562 win 96 <nop,nop,timestamp 1463150588 1463373385>

                0x0000:  000c 29fc e9eb 000c 291e 7876 0800 4500  ..).....).xv..E.

                0x0010:  00b7 b8ab 4000 4006 ef40 c0a8 8883 c0a8  ....@.@..@......

                0x0020:  8880 cee4 1e78 bfeb 5fef 1676 6965 8018  .....x.._..vie..

                0x0030:  0060 92fe 0000 0101 080a 5735 e7fc 5739  .`........W5..W9

                0x0040:  4e49 0000 007f 1901 0003 0100 0000 5496  NI............T.

                0x0050:  0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....

                0x0060:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x0070:  6973 7061 6e2d 7465 7374 3161 0000 5410  ispan-test1a..T.

                0x0080:  ab31 738c 8f58 f744 6752 7dd2 8cf4 c300  .1s..X.DgR}.....

                0x0090:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x00a0:  6973 7061 6e2d 7465 7374 3461 0002 0002  ispan-test4a....

                0x00b0:  0003 00ff ff01 0016 0038 0009 636c 7573  .........8..clus

                0x00c0:  7465 7265 64 tered

                 

                08:44:37.408843 IP (tos 0x0, ttl  64, id 10642, offset 0, flags [DF], proto: TCP (6), length: 230) 192.168.136.128.7800 > 192.168.136.131.52964: P, cksum 0xd1e3 (correct), 1562:1740(178) ack 1338 win 113 <nop,nop,timestamp 1463373598 1463150588>

                0x0000:  000c 291e 7876 000c 29fc e9eb 0800 4500  ..).xv..).....E.

                0x0010:  00e6 2992 4000 4006 7e2b c0a8 8880 c0a8  ..).@.@.~+......

                0x0020:  8883 1e78 cee4 1676 6965 bfeb 6072 8018  ...x...vie..`r..

                0x0030:  0071 d1e3 0000 0101 080a 5739 4f1e 5735  .q........W9O.W5

                0x0040:  e7fc 0000 00ae 1901 0003 0100 0000 5410  ..............T.

                0x0050:  ab31 738c 8f58 f744 6752 7dd2 8cf4 c300  .1s..X.DgR}.....

                0x0060:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x0070:  6973 7061 6e2d 7465 7374 3461 0000 5496  ispan-test4a..T.

                0x0080:  0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....

                0x0090:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x00a0:  6973 7061 6e2d 7465 7374 3161 0002 0002  ispan-test1a....

                0x00b0:  0003 01ff ff00 0054 9608 6726 a4bb 5db1 .......T..g&..].

                0x00c0:  fbc0 b5f7 5f04 b885 0001 0005 526f 772d  ...._.......Row-

                0x00d0:  3101 0011 696e 6669 6e69 7370 616e 2d74  1...infinispan-t

                0x00e0:  6573 7431 6100 1600 3800 0963 6c75 7374  est1a...8..clust

                0x00f0:  6572 6564 ered

                 

                 

                No other network traffic for this tcpdump filter…Timeout after 60 seconds… hotrod fails to start…

                 

                 

                Good (working) case:

                 

                10:46:40,993 TRACE [org.jgroups.blocks.Request] (MSC service thread 1-1) sending request (id=2)

                10:46:40,995 TRACE [org.jgroups.protocols.RSVP] (MSC service thread 1-1) infinispan-test4a/flexnet: REQ --> infinispan-test1a/flexnet

                10:46:40,995 TRACE [org.jgroups.protocols.UNICAST3] (MSC service thread 1-1) infinispan-test4a/flexnet --> DATA(infinispan-test1a/flexnet: #3, conn_id=0)

                10:46:40,995 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) null: sending msg to infinispan-test1a/flexnet, src=infinispan-test4a/flexnet, headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]

                10:46:40,995 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) dest=192.168.136.128:7800 (267 bytes)

                 

                10:46:41,006 TRACE [org.jgroups.protocols.TCP] (Incoming-4,shared=tcp) null: received [dst: infinispan-test4a/flexnet, src: infinispan-test1a/flexnet (3 headers), size=0 bytes, flags=DONT_BUNDLE|RSVP|INTERNAL], headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=4, TCP: [channel_name=clustered]

                10:46:41,006 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet <-- DATA(infinispan-test1a/flexnet: #2, conn_id=4)

                10:46:41,006 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet: delivering #2 - #2 (1 messages)

                 

                Less than a half-second later…

                 

                10:46:41,499 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Response: SuccessfulResponse{responseValue=CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}}

                10:46:41,499 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (MSC service thread 1-1) Updating local consistent hash(es) for cache cacheTest: new topology = CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}

                10:46:41,500 TRACE [org.infinispan.topology.CacheTopology] (MSC service thread 1-1) Current consistent hash's routing table: 0: 0 2, 1: 0 2, 2: 0 2, 3: 0 2, 4: 0 2, 5: 0 2, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0 1, 13: 0 1, 14: 2 0, 15: 2 1, 16: 2 0, 17: 2 1, 18: 2 0, 19: 2 1, 20: 1 2, 21: 1 2, 22: 1 2, 23: 1 2, 24: 1 2, 25: 1 2, 26: 1 2, 27: 1 0, 28: 1 0, 29: 1 0, 30: 1 0, 31: 1 0, 32: 1 0, 33: 2 0, 34: 2 1, 35: 2 0, 36: 2 1, 37: 2 0, 38: 2 1, 39: 2 0

                10:46:41,500 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (MSC service thread 1-1) Installing new cache topology CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null} on cache cacheTest

                10:46:41,502 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) Received new topology for cache cacheTest, isRebalance = false, isMember = false, topology = CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}

                10:46:41,502 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (MSC service thread 1-1) Signalling topology 4 is installed

                10:46:41,502 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) On cache cacheTest we have: added segments: []

                10:46:41,503 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) Topology update processed, rebalanceInProgress = false, isRebalance = false, pending CH = null

                10:46:41,503 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (MSC service thread 1-1) Signalling transaction data received for topology 4

                10:46:41,503 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (MSC service thread 1-1) Tasks executed=0, still pending=0

                10:46:41,503 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Checking for transactions originated on leavers. Current members are [infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet], remote transactions: 0

                10:46:41,504 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) No global transactions pertain to originator(s) who have left the cluster.

                10:46:41,504 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Completed cleaning transactions originating on leavers. Remote transactions remaining: 0

                10:46:41,504 DEBUG [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Topology changed, recalculating minTopologyId

                10:46:41,505 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Changing minimum topology ID from -1 to 4

                 

                 

                10:46:40.996039 IP (tos 0x0, ttl  64, id 37778, offset 0, flags [DF], proto: TCP (6), length: 323) 192.168.136.131.51280 > 192.168.136.128.7800: P, cksum 0x938a ( (-> 0x0c63), 936:1207(271) ack 1422 win 88 <nop,nop,timestamp 1470474025 1470696498>

                0x0000:  000c 29fc e9eb 000c 291e 7876 0800 4500  ..).....).xv..E.

                0x0010:  0143 9392 4000 4006 13ce c0a8 8883 c0a8  .C..@.@.........

                0x0020:  8880 c850 1e78 b330 359d 28f9 4677 8018  ...P.x.05.(.Fw..

                0x0030:  0058 938a 0000 0101 080a 57a5 a729 57a9  .X........W..)W.

                0x0040:  0c32 0000 010b 1901 0007 00a3 0000 545c  .2............T\

                0x0050:  0787 e601 6f91 fddc 03ab e137 15e0 a500  ....o......7....

                0x0060:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x0070:  6973 7061 6e2d 7465 7374 3161 0000 5428  ispan-test1a..T(

                0x0080:  a557 dca7 9945 6712 a54e 33d2 9bd3 e700  .W...Eg..N3.....

                0x0090:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x00a0:  6973 7061 6e2d 7465 7374 3461 0000 0075  ispan-test4a...u

                0x00b0:  0301 fe03 3e00 0011 093e 0963 6163 6865 ....>....>.cache

                0x00c0:  5465 7374 4900 033c 0000 5428 a557 dca7  TestI..<..T(.W..

                0x00d0:  9945 6712 a54e 33d2 9bd3 e700 0100 0000  .Eg..N3.........

                0x00e0:  0552 6f77 2d31 0100 0000 1169 6e66 696e  .Row-1.....infin

                0x00f0:  6973 7061 6e2d 7465 7374 3461 0357 035e  ispan-test4a.W.^

                0x0100:  0349 0000 0028 0000 0002 0000 0000 0000  .I...(..........

                0x0110:  ea60 0001 3f80 0000 4b00 0000 0001 0101  .`..?...K.......

                0x0120:  4b00 0000 0300 0400 c800 4700 0102 0100  K.........G.....

                0x0130:  c800 3700 5b01 0000 0040 006b 0001 0300  ..7.[....@.k....

                0x0140:  0000 0016 0038 0009 636c 7573 7465 7265  .....8..clustere

                0x0150: 64 d

                 

                 

                10:46:40.997457 IP (tos 0x0, ttl  64, id 18822, offset 0, flags [DF], proto: TCP (6), length: 192) 192.168.136.128.7800 > 192.168.136.131.51280: P, cksum 0x104c (correct), 1422:1562(140) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>

                0x0000:  000c 291e 7876 000c 29fc e9eb 0800 4500  ..).xv..).....E.

                0x0010:  00c0 4986 4000 4006 5e5d c0a8 8880 c0a8  ..I.@.@.^]......

                0x0020:  8883 1e78 c850 28f9 4677 b330 36ac 8018  ...x.P(.Fw.06...

                0x0030:  0068 104c 0000 0101 080a 57a9 0e4c 57a5  .h.L......W..LW.

                0x0040:  a729 0000 0088 1901 0003 0182 0000 5428  .)............T(

                0x0050:  a557 dca7 9945 6712 a54e 33d2 9bd3 e700  .W...Eg..N3.....

                0x0060:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x0070:  6973 7061 6e2d 7465 7374 3461 0000 545c  ispan-test4a..T\

                0x0080:  0787 e601 6f91 fddc 03ab e137 15e0 a500  ....o......7....

                0x0090:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x00a0:  6973 7061 6e2d 7465 7374 3161 0003 0037  ispan-test1a...7

                0x00b0:  005b 0300 0000 4000 6b00 0102 0004 0000  .[....@.k.......

                0x00c0:  1600 3800 0963 6c75 7374 6572 6564       ..8..clustered

                 

                 

                10:46:40.997965 IP (tos 0x0, ttl  64, id 18823, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 1562:3010(1448) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>

                0x0000:  000c 291e 7876 000c 29fc e9eb 0800 4500  ..).xv..).....E.

                0x0010:  05dc 4987 4000 4006 5940 c0a8 8880 c0a8  ..I.@.@.Y@......

                0x0020:  8883 1e78 c850 28f9 4703 b330 36ac 8010  ...x.P(.G..06...

                0x0030:  0068 8eff 0000 0101 080a 57a9 0e4c 57a5  .h........W..LW.

                0x0040:  a729 0000 14cc 1901 0007 0023 0000 5428  .).........#..T(

                0x0050:  a557 dca7 9945 6712 a54e 33d2 9bd3 e700  .W...Eg..N3.....

                0x0060:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x0070:  6973 7061 6e2d 7465 7374 3461 0000 545c  ispan-test4a..T\

                0x0080:  0787 e601 6f91 fddc 03ab e137 15e0 a500  ....o......7....

                0x0090:  0100 0552 6f77 2d31 0100 1169 6e66 696e  ...Row-1...infin

                0x00a0:  6973 7061 6e2d 7465 7374 3161 0000 143d  ispan-test1a...=

                0x00b0:  0301 fe03 0501 034b 0000 0004 0333 0000  .......K.....3..

                0x00c0:  0028 0000 0002 0300 0003 033c 0000 545c  .(.........<..T\

                0x00d0:  0787 e601 6f91 fddc 03ab e137 15e0 a500  ....o......7....

                0x00e0:  0100 0000 0552 6f77 2d31 0100 0000 1169  .....Row-1.....i

                0x00f0:  6e66 696e 6973 7061 6e2d 7465 7374 3161  nfinispan-test1a

                0x0100:  033c 0000 54fe 1b3a e5c1 c349 e45f c0fe  .<..T..:...I._..

                0x0110:  3efb 74d8 8100 0100 0000 0552 6f77 2d31  >.t........Row-1

                0x0120:  0100 0000 1169 6e66 696e 6973 7061 6e2d  .....infinispan-

                0x0130:  7465 7374 3261 033c 0000 54f9 56b4 d57b  test2a.<..T.V..{

                0x0140:  cc23 ee16 266c 457e 12fd 8a00 0100 0000 .#..&lE~........

                0x0150:  0552 6f77 2d31 0100 0000 1169 6e66 696e  .Row-1.....infin

                0x0160:  6973 7061 6e2d 7465 7374 3361 0302 0003 ispan-test3a....

                0x0170:  033c 0000 54f9 56b4 d57b cc23 ee16 266c .<..T.V..{.#..&l

                0x0180:  457e 12fd 8a00 0100 0000 0552 6f77 2d31  E~.........Row-1

                0x0190:  0100 0000 1169 6e66 696e 6973 7061 6e2d  .....infinispan-

                0x01a0:  7465 7374 3361 4e3f 8000 0003 3c00 0054  test3aN?....<..T

                0x01b0:  5c07 87e6 016f 91fd dc03 abe1 3715 e0a5  \....o......7...

                0x01c0:  0001 0000 0005 526f 772d 3101 0000 0011  ......Row-1.....

                0x01d0:  696e 6669 6e69 7370 616e 2d74 6573 7431  infinispan-test1

                0x01e0:  614e 3f80 0000 033c 0000 54fe 1b3a e5c1  aN?....<..T..:..

                0x01f0:  c349 e45f c0fe 3efb 74d8 8100 0100 0000  .I._..>.t.......

                 

                10:46:40.997991 IP (tos 0x0, ttl  64, id 18824, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 3010:4458(1448) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>

                0x0000:  000c 291e 7876 000c 29fc e9eb 0800 4500  ..).xv..).....E.

                0x0010:  05dc 4988 4000 4006 593f c0a8 8880 c0a8  ..I.@.@.Y?......

                0x0020:  8883 1e78 c850 28f9 4cab b330 36ac 8010  ...x.P(.L..06...

                0x0030:  0068 5708 0000 0101 080a 57a9 0e4c 57a5  .hW.......W..LW.

                0x0040:  a729 7370 616e 2d74 6573 7431 6103 3c00  .)span-test1a.<.

                0x0050:  0054 fe1b 3ae5 c1c3 49e4 5fc0 fe3e fb74  .T..:...I._..>.t

                0x0060:  d881 0001 0000 0005 526f 772d 3101 0000  ........Row-1...

                0x0070:  0011 696e 6669 6e69 7370 616e 2d74 6573  ..infinispan-tes

                0x0080:  7432 6135 043b ff32 0400 0000 0203 3c00  t2a5.;.2......<.

                0x0090:  0054 5c07 87e6 016f 91fd dc03 abe1 3715  .T\....o......7.

                0x00a0:  e0a5 0001 0000 0005 526f 772d 3101 0000  ........Row-1...

                0x00b0:  0011 696e 6669 6e69 7370 616e 2d74 6573  ..infinispan-tes

                0x00c0:  7431 6103 3c00 0054 fe1b 3ae5 c1c3 49e4  t1a.<..T..:...I.

                0x00d0:  5fc0 fe3e fb74 d881 0001 0000 0005 526f  _..>.t........Ro

                0x00e0:  772d 3101 0000 0011 696e 6669 6e69 7370  w-1.....infinisp

                0x00f0:  616e 2d74 6573 7432 6135 043b ff32 0400  an-test2a5.;.2..

                0x0100:  0000 0203 3c00 0054 5c07 87e6 016f 91fd  ....<..T\....o..

                0x0110:  dc03 abe1 3715 e0a5 0001 0000 0005 526f  ....7.........Ro

                0x0120:  772d 3101 0000 0011 696e 6669 6e69 7370  w-1.....infinisp

                0x0130:  616e 2d74 6573 7431 6103 3c00 0054 fe1b  an-test1a.<..T..

                0x0140:  3ae5 c1c3 49e4 5fc0 fe3e fb74 d881 0001  :...I._..>.t....

                0x0150:  0000 0005 526f 772d 3101 0000 0011 696e  ....Row-1.....in

                0x0160:  6669 6e69 7370 616e 2d74 6573 7432 6135  finispan-test2a5

                0x0170:  043b ff32 0400 0000 0203 3c00 0054 5c07  .;.2......<..T\.

                0x0180:  87e6 016f 91fd dc03 abe1 3715 e0a5 0001  ...o......7.....

                0x0190:  0000 0005 526f 772d 3101 0000 0011 696e  ....Row-1.....in

                0x01a0:  6669 6e69 7370 616e 2d74 6573 7431 6103  finispan-test1a.

                0x01b0:  3c00 0054 fe1b 3ae5 c1c3 49e4 5fc0 fe3e <..T..:...I._..>

                0x01c0:  fb74 d881 0001 0000 0005 526f 772d 3101  .t........Row-1.

                0x01d0:  0000 0011 696e 6669 6e69 7370 616e 2d74  ....infinispan-t

                0x01e0:  6573 7432 6135 043b ff32 0400 0000 0203  est2a5.;.2......

                0x01f0:  3c00 0054 5c07 87e6 016f 91fd dc03 abe1  <..T\....o......

                 

                 

                10:46:40.999497 IP (tos 0x0, ttl  64, id 18825, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 4458:5906(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474026>

                 

                10:46:40.999520 IP (tos 0x0, ttl  64, id 18826, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 5906:7354(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474026>

                 

                10:46:40.999541 IP (tos 0x0, ttl  64, id 18827, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 7354:8802(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>

                 

                10:46:40.999556 IP (tos 0x0, ttl  64, id 18828, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 8802:10250(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>

                 

                10:46:40.999574 IP (tos 0x0, ttl  64, id 18829, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 10250:11698(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>

                 

                10:46:40.999590 IP (tos 0x0, ttl  64, id 18830, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: P 11698:13146(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>

                 

                10:46:41.002149 IP (tos 0x0, ttl  64, id 18831, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 13146:14594(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>

                 

                10:46:41.002174 IP (tos 0x0, ttl  64, id 18832, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 14594:16042(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>

                 

                10:46:41.002239 IP (tos 0x0, ttl  64, id 18833, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 16042:17490(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>

                 

                10:46:41.002263 IP (tos 0x0, ttl  64, id 18834, offset 0, flags [DF], proto: TCP (6), length: 57) 192.168.136.128.7800 > 192.168.136.131.51280: P, cksum 0xb581 (correct), 17490:17495(5) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>

                0x0000:  000c 291e 7876 000c 29fc e9eb 0800 4500  ..).xv..).....E.

                0x0010:  0039 4992 4000 4006 5ed8 c0a8 8880 c0a8  .9I.@.@.^.......

                0x0020:  8883 1e78 c850 28f9 853b b330 36ac 8018  ...x.P(..;.06...

                0x0030:  0068 b581 0000 0101 080a 57a9 0e51 57a5  .h........W..QW.

                0x0040:  a72c 7465 7265 64 .,tered