4 Replies Latest reply on Aug 14, 2014 6:32 PM by fulton

Hot Rod not starting

fulton Aug 5, 2014 3:04 PM

We have a four node cluster with each running a Hot Rod server instance.

Below is a pasted log of an infinispan service starting but where the hot rod client port does not start…

Here are some interesting lines:

13:26:33,737 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010000: HotRodServer starting

13:26:33,738 INFO [org.infinispan.server.endpoint] (MSC service thread 1-1) JDGS010001: HotRodServer listening on 192.168.136.129:11222

13:27:34,880 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-1) MSC00001: Failed to start service jboss.endpoint.hotrod.hotrod-connector: org.jboss.msc.service.StartException in service jboss.endpoint.hotrod.hotrod-connector: JDGS010004: Failed to start HotRodServer

Is there anything in this log that indicates why the hot rod service fails to start?

This machine(the cluster) is now in a state where repeated attempts of restarting the infinispan service will not result in a listener on 11222…

On a working node (listener was active on 11222), a restart of the infinispan service fails to result in a running hot rod listener…

If even just one of the nodes in the “existing/bad” cluster remains active, any new node starting will not start a listener on 11222…

Only when all nodes in the cluster were stopped and then all were restarted did all new instances start a listener on 11222…

Two logs attached, the second includes a shutdown sequence…

1. Re: Hot Rod not starting

wdfink Aug 5, 2014 3:24 PM (in response to fulton)

Where do you run Infinispan and how do you configure it?
An example and a bit information about the environment might be helpful
Actions
2. Re: Hot Rod not starting

fulton Aug 7, 2014 8:26 PM (in response to wdfink)
Here is info on the configuration and how testing was done.

The test environment is an Infinispan cluster on four virtual machines (RHEL 5.10, 2.6.18-371.9.1.el5).
The cluster is configured to communicate using TCP:
cluster="flexnet" stack="tcp" />
default-stack="${jboss.default.jgroups.stack:tcp}">

The machines are statically configured to know about each other:
<property name="initial_hosts">infinispan-test1a[7800],infinispan-test2a[7800],infinispan-test3a[7800],infinispan-test4a[7800],</property>

On each server VM, there is a simple client which uses the hotrod interface.
The client contains a loop that 10 times a second just does a put.
key = "12345";
value = "a value";
pprev = cache.put(key, value);
All four clients use the same code (same key, same value).

Each client is started with the IP address and TCP port of its local server and the cache name:
For example:
./test1 192.168.136.131 11222 cacheTest

Where cacheTest is defined as:
<distributed-cache name="cacheTest" mode="SYNC"
segments="20" owners="2" remote-timeout="30000" start="EAGER" />
This test configuration will run forever with no reported errors if left alone…

The following sequence of steps will eventually result in one or more nodes failing to start a hotrod listener on port 11222 after a restart:

[Server1]# service infinispan stop
Wait 30 seconds
[Server1]# service infinispan start
Wait 60 seconds while performing: netstat -nat | grep 11222
[Server2]# service infinispan stop
Wait 30 seconds
[Server2]# service infinispan start
Wait 60 seconds while performing: netstat -nat | grep 11222
[Server3]# service infinispan stop
Wait 30 seconds
[Server3]# service infinispan start
Wait 60 seconds while performing: netstat -nat | grep 11222
[Server4]# service infinispan stop
Wait 30 seconds
[Server4]# service infinispan start
Wait 60 seconds while performing: netstat -nat | grep 11222
Repeat the above until the failure occurs…this may take up to 30 minutes…

The pcap file has the capture starting where startProtocolServer(Configuration) is called in ProtocolServerService.java.

failure.pcap.zip 28.0 KB
Actions
3. Re: Hot Rod not starting

fulton Aug 13, 2014 10:06 PM (in response to wdfink)
We are running a four node cluster using hot rod. The issue that we are seeing happens when we restart a server and the server fails to start. The issue cascades if we restart another server in the same cluster. There seems to be issues with state transfer.

Below is the trace from when we see the node failing to start.

5:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.statetransfer.StateConsumerImpl.start() on component org.infinispan.statetransfer.StateConsumer
15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method private void org.infinispan.distribution.DistributionManagerImpl.start() throws java.lang.Exception on component org.infinispan.distribution.DistributionManager
15:50:32,365 TRACE [org.infinispan.distribution.DistributionManagerImpl] (MSC service thread 1-1) starting distribution manager on infinispan-test2/flexnet
15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.eviction.EvictionManagerImpl.start() on component org.infinispan.eviction.EvictionManager
15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.persistence.manager.PersistenceManagerImpl.preload() on component org.infinispan.persistence.manager.PersistenceManager
15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.statetransfer.StateProviderImpl.start() on component org.infinispan.statetransfer.StateProvider
15:50:32,365 TRACE [org.infinispan.factories.ComponentRegistry] (MSC service thread 1-1) Invoking start method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on component org.infinispan.statetransfer.StateTransferManager
15:50:32,365 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (MSC service thread 1-1) Starting StateTransferManager of cache cacheTest on node infinispan-test2/flexnet
15:50:32,367 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (MSC service thread 1-1) Node infinispan-test2/flexnet joining cache cacheTest
15:50:32,368 TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-1) dests=[infinispan-test3/flexnet], command=CacheTopologyControlCommand{cache=cacheTest, type=JOIN, sender=infinispan-test2/flexnet, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory@274744d4, hashFunction=MurmurHash3, numSegments=40, numOwners=2, timeout=60000, totalOrder=false, distributed=true}, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=49}, mode=SYNCHRONOUS, timeout=60000
15:50:32,368 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Replication task sending CacheTopologyControlCommand{cache=cacheTest, type=JOIN, sender=infinispan-test2/flexnet, joinInfo=CacheJoinInfo{consistentHashFactory=org.infinispan.distribution.ch.TopologyAwareConsistentHashFactory@274744d4, hashFunction=MurmurHash3, numSegments=40, numOwners=2, timeout=60000, totalOrder=false, distributed=true}, topologyId=0, currentCH=null, pendingCH=null, throwable=null, viewId=49} to single recipient infinispan-test3/flexnet with response mode GET_ALL
15:50:32,370 TRACE [org.infinispan.commons.marshall.MarshallableTypeHints] (MSC service thread 1-1) Cache a buffer size predictor for 'org.infinispan.topology.CacheTopologyControlCommand' assuming its serializability is unknown
15:50:32,370 TRACE [org.infinispan.commons.marshall.AdaptiveBufferSizePredictor] (MSC service thread 1-1) Next predicted buffer size for object type 'org.infinispan.topology.CacheTopologyControlCommand' will be 512
15:50:32,382 TRACE [org.infinispan.marshall.core.VersionAwareMarshaller] (MSC service thread 1-1) Wrote version 510
15:50:32,383 TRACE [org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller] (MSC service thread 1-1) Stop marshaller
15:50:32,385 TRACE [org.jgroups.blocks.Request] (MSC service thread 1-1) sending request (id=2)
15:50:32,386 TRACE [org.jgroups.protocols.RSVP] (MSC service thread 1-1) infinispan-test2/flexnet: REQ --> infinispan-test3/flexnet
15:50:32,386 TRACE [org.jgroups.protocols.UNICAST3] (MSC service thread 1-1) infinispan-test2/flexnet --> DATA(infinispan-test3/flexnet: #3, conn_id=0)
15:50:32,386 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) null: sending msg to infinispan-test3/flexnet, src=infinispan-test2/flexnet, headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]
15:50:32,387 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) dest=192.168.135.135:7800 (264 bytes)
15:50:32,396 TRACE [org.jgroups.protocols.TCP] (Incoming-2,shared=tcp) null: received [dst: infinispan-test2/flexnet, src: infinispan-test3/flexnet (3 headers), size=0 bytes, flags=DONT_BUNDLE|RSVP|INTERNAL], headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=56, TCP: [channel_name=clustered]
15:50:32,397 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-2,shared=tcp) infinispan-test2/flexnet <-- DATA(infinispan-test3/flexnet: #2, conn_id=56)
15:50:32,397 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-2,shared=tcp) infinispan-test2/flexnet: delivering #2 - #2 (1 messages)
15:50:32,423 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: received [dst: infinispan-test2/flexnet, src: infinispan-test3/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=3, TCP: [channel_name=clustered]
15:50:32,423 TRACE [org.jgroups.protocols.UNICAST3] (INT-1,shared=tcp) infinispan-test2/flexnet <-- ACK(infinispan-test3/flexnet: #3, conn-id=0)
15:50:32,526 TRACE [org.jgroups.protocols.UNICAST3] (Timer-2,shared=tcp) infinispan-test2/flexnet --> ACK(infinispan-test3/flexnet: #2)
15:50:32,527 TRACE [org.jgroups.protocols.TCP] (Timer-2,shared=tcp) null: sending msg to infinispan-test3/flexnet, src=infinispan-test2/flexnet, headers are UNICAST3: ACK, seqno=2, conn_id=56, TCP: [channel_name=clustered]
15:50:32,527 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (123 bytes (0.19% of max_bundle_size) to 1 destination(s)
15:50:32,527 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.135:7800 (126 bytes)
15:50:32,812 DEBUG [org.jgroups.protocols.FD] (Timer-5,shared=tcp) infinispan-test2/flexnet: sending are-you-alive msg to infinispan-test3/flexnet
15:50:32,812 TRACE [org.jgroups.protocols.TCP] (Timer-5,shared=tcp) null: sending msg to infinispan-test3/flexnet, src=infinispan-test2/flexnet, headers are FD: heartbeat, TCP: [channel_name=clustered]
15:50:32,812 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (122 bytes (0.19% of max_bundle_size) to 1 destination(s)
15:50:32,812 TRACE [org.jgroups.protocols.TCP] (TransferQueue

15:50:32,025 TRACE [org.jgroups.protocols.TCP] (INT-2,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=1, conn_id=56, TCP: [channel_name=clustered]
15:50:32,025 TRACE [org.jgroups.protocols.UNICAST3] (INT-2,shared=tcp) infinispan-test3/flexnet <-- ACK(infinispan-test2/flexnet: #1, conn-id=56)
15:50:32,169 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are FD_SOCK: GET_CACHE, TCP: [channel_name=clustered]
15:50:32,169 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: sending msg to infinispan-test2/flexnet, src=infinispan-test3/flexnet, headers are FD_SOCK: GET_CACHE_RSP, TCP: [channel_name=clustered]
15:50:32,169 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (353 bytes (0.55% of max_bundle_size) to 1 destination(s)
15:50:32,169 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.134:7800 (356 bytes)
15:50:32,171 TRACE [org.jgroups.protocols.FD_SOCK] (FD_SOCK acceptor,clustered,infinispan-test3/flexnet) accepted connection from /192.168.135.134:39759
15:50:32,171 TRACE [org.jgroups.protocols.FD_SOCK] (FD_SOCK acceptor,clustered,infinispan-test3/flexnet) waiting for client connections on bind-node/192.168.135.135:57600
15:50:32,295 DEBUG [org.jgroups.protocols.FD] (Timer-4,shared=tcp) infinispan-test3/flexnet: sending are-you-alive msg to infinispan-test4/flexnet
15:50:32,295 TRACE [org.jgroups.protocols.TCP] (Timer-4,shared=tcp) null: sending msg to infinispan-test4/flexnet, src=infinispan-test3/flexnet, headers are FD: heartbeat, TCP: [channel_name=clustered]
15:50:32,295 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (122 bytes (0.19% of max_bundle_size) to 1 destination(s)
15:50:32,295 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.136:7800 (125 bytes)
15:50:32,296 TRACE [org.jgroups.protocols.TCP] (INT-2,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test4/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are FD: heartbeat ack, TCP: [channel_name=clustered]
15:50:32,388 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (4 headers), size=116 bytes, flags=OOB|DONT_BUNDLE|NO_TOTAL_ORDER|RSVP], headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]
15:50:32,388 TRACE [org.jgroups.protocols.UNICAST3] (INT-1,shared=tcp) infinispan-test3/flexnet <-- DATA(infinispan-test2/flexnet: #3, conn_id=0)
15:50:32,388 TRACE [org.jgroups.protocols.UNICAST3] (INT-1,shared=tcp) infinispan-test3/flexnet: delivering infinispan-test2/flexnet#3
15:50:32,388 TRACE [org.jgroups.protocols.UFC] (INT-1,shared=tcp) infinispan-test2/flexnet used 116 credits, 1999884 remaining
15:50:32,388 TRACE [org.jgroups.protocols.RSVP] (INT-1,shared=tcp) infinispan-test3/flexnet: REQ <-- infinispan-test2/flexnet
15:50:32,388 TRACE [org.jgroups.protocols.RSVP] (INT-1,shared=tcp) infinispan-test3/flexnet: RSP --> infinispan-test2/flexnet
15:50:32,388 TRACE [org.jgroups.protocols.UNICAST3] (INT-1,shared=tcp) infinispan-test3/flexnet --> DATA(infinispan-test2/flexnet: #2, conn_id=56)
15:50:32,388 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: sending msg to infinispan-test2/flexnet, src=infinispan-test3/flexnet, headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=56, TCP: [channel_name=clustered]
15:50:32,388 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) dest=192.168.135.134:7800 (134 bytes)
15:50:32,389 TRACE [org.jgroups.blocks.RequestCorrelator] (INT-1,shared=tcp) calling (org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher) with request 2
15:50:32,423 TRACE [org.jgroups.protocols.UNICAST3] (Timer-3,shared=tcp) infinispan-test3/flexnet --> ACK(infinispan-test2/flexnet: #3)
15:50:32,423 TRACE [org.jgroups.protocols.TCP] (Timer-3,shared=tcp) null: sending msg to infinispan-test2/flexnet, src=infinispan-test3/flexnet, headers are UNICAST3: ACK, seqno=3, TCP: [channel_name=clustered]
15:50:32,423 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (123 bytes (0.19% of max_bundle_size) to 1 destination(s)
15:50:32,423 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.134:7800 (126 bytes)
15:50:32,528 TRACE [org.jgroups.protocols.TCP] (INT-2,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are UNICAST3: ACK, seqno=2, conn_id=56, TCP: [channel_name=clustered]
15:50:32,528 TRACE [org.jgroups.protocols.UNICAST3] (INT-2,shared=tcp) infinispan-test3/flexnet <-- ACK(infinispan-test2/flexnet: #2, conn-id=56)
15:50:32,813 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: received [dst: infinispan-test3/flexnet, src: infinispan-test2/flexnet (2 headers), size=0 bytes, flags=INTERNAL], headers are FD: heartbeat, TCP: [channel_name=clustered]
15:50:32,813 TRACE [org.jgroups.protocols.FD] (INT-1,shared=tcp) infinispan-test3/flexnet: received are-you-alive from infinispan-test2/flexnet, sending response
15:50:32,813 TRACE [org.jgroups.protocols.TCP] (INT-1,shared=tcp) null: sending msg to infinispan-test2/flexnet, src=infinispan-test3/flexnet, headers are FD: heartbeat ack, TCP: [channel_name=clustered]
15:50:32,813 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) null: sending 1 msgs (168 bytes (0.26% of max_bundle_size) to 1 destination(s)
15:50:32,813 TRACE [org.jgroups.protocols.TCP] (TransferQueueBundler,shared=tcp) dest=192.168.135.134:7800 (171 bytes)
15:50:33,296 DEBUG [org.jgroups.protocols.FD] (Timer-3,shared=tcp) infinispan-test3/flexnet:

clustered.xml 11.4 KB
Actions
4. Re: Hot Rod not starting

fulton Aug 14, 2014 6:32 PM (in response to fulton)

Starting Infinispan service fails to start HotRod listener…

Two traces, first one shows the failure… the second shows success…

infinispan-test1a 192.168.136.128
infinispan-test4a 192.168.136.131

Failure case:

08:44:37,191 TRACE [org.jgroups.blocks.Request] (MSC service thread 1-2) sending request (id=2)
08:44:37,192 TRACE [org.jgroups.protocols.RSVP] (MSC service thread 1-2) infinispan-test4a/flexnet: REQ --> infinispan-test1a/flexnet
08:44:37,193 TRACE [org.jgroups.protocols.UNICAST3] (MSC service thread 1-2) infinispan-test4a/flexnet --> DATA(infinispan-test1a/flexnet: #3, conn_id=0)
08:44:37,193 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-2) null: sending msg to infinispan-test1a/flexnet, src=infinispan-test4a/flexnet, headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]
08:44:37,193 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-2) dest=192.168.136.128:7800 (267 bytes)

08:44:37,195 TRACE [org.jgroups.protocols.TCP] (Incoming-4,shared=tcp) null: received [dst: infinispan-test4a/flexnet, src: infinispan-test1a/flexnet (3 headers), size=0 bytes, flags=DONT_BUNDLE|RSVP|INTERNAL], headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=25, TCP: [channel_name=clustered]
08:44:37,195 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet <-- DATA(infinispan-test1a/flexnet: #2, conn_id=25)
08:44:37,196 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet: delivering #2 - #2 (1 messages)

One minute later…

08:45:37,212 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC00001: Failed to start service jboss.endpoint.hotrod.hotrod-connector: org.jboss.msc.service.StartException in service jboss.endpoint.hotrod.hotrod-connector: JDGS010004: Failed to start HotRodServer
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.start(ProtocolServerService.java:113)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl

/usr/sbin/tcpdump -ieth1 -s512 -vv -nn -XX -w /tmp/tcpdump tcp port 7800

(tcp handshake packets removed for clarity)

08:44:37.193444 IP (tos 0x0, ttl 64, id 47273, offset 0, flags [DF], proto: TCP (6), length: 323) 192.168.136.131.52964 > 192.168.136.128.7800: P, cksum 0x938a ( (-> 0x50e9), 936:1207(271) ack 1422 win 88 <nop,nop,timestamp 1463150373 1463372874>
0x0000: 000c 29fc e9eb 000c 291e 7876 0800 4500 ..).....).xv..E.
0x0010: 0143 b8a9 4000 4006 eeb6 c0a8 8883 c0a8 .C..@.@.........
0x0020: 8880 cee4 1e78 bfeb 5ee0 1676 68d9 8018 .....x..^..vh...
0x0030: 0058 938a 0000 0101 080a 5735 e725 5739 .X........W5.%W9
0x0040: 4c4a 0000 010b 1901 0007 00a3 0000 5496 LJ............T.
0x0050: 0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3161 0000 5410 ispan-test1a..T.
0x0080: ab31 738c 8f58 f744 6752 7dd2 8cf4 c300 .1s..X.DgR}.....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3461 0000 0075 ispan-test4a...u
0x00b0: 0301 fe03 3e00 0011 093e 0963 6163 6865 ....>....>.cache
0x00c0: 5465 7374 4900 033c 0000 5410 ab31 738c TestI..<..T..1s.
0x00d0: 8f58 f744 6752 7dd2 8cf4 c300 0100 0000 .X.DgR}.........
0x00e0: 0552 6f77 2d31 0100 0000 1169 6e66 696e .Row-1.....infin
0x00f0: 6973 7061 6e2d 7465 7374 3461 0357 035e ispan-test4a.W.^
0x0100: 0349 0000 0028 0000 0002 0000 0000 0000 .I...(..........
0x0110: ea60 0001 3f80 0000 4b00 0000 0001 0101 .`..?...K.......
0x0120: 4b00 0000 1700 0400 c800 4700 0102 0100 K.........G.....
0x0130: c800 3700 5b01 0000 0040 006b 0001 0300 ..7.[....@.k....
0x0140: 0000 0016 0038 0009 636c 7573 7465 7265 .....8..clustere
0x0150: 64 d

08:44:37.194778 IP (tos 0x0, ttl 64, id 10641, offset 0, flags [DF], proto: TCP (6), length: 192) 192.168.136.128.7800 > 192.168.136.131.52964: P, cksum 0x64e9 (correct), 1422:1562(140) ack 1207 win 104 <nop,nop,timestamp 1463373385 1463150373>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 00c0 2991 4000 4006 7e52 c0a8 8880 c0a8 ..).@.@.~R......
0x0020: 8883 1e78 cee4 1676 68d9 bfeb 5fef 8018 ...x...vh..._...
0x0030: 0068 64e9 0000 0101 080a 5739 4e49 5735 .hd.......W9NIW5
0x0040: e725 0000 0088 1901 0003 0182 0000 5410 .%............T.
0x0050: ab31 738c 8f58 f744 6752 7dd2 8cf4 c300 .1s..X.DgR}.....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3461 0000 5496 ispan-test4a..T.
0x0080: 0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3161 0003 0037 ispan-test1a...7
0x00b0: 005b 0300 0000 4000 6b00 0102 0019 0000 .[....@.k.......
0x00c0: 1600 3800 0963 6c75 7374 6572 6564 ..8..clustered

08:44:37.407984 IP (tos 0x0, ttl 64, id 47275, offset 0, flags [DF], proto: TCP (6), length: 183) 192.168.136.131.52964 > 192.168.136.128.7800: P, cksum 0x92fe ( (-> 0xf1ec), 1207:1338(131) ack 1562 win 96 <nop,nop,timestamp 1463150588 1463373385>
0x0000: 000c 29fc e9eb 000c 291e 7876 0800 4500 ..).....).xv..E.
0x0010: 00b7 b8ab 4000 4006 ef40 c0a8 8883 c0a8 ....@.@..@......
0x0020: 8880 cee4 1e78 bfeb 5fef 1676 6965 8018 .....x.._..vie..
0x0030: 0060 92fe 0000 0101 080a 5735 e7fc 5739 .`........W5..W9
0x0040: 4e49 0000 007f 1901 0003 0100 0000 5496 NI............T.
0x0050: 0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3161 0000 5410 ispan-test1a..T.
0x0080: ab31 738c 8f58 f744 6752 7dd2 8cf4 c300 .1s..X.DgR}.....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3461 0002 0002 ispan-test4a....
0x00b0: 0003 00ff ff01 0016 0038 0009 636c 7573 .........8..clus
0x00c0: 7465 7265 64 tered

08:44:37.408843 IP (tos 0x0, ttl 64, id 10642, offset 0, flags [DF], proto: TCP (6), length: 230) 192.168.136.128.7800 > 192.168.136.131.52964: P, cksum 0xd1e3 (correct), 1562:1740(178) ack 1338 win 113 <nop,nop,timestamp 1463373598 1463150588>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 00e6 2992 4000 4006 7e2b c0a8 8880 c0a8 ..).@.@.~+......
0x0020: 8883 1e78 cee4 1676 6965 bfeb 6072 8018 ...x...vie..`r..
0x0030: 0071 d1e3 0000 0101 080a 5739 4f1e 5735 .q........W9O.W5
0x0040: e7fc 0000 00ae 1901 0003 0100 0000 5410 ..............T.
0x0050: ab31 738c 8f58 f744 6752 7dd2 8cf4 c300 .1s..X.DgR}.....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3461 0000 5496 ispan-test4a..T.
0x0080: 0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3161 0002 0002 ispan-test1a....
0x00b0: 0003 01ff ff00 0054 9608 6726 a4bb 5db1 .......T..g&..].
0x00c0: fbc0 b5f7 5f04 b885 0001 0005 526f 772d ...._.......Row-
0x00d0: 3101 0011 696e 6669 6e69 7370 616e 2d74 1...infinispan-t
0x00e0: 6573 7431 6100 1600 3800 0963 6c75 7374 est1a...8..clust
0x00f0: 6572 6564 ered

No other network traffic for this tcpdump filter…Timeout after 60 seconds… hotrod fails to start…

Good (working) case:

10:46:40,993 TRACE [org.jgroups.blocks.Request] (MSC service thread 1-1) sending request (id=2)
10:46:40,995 TRACE [org.jgroups.protocols.RSVP] (MSC service thread 1-1) infinispan-test4a/flexnet: REQ --> infinispan-test1a/flexnet
10:46:40,995 TRACE [org.jgroups.protocols.UNICAST3] (MSC service thread 1-1) infinispan-test4a/flexnet --> DATA(infinispan-test1a/flexnet: #3, conn_id=0)
10:46:40,995 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) null: sending msg to infinispan-test1a/flexnet, src=infinispan-test4a/flexnet, headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]
10:46:40,995 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) dest=192.168.136.128:7800 (267 bytes)

10:46:41,006 TRACE [org.jgroups.protocols.TCP] (Incoming-4,shared=tcp) null: received [dst: infinispan-test4a/flexnet, src: infinispan-test1a/flexnet (3 headers), size=0 bytes, flags=DONT_BUNDLE|RSVP|INTERNAL], headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=4, TCP: [channel_name=clustered]
10:46:41,006 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet <-- DATA(infinispan-test1a/flexnet: #2, conn_id=4)
10:46:41,006 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet: delivering #2 - #2 (1 messages)

Less than a half-second later…

10:46:41,499 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Response: SuccessfulResponse{responseValue=CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}}
10:46:41,499 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (MSC service thread 1-1) Updating local consistent hash(es) for cache cacheTest: new topology = CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}
10:46:41,500 TRACE [org.infinispan.topology.CacheTopology] (MSC service thread 1-1) Current consistent hash's routing table: 0: 0 2, 1: 0 2, 2: 0 2, 3: 0 2, 4: 0 2, 5: 0 2, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0 1, 13: 0 1, 14: 2 0, 15: 2 1, 16: 2 0, 17: 2 1, 18: 2 0, 19: 2 1, 20: 1 2, 21: 1 2, 22: 1 2, 23: 1 2, 24: 1 2, 25: 1 2, 26: 1 2, 27: 1 0, 28: 1 0, 29: 1 0, 30: 1 0, 31: 1 0, 32: 1 0, 33: 2 0, 34: 2 1, 35: 2 0, 36: 2 1, 37: 2 0, 38: 2 1, 39: 2 0
10:46:41,500 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (MSC service thread 1-1) Installing new cache topology CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null} on cache cacheTest
10:46:41,502 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) Received new topology for cache cacheTest, isRebalance = false, isMember = false, topology = CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}
10:46:41,502 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (MSC service thread 1-1) Signalling topology 4 is installed
10:46:41,502 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) On cache cacheTest we have: added segments: []
10:46:41,503 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) Topology update processed, rebalanceInProgress = false, isRebalance = false, pending CH = null
10:46:41,503 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (MSC service thread 1-1) Signalling transaction data received for topology 4
10:46:41,503 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (MSC service thread 1-1) Tasks executed=0, still pending=0
10:46:41,503 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Checking for transactions originated on leavers. Current members are [infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet], remote transactions: 0
10:46:41,504 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) No global transactions pertain to originator(s) who have left the cluster.
10:46:41,504 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Completed cleaning transactions originating on leavers. Remote transactions remaining: 0
10:46:41,504 DEBUG [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Topology changed, recalculating minTopologyId
10:46:41,505 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Changing minimum topology ID from -1 to 4
…

10:46:40.996039 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [DF], proto: TCP (6), length: 323) 192.168.136.131.51280 > 192.168.136.128.7800: P, cksum 0x938a ( (-> 0x0c63), 936:1207(271) ack 1422 win 88 <nop,nop,timestamp 1470474025 1470696498>
0x0000: 000c 29fc e9eb 000c 291e 7876 0800 4500 ..).....).xv..E.
0x0010: 0143 9392 4000 4006 13ce c0a8 8883 c0a8 .C..@.@.........
0x0020: 8880 c850 1e78 b330 359d 28f9 4677 8018 ...P.x.05.(.Fw..
0x0030: 0058 938a 0000 0101 080a 57a5 a729 57a9 .X........W..)W.
0x0040: 0c32 0000 010b 1901 0007 00a3 0000 545c .2............T\
0x0050: 0787 e601 6f91 fddc 03ab e137 15e0 a500 ....o......7....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3161 0000 5428 ispan-test1a..T(
0x0080: a557 dca7 9945 6712 a54e 33d2 9bd3 e700 .W...Eg..N3.....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3461 0000 0075 ispan-test4a...u
0x00b0: 0301 fe03 3e00 0011 093e 0963 6163 6865 ....>....>.cache
0x00c0: 5465 7374 4900 033c 0000 5428 a557 dca7 TestI..<..T(.W..
0x00d0: 9945 6712 a54e 33d2 9bd3 e700 0100 0000 .Eg..N3.........
0x00e0: 0552 6f77 2d31 0100 0000 1169 6e66 696e .Row-1.....infin
0x00f0: 6973 7061 6e2d 7465 7374 3461 0357 035e ispan-test4a.W.^
0x0100: 0349 0000 0028 0000 0002 0000 0000 0000 .I...(..........
0x0110: ea60 0001 3f80 0000 4b00 0000 0001 0101 .`..?...K.......
0x0120: 4b00 0000 0300 0400 c800 4700 0102 0100 K.........G.....
0x0130: c800 3700 5b01 0000 0040 006b 0001 0300 ..7.[....@.k....
0x0140: 0000 0016 0038 0009 636c 7573 7465 7265 .....8..clustere
0x0150: 64 d

10:46:40.997457 IP (tos 0x0, ttl 64, id 18822, offset 0, flags [DF], proto: TCP (6), length: 192) 192.168.136.128.7800 > 192.168.136.131.51280: P, cksum 0x104c (correct), 1422:1562(140) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 00c0 4986 4000 4006 5e5d c0a8 8880 c0a8 ..I.@.@.^]......
0x0020: 8883 1e78 c850 28f9 4677 b330 36ac 8018 ...x.P(.Fw.06...
0x0030: 0068 104c 0000 0101 080a 57a9 0e4c 57a5 .h.L......W..LW.
0x0040: a729 0000 0088 1901 0003 0182 0000 5428 .)............T(
0x0050: a557 dca7 9945 6712 a54e 33d2 9bd3 e700 .W...Eg..N3.....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3461 0000 545c ispan-test4a..T\
0x0080: 0787 e601 6f91 fddc 03ab e137 15e0 a500 ....o......7....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3161 0003 0037 ispan-test1a...7
0x00b0: 005b 0300 0000 4000 6b00 0102 0004 0000 .[....@.k.......
0x00c0: 1600 3800 0963 6c75 7374 6572 6564 ..8..clustered

10:46:40.997965 IP (tos 0x0, ttl 64, id 18823, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 1562:3010(1448) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 05dc 4987 4000 4006 5940 c0a8 8880 c0a8 ..I.@.@.Y@......
0x0020: 8883 1e78 c850 28f9 4703 b330 36ac 8010 ...x.P(.G..06...
0x0030: 0068 8eff 0000 0101 080a 57a9 0e4c 57a5 .h........W..LW.
0x0040: a729 0000 14cc 1901 0007 0023 0000 5428 .).........#..T(
0x0050: a557 dca7 9945 6712 a54e 33d2 9bd3 e700 .W...Eg..N3.....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3461 0000 545c ispan-test4a..T\
0x0080: 0787 e601 6f91 fddc 03ab e137 15e0 a500 ....o......7....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3161 0000 143d ispan-test1a...=
0x00b0: 0301 fe03 0501 034b 0000 0004 0333 0000 .......K.....3..
0x00c0: 0028 0000 0002 0300 0003 033c 0000 545c .(.........<..T\
0x00d0: 0787 e601 6f91 fddc 03ab e137 15e0 a500 ....o......7....
0x00e0: 0100 0000 0552 6f77 2d31 0100 0000 1169 .....Row-1.....i
0x00f0: 6e66 696e 6973 7061 6e2d 7465 7374 3161 nfinispan-test1a
0x0100: 033c 0000 54fe 1b3a e5c1 c349 e45f c0fe .<..T..:...I._..
0x0110: 3efb 74d8 8100 0100 0000 0552 6f77 2d31 >.t........Row-1
0x0120: 0100 0000 1169 6e66 696e 6973 7061 6e2d .....infinispan-
0x0130: 7465 7374 3261 033c 0000 54f9 56b4 d57b test2a.<..T.V..{
0x0140: cc23 ee16 266c 457e 12fd 8a00 0100 0000 .#..&lE~........
0x0150: 0552 6f77 2d31 0100 0000 1169 6e66 696e .Row-1.....infin
0x0160: 6973 7061 6e2d 7465 7374 3361 0302 0003 ispan-test3a....
0x0170: 033c 0000 54f9 56b4 d57b cc23 ee16 266c .<..T.V..{.#..&l
0x0180: 457e 12fd 8a00 0100 0000 0552 6f77 2d31 E~.........Row-1
0x0190: 0100 0000 1169 6e66 696e 6973 7061 6e2d .....infinispan-
0x01a0: 7465 7374 3361 4e3f 8000 0003 3c00 0054 test3aN?....<..T
0x01b0: 5c07 87e6 016f 91fd dc03 abe1 3715 e0a5 \....o......7...
0x01c0: 0001 0000 0005 526f 772d 3101 0000 0011 ......Row-1.....
0x01d0: 696e 6669 6e69 7370 616e 2d74 6573 7431 infinispan-test1
0x01e0: 614e 3f80 0000 033c 0000 54fe 1b3a e5c1 aN?....<..T..:..
0x01f0: c349 e45f c0fe 3efb 74d8 8100 0100 0000 .I._..>.t.......

10:46:40.997991 IP (tos 0x0, ttl 64, id 18824, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 3010:4458(1448) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 05dc 4988 4000 4006 593f c0a8 8880 c0a8 ..I.@.@.Y?......
0x0020: 8883 1e78 c850 28f9 4cab b330 36ac 8010 ...x.P(.L..06...
0x0030: 0068 5708 0000 0101 080a 57a9 0e4c 57a5 .hW.......W..LW.
0x0040: a729 7370 616e 2d74 6573 7431 6103 3c00 .)span-test1a.<.
0x0050: 0054 fe1b 3ae5 c1c3 49e4 5fc0 fe3e fb74 .T..:...I._..>.t
0x0060: d881 0001 0000 0005 526f 772d 3101 0000 ........Row-1...
0x0070: 0011 696e 6669 6e69 7370 616e 2d74 6573 ..infinispan-tes
0x0080: 7432 6135 043b ff32 0400 0000 0203 3c00 t2a5.;.2......<.
0x0090: 0054 5c07 87e6 016f 91fd dc03 abe1 3715 .T\....o......7.
0x00a0: e0a5 0001 0000 0005 526f 772d 3101 0000 ........Row-1...
0x00b0: 0011 696e 6669 6e69 7370 616e 2d74 6573 ..infinispan-tes
0x00c0: 7431 6103 3c00 0054 fe1b 3ae5 c1c3 49e4 t1a.<..T..:...I.
0x00d0: 5fc0 fe3e fb74 d881 0001 0000 0005 526f _..>.t........Ro
0x00e0: 772d 3101 0000 0011 696e 6669 6e69 7370 w-1.....infinisp
0x00f0: 616e 2d74 6573 7432 6135 043b ff32 0400 an-test2a5.;.2..
0x0100: 0000 0203 3c00 0054 5c07 87e6 016f 91fd ....<..T\....o..
0x0110: dc03 abe1 3715 e0a5 0001 0000 0005 526f ....7.........Ro
0x0120: 772d 3101 0000 0011 696e 6669 6e69 7370 w-1.....infinisp
0x0130: 616e 2d74 6573 7431 6103 3c00 0054 fe1b an-test1a.<..T..
0x0140: 3ae5 c1c3 49e4 5fc0 fe3e fb74 d881 0001 :...I._..>.t....
0x0150: 0000 0005 526f 772d 3101 0000 0011 696e ....Row-1.....in
0x0160: 6669 6e69 7370 616e 2d74 6573 7432 6135 finispan-test2a5
0x0170: 043b ff32 0400 0000 0203 3c00 0054 5c07 .;.2......<..T\.
0x0180: 87e6 016f 91fd dc03 abe1 3715 e0a5 0001 ...o......7.....
0x0190: 0000 0005 526f 772d 3101 0000 0011 696e ....Row-1.....in
0x01a0: 6669 6e69 7370 616e 2d74 6573 7431 6103 finispan-test1a.
0x01b0: 3c00 0054 fe1b 3ae5 c1c3 49e4 5fc0 fe3e <..T..:...I._..>
0x01c0: fb74 d881 0001 0000 0005 526f 772d 3101 .t........Row-1.
0x01d0: 0000 0011 696e 6669 6e69 7370 616e 2d74 ....infinispan-t
0x01e0: 6573 7432 6135 043b ff32 0400 0000 0203 est2a5.;.2......
0x01f0: 3c00 0054 5c07 87e6 016f 91fd dc03 abe1 <..T\....o......

10:46:40.999497 IP (tos 0x0, ttl 64, id 18825, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 4458:5906(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474026>

10:46:40.999520 IP (tos 0x0, ttl 64, id 18826, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 5906:7354(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474026>

10:46:40.999541 IP (tos 0x0, ttl 64, id 18827, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 7354:8802(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>

10:46:40.999556 IP (tos 0x0, ttl 64, id 18828, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 8802:10250(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>

10:46:40.999574 IP (tos 0x0, ttl 64, id 18829, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 10250:11698(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>

10:46:40.999590 IP (tos 0x0, ttl 64, id 18830, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: P 11698:13146(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>

10:46:41.002149 IP (tos 0x0, ttl 64, id 18831, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 13146:14594(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>

10:46:41.002174 IP (tos 0x0, ttl 64, id 18832, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 14594:16042(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>

10:46:41.002239 IP (tos 0x0, ttl 64, id 18833, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 16042:17490(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>

10:46:41.002263 IP (tos 0x0, ttl 64, id 18834, offset 0, flags [DF], proto: TCP (6), length: 57) 192.168.136.128.7800 > 192.168.136.131.51280: P, cksum 0xb581 (correct), 17490:17495(5) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 0039 4992 4000 4006 5ed8 c0a8 8880 c0a8 .9I.@.@.^.......
0x0020: 8883 1e78 c850 28f9 853b b330 36ac 8018 ...x.P(..;.06...
0x0030: 0068 b581 0000 0101 080a 57a9 0e51 57a5 .h........W..QW.
0x0040: a72c 7465 7265 64 .,tered
Actions

Go to original post