Starting Infinispan service fails to start HotRod listener…
Two traces, first one shows the failure… the second shows success…
infinispan-test1a 192.168.136.128
infinispan-test4a 192.168.136.131
Failure case:
08:44:37,191 TRACE [org.jgroups.blocks.Request] (MSC service thread 1-2) sending request (id=2)
08:44:37,192 TRACE [org.jgroups.protocols.RSVP] (MSC service thread 1-2) infinispan-test4a/flexnet: REQ --> infinispan-test1a/flexnet
08:44:37,193 TRACE [org.jgroups.protocols.UNICAST3] (MSC service thread 1-2) infinispan-test4a/flexnet --> DATA(infinispan-test1a/flexnet: #3, conn_id=0)
08:44:37,193 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-2) null: sending msg to infinispan-test1a/flexnet, src=infinispan-test4a/flexnet, headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]
08:44:37,193 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-2) dest=192.168.136.128:7800 (267 bytes)
08:44:37,195 TRACE [org.jgroups.protocols.TCP] (Incoming-4,shared=tcp) null: received [dst: infinispan-test4a/flexnet, src: infinispan-test1a/flexnet (3 headers), size=0 bytes, flags=DONT_BUNDLE|RSVP|INTERNAL], headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=25, TCP: [channel_name=clustered]
08:44:37,195 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet <-- DATA(infinispan-test1a/flexnet: #2, conn_id=25)
08:44:37,196 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet: delivering #2 - #2 (1 messages)
One minute later…
08:45:37,212 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC00001: Failed to start service jboss.endpoint.hotrod.hotrod-connector: org.jboss.msc.service.StartException in service jboss.endpoint.hotrod.hotrod-connector: JDGS010004: Failed to start HotRodServer
at org.infinispan.server.endpoint.subsystem.ProtocolServerService.start(ProtocolServerService.java:113)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51]
at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51]
Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.start() throws java.lang.Exception on object of type StateTransferManagerImpl
/usr/sbin/tcpdump -ieth1 -s512 -vv -nn -XX -w /tmp/tcpdump tcp port 7800
(tcp handshake packets removed for clarity)
08:44:37.193444 IP (tos 0x0, ttl 64, id 47273, offset 0, flags [DF], proto: TCP (6), length: 323) 192.168.136.131.52964 > 192.168.136.128.7800: P, cksum 0x938a ( (-> 0x50e9), 936:1207(271) ack 1422 win 88 <nop,nop,timestamp 1463150373 1463372874>
0x0000: 000c 29fc e9eb 000c 291e 7876 0800 4500 ..).....).xv..E.
0x0010: 0143 b8a9 4000 4006 eeb6 c0a8 8883 c0a8 .C..@.@.........
0x0020: 8880 cee4 1e78 bfeb 5ee0 1676 68d9 8018 .....x..^..vh...
0x0030: 0058 938a 0000 0101 080a 5735 e725 5739 .X........W5.%W9
0x0040: 4c4a 0000 010b 1901 0007 00a3 0000 5496 LJ............T.
0x0050: 0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3161 0000 5410 ispan-test1a..T.
0x0080: ab31 738c 8f58 f744 6752 7dd2 8cf4 c300 .1s..X.DgR}.....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3461 0000 0075 ispan-test4a...u
0x00b0: 0301 fe03 3e00 0011 093e 0963 6163 6865 ....>....>.cache
0x00c0: 5465 7374 4900 033c 0000 5410 ab31 738c TestI..<..T..1s.
0x00d0: 8f58 f744 6752 7dd2 8cf4 c300 0100 0000 .X.DgR}.........
0x00e0: 0552 6f77 2d31 0100 0000 1169 6e66 696e .Row-1.....infin
0x00f0: 6973 7061 6e2d 7465 7374 3461 0357 035e ispan-test4a.W.^
0x0100: 0349 0000 0028 0000 0002 0000 0000 0000 .I...(..........
0x0110: ea60 0001 3f80 0000 4b00 0000 0001 0101 .`..?...K.......
0x0120: 4b00 0000 1700 0400 c800 4700 0102 0100 K.........G.....
0x0130: c800 3700 5b01 0000 0040 006b 0001 0300 ..7.[....@.k....
0x0140: 0000 0016 0038 0009 636c 7573 7465 7265 .....8..clustere
0x0150: 64 d
08:44:37.194778 IP (tos 0x0, ttl 64, id 10641, offset 0, flags [DF], proto: TCP (6), length: 192) 192.168.136.128.7800 > 192.168.136.131.52964: P, cksum 0x64e9 (correct), 1422:1562(140) ack 1207 win 104 <nop,nop,timestamp 1463373385 1463150373>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 00c0 2991 4000 4006 7e52 c0a8 8880 c0a8 ..).@.@.~R......
0x0020: 8883 1e78 cee4 1676 68d9 bfeb 5fef 8018 ...x...vh..._...
0x0030: 0068 64e9 0000 0101 080a 5739 4e49 5735 .hd.......W9NIW5
0x0040: e725 0000 0088 1901 0003 0182 0000 5410 .%............T.
0x0050: ab31 738c 8f58 f744 6752 7dd2 8cf4 c300 .1s..X.DgR}.....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3461 0000 5496 ispan-test4a..T.
0x0080: 0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3161 0003 0037 ispan-test1a...7
0x00b0: 005b 0300 0000 4000 6b00 0102 0019 0000 .[....@.k.......
0x00c0: 1600 3800 0963 6c75 7374 6572 6564 ..8..clustered
08:44:37.407984 IP (tos 0x0, ttl 64, id 47275, offset 0, flags [DF], proto: TCP (6), length: 183) 192.168.136.131.52964 > 192.168.136.128.7800: P, cksum 0x92fe ( (-> 0xf1ec), 1207:1338(131) ack 1562 win 96 <nop,nop,timestamp 1463150588 1463373385>
0x0000: 000c 29fc e9eb 000c 291e 7876 0800 4500 ..).....).xv..E.
0x0010: 00b7 b8ab 4000 4006 ef40 c0a8 8883 c0a8 ....@.@..@......
0x0020: 8880 cee4 1e78 bfeb 5fef 1676 6965 8018 .....x.._..vie..
0x0030: 0060 92fe 0000 0101 080a 5735 e7fc 5739 .`........W5..W9
0x0040: 4e49 0000 007f 1901 0003 0100 0000 5496 NI............T.
0x0050: 0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3161 0000 5410 ispan-test1a..T.
0x0080: ab31 738c 8f58 f744 6752 7dd2 8cf4 c300 .1s..X.DgR}.....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3461 0002 0002 ispan-test4a....
0x00b0: 0003 00ff ff01 0016 0038 0009 636c 7573 .........8..clus
0x00c0: 7465 7265 64 tered
08:44:37.408843 IP (tos 0x0, ttl 64, id 10642, offset 0, flags [DF], proto: TCP (6), length: 230) 192.168.136.128.7800 > 192.168.136.131.52964: P, cksum 0xd1e3 (correct), 1562:1740(178) ack 1338 win 113 <nop,nop,timestamp 1463373598 1463150588>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 00e6 2992 4000 4006 7e2b c0a8 8880 c0a8 ..).@.@.~+......
0x0020: 8883 1e78 cee4 1676 6965 bfeb 6072 8018 ...x...vie..`r..
0x0030: 0071 d1e3 0000 0101 080a 5739 4f1e 5735 .q........W9O.W5
0x0040: e7fc 0000 00ae 1901 0003 0100 0000 5410 ..............T.
0x0050: ab31 738c 8f58 f744 6752 7dd2 8cf4 c300 .1s..X.DgR}.....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3461 0000 5496 ispan-test4a..T.
0x0080: 0867 26a4 bb5d b1fb c0b5 f75f 04b8 8500 .g&..]....._....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3161 0002 0002 ispan-test1a....
0x00b0: 0003 01ff ff00 0054 9608 6726 a4bb 5db1 .......T..g&..].
0x00c0: fbc0 b5f7 5f04 b885 0001 0005 526f 772d ...._.......Row-
0x00d0: 3101 0011 696e 6669 6e69 7370 616e 2d74 1...infinispan-t
0x00e0: 6573 7431 6100 1600 3800 0963 6c75 7374 est1a...8..clust
0x00f0: 6572 6564 ered
No other network traffic for this tcpdump filter…Timeout after 60 seconds… hotrod fails to start…
Good (working) case:
10:46:40,993 TRACE [org.jgroups.blocks.Request] (MSC service thread 1-1) sending request (id=2)
10:46:40,995 TRACE [org.jgroups.protocols.RSVP] (MSC service thread 1-1) infinispan-test4a/flexnet: REQ --> infinispan-test1a/flexnet
10:46:40,995 TRACE [org.jgroups.protocols.UNICAST3] (MSC service thread 1-1) infinispan-test4a/flexnet --> DATA(infinispan-test1a/flexnet: #3, conn_id=0)
10:46:40,995 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) null: sending msg to infinispan-test1a/flexnet, src=infinispan-test4a/flexnet, headers are RequestCorrelator: id=200, type=REQ, id=2, rsp_expected=true, RSVP: REQ(0), UNICAST3: DATA, seqno=3, TCP: [channel_name=clustered]
10:46:40,995 TRACE [org.jgroups.protocols.TCP] (MSC service thread 1-1) dest=192.168.136.128:7800 (267 bytes)
10:46:41,006 TRACE [org.jgroups.protocols.TCP] (Incoming-4,shared=tcp) null: received [dst: infinispan-test4a/flexnet, src: infinispan-test1a/flexnet (3 headers), size=0 bytes, flags=DONT_BUNDLE|RSVP|INTERNAL], headers are RSVP: RSP(0), UNICAST3: DATA, seqno=2, conn_id=4, TCP: [channel_name=clustered]
10:46:41,006 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet <-- DATA(infinispan-test1a/flexnet: #2, conn_id=4)
10:46:41,006 TRACE [org.jgroups.protocols.UNICAST3] (Incoming-4,shared=tcp) infinispan-test4a/flexnet: delivering #2 - #2 (1 messages)
Less than a half-second later…
10:46:41,499 TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (MSC service thread 1-1) Response: SuccessfulResponse{responseValue=CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}}
10:46:41,499 DEBUG [org.infinispan.topology.LocalTopologyManagerImpl] (MSC service thread 1-1) Updating local consistent hash(es) for cache cacheTest: new topology = CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}
10:46:41,500 TRACE [org.infinispan.topology.CacheTopology] (MSC service thread 1-1) Current consistent hash's routing table: 0: 0 2, 1: 0 2, 2: 0 2, 3: 0 2, 4: 0 2, 5: 0 2, 6: 0 1, 7: 0 1, 8: 0 1, 9: 0 1, 10: 0 1, 11: 0 1, 12: 0 1, 13: 0 1, 14: 2 0, 15: 2 1, 16: 2 0, 17: 2 1, 18: 2 0, 19: 2 1, 20: 1 2, 21: 1 2, 22: 1 2, 23: 1 2, 24: 1 2, 25: 1 2, 26: 1 2, 27: 1 0, 28: 1 0, 29: 1 0, 30: 1 0, 31: 1 0, 32: 1 0, 33: 2 0, 34: 2 1, 35: 2 0, 36: 2 1, 37: 2 0, 38: 2 1, 39: 2 0
10:46:41,500 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (MSC service thread 1-1) Installing new cache topology CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null} on cache cacheTest
10:46:41,502 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) Received new topology for cache cacheTest, isRebalance = false, isMember = false, topology = CacheTopology{id=4, currentCH=DefaultConsistentHash{numSegments=40, numOwners=2, members=[infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet]}, pendingCH=null}
10:46:41,502 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (MSC service thread 1-1) Signalling topology 4 is installed
10:46:41,502 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) On cache cacheTest we have: added segments: []
10:46:41,503 TRACE [org.infinispan.statetransfer.StateConsumerImpl] (MSC service thread 1-1) Topology update processed, rebalanceInProgress = false, isRebalance = false, pending CH = null
10:46:41,503 TRACE [org.infinispan.statetransfer.StateTransferLockImpl] (MSC service thread 1-1) Signalling transaction data received for topology 4
10:46:41,503 TRACE [org.infinispan.util.concurrent.BlockingTaskAwareExecutorServiceImpl] (MSC service thread 1-1) Tasks executed=0, still pending=0
10:46:41,503 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Checking for transactions originated on leavers. Current members are [infinispan-test1a/flexnet, infinispan-test2a/flexnet, infinispan-test3a/flexnet], remote transactions: 0
10:46:41,504 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) No global transactions pertain to originator(s) who have left the cluster.
10:46:41,504 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Completed cleaning transactions originating on leavers. Remote transactions remaining: 0
10:46:41,504 DEBUG [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Topology changed, recalculating minTopologyId
10:46:41,505 TRACE [org.infinispan.transaction.TransactionTable] (MSC service thread 1-1) Changing minimum topology ID from -1 to 4
…
10:46:40.996039 IP (tos 0x0, ttl 64, id 37778, offset 0, flags [DF], proto: TCP (6), length: 323) 192.168.136.131.51280 > 192.168.136.128.7800: P, cksum 0x938a ( (-> 0x0c63), 936:1207(271) ack 1422 win 88 <nop,nop,timestamp 1470474025 1470696498>
0x0000: 000c 29fc e9eb 000c 291e 7876 0800 4500 ..).....).xv..E.
0x0010: 0143 9392 4000 4006 13ce c0a8 8883 c0a8 .C..@.@.........
0x0020: 8880 c850 1e78 b330 359d 28f9 4677 8018 ...P.x.05.(.Fw..
0x0030: 0058 938a 0000 0101 080a 57a5 a729 57a9 .X........W..)W.
0x0040: 0c32 0000 010b 1901 0007 00a3 0000 545c .2............T\
0x0050: 0787 e601 6f91 fddc 03ab e137 15e0 a500 ....o......7....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3161 0000 5428 ispan-test1a..T(
0x0080: a557 dca7 9945 6712 a54e 33d2 9bd3 e700 .W...Eg..N3.....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3461 0000 0075 ispan-test4a...u
0x00b0: 0301 fe03 3e00 0011 093e 0963 6163 6865 ....>....>.cache
0x00c0: 5465 7374 4900 033c 0000 5428 a557 dca7 TestI..<..T(.W..
0x00d0: 9945 6712 a54e 33d2 9bd3 e700 0100 0000 .Eg..N3.........
0x00e0: 0552 6f77 2d31 0100 0000 1169 6e66 696e .Row-1.....infin
0x00f0: 6973 7061 6e2d 7465 7374 3461 0357 035e ispan-test4a.W.^
0x0100: 0349 0000 0028 0000 0002 0000 0000 0000 .I...(..........
0x0110: ea60 0001 3f80 0000 4b00 0000 0001 0101 .`..?...K.......
0x0120: 4b00 0000 0300 0400 c800 4700 0102 0100 K.........G.....
0x0130: c800 3700 5b01 0000 0040 006b 0001 0300 ..7.[....@.k....
0x0140: 0000 0016 0038 0009 636c 7573 7465 7265 .....8..clustere
0x0150: 64 d
10:46:40.997457 IP (tos 0x0, ttl 64, id 18822, offset 0, flags [DF], proto: TCP (6), length: 192) 192.168.136.128.7800 > 192.168.136.131.51280: P, cksum 0x104c (correct), 1422:1562(140) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 00c0 4986 4000 4006 5e5d c0a8 8880 c0a8 ..I.@.@.^]......
0x0020: 8883 1e78 c850 28f9 4677 b330 36ac 8018 ...x.P(.Fw.06...
0x0030: 0068 104c 0000 0101 080a 57a9 0e4c 57a5 .h.L......W..LW.
0x0040: a729 0000 0088 1901 0003 0182 0000 5428 .)............T(
0x0050: a557 dca7 9945 6712 a54e 33d2 9bd3 e700 .W...Eg..N3.....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3461 0000 545c ispan-test4a..T\
0x0080: 0787 e601 6f91 fddc 03ab e137 15e0 a500 ....o......7....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3161 0003 0037 ispan-test1a...7
0x00b0: 005b 0300 0000 4000 6b00 0102 0004 0000 .[....@.k.......
0x00c0: 1600 3800 0963 6c75 7374 6572 6564 ..8..clustered
10:46:40.997965 IP (tos 0x0, ttl 64, id 18823, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 1562:3010(1448) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 05dc 4987 4000 4006 5940 c0a8 8880 c0a8 ..I.@.@.Y@......
0x0020: 8883 1e78 c850 28f9 4703 b330 36ac 8010 ...x.P(.G..06...
0x0030: 0068 8eff 0000 0101 080a 57a9 0e4c 57a5 .h........W..LW.
0x0040: a729 0000 14cc 1901 0007 0023 0000 5428 .).........#..T(
0x0050: a557 dca7 9945 6712 a54e 33d2 9bd3 e700 .W...Eg..N3.....
0x0060: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x0070: 6973 7061 6e2d 7465 7374 3461 0000 545c ispan-test4a..T\
0x0080: 0787 e601 6f91 fddc 03ab e137 15e0 a500 ....o......7....
0x0090: 0100 0552 6f77 2d31 0100 1169 6e66 696e ...Row-1...infin
0x00a0: 6973 7061 6e2d 7465 7374 3161 0000 143d ispan-test1a...=
0x00b0: 0301 fe03 0501 034b 0000 0004 0333 0000 .......K.....3..
0x00c0: 0028 0000 0002 0300 0003 033c 0000 545c .(.........<..T\
0x00d0: 0787 e601 6f91 fddc 03ab e137 15e0 a500 ....o......7....
0x00e0: 0100 0000 0552 6f77 2d31 0100 0000 1169 .....Row-1.....i
0x00f0: 6e66 696e 6973 7061 6e2d 7465 7374 3161 nfinispan-test1a
0x0100: 033c 0000 54fe 1b3a e5c1 c349 e45f c0fe .<..T..:...I._..
0x0110: 3efb 74d8 8100 0100 0000 0552 6f77 2d31 >.t........Row-1
0x0120: 0100 0000 1169 6e66 696e 6973 7061 6e2d .....infinispan-
0x0130: 7465 7374 3261 033c 0000 54f9 56b4 d57b test2a.<..T.V..{
0x0140: cc23 ee16 266c 457e 12fd 8a00 0100 0000 .#..&lE~........
0x0150: 0552 6f77 2d31 0100 0000 1169 6e66 696e .Row-1.....infin
0x0160: 6973 7061 6e2d 7465 7374 3361 0302 0003 ispan-test3a....
0x0170: 033c 0000 54f9 56b4 d57b cc23 ee16 266c .<..T.V..{.#..&l
0x0180: 457e 12fd 8a00 0100 0000 0552 6f77 2d31 E~.........Row-1
0x0190: 0100 0000 1169 6e66 696e 6973 7061 6e2d .....infinispan-
0x01a0: 7465 7374 3361 4e3f 8000 0003 3c00 0054 test3aN?....<..T
0x01b0: 5c07 87e6 016f 91fd dc03 abe1 3715 e0a5 \....o......7...
0x01c0: 0001 0000 0005 526f 772d 3101 0000 0011 ......Row-1.....
0x01d0: 696e 6669 6e69 7370 616e 2d74 6573 7431 infinispan-test1
0x01e0: 614e 3f80 0000 033c 0000 54fe 1b3a e5c1 aN?....<..T..:..
0x01f0: c349 e45f c0fe 3efb 74d8 8100 0100 0000 .I._..>.t.......
10:46:40.997991 IP (tos 0x0, ttl 64, id 18824, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 3010:4458(1448) ack 1207 win 104 <nop,nop,timestamp 1470697036 1470474025>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 05dc 4988 4000 4006 593f c0a8 8880 c0a8 ..I.@.@.Y?......
0x0020: 8883 1e78 c850 28f9 4cab b330 36ac 8010 ...x.P(.L..06...
0x0030: 0068 5708 0000 0101 080a 57a9 0e4c 57a5 .hW.......W..LW.
0x0040: a729 7370 616e 2d74 6573 7431 6103 3c00 .)span-test1a.<.
0x0050: 0054 fe1b 3ae5 c1c3 49e4 5fc0 fe3e fb74 .T..:...I._..>.t
0x0060: d881 0001 0000 0005 526f 772d 3101 0000 ........Row-1...
0x0070: 0011 696e 6669 6e69 7370 616e 2d74 6573 ..infinispan-tes
0x0080: 7432 6135 043b ff32 0400 0000 0203 3c00 t2a5.;.2......<.
0x0090: 0054 5c07 87e6 016f 91fd dc03 abe1 3715 .T\....o......7.
0x00a0: e0a5 0001 0000 0005 526f 772d 3101 0000 ........Row-1...
0x00b0: 0011 696e 6669 6e69 7370 616e 2d74 6573 ..infinispan-tes
0x00c0: 7431 6103 3c00 0054 fe1b 3ae5 c1c3 49e4 t1a.<..T..:...I.
0x00d0: 5fc0 fe3e fb74 d881 0001 0000 0005 526f _..>.t........Ro
0x00e0: 772d 3101 0000 0011 696e 6669 6e69 7370 w-1.....infinisp
0x00f0: 616e 2d74 6573 7432 6135 043b ff32 0400 an-test2a5.;.2..
0x0100: 0000 0203 3c00 0054 5c07 87e6 016f 91fd ....<..T\....o..
0x0110: dc03 abe1 3715 e0a5 0001 0000 0005 526f ....7.........Ro
0x0120: 772d 3101 0000 0011 696e 6669 6e69 7370 w-1.....infinisp
0x0130: 616e 2d74 6573 7431 6103 3c00 0054 fe1b an-test1a.<..T..
0x0140: 3ae5 c1c3 49e4 5fc0 fe3e fb74 d881 0001 :...I._..>.t....
0x0150: 0000 0005 526f 772d 3101 0000 0011 696e ....Row-1.....in
0x0160: 6669 6e69 7370 616e 2d74 6573 7432 6135 finispan-test2a5
0x0170: 043b ff32 0400 0000 0203 3c00 0054 5c07 .;.2......<..T\.
0x0180: 87e6 016f 91fd dc03 abe1 3715 e0a5 0001 ...o......7.....
0x0190: 0000 0005 526f 772d 3101 0000 0011 696e ....Row-1.....in
0x01a0: 6669 6e69 7370 616e 2d74 6573 7431 6103 finispan-test1a.
0x01b0: 3c00 0054 fe1b 3ae5 c1c3 49e4 5fc0 fe3e <..T..:...I._..>
0x01c0: fb74 d881 0001 0000 0005 526f 772d 3101 .t........Row-1.
0x01d0: 0000 0011 696e 6669 6e69 7370 616e 2d74 ....infinispan-t
0x01e0: 6573 7432 6135 043b ff32 0400 0000 0203 est2a5.;.2......
0x01f0: 3c00 0054 5c07 87e6 016f 91fd dc03 abe1 <..T\....o......
10:46:40.999497 IP (tos 0x0, ttl 64, id 18825, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 4458:5906(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474026>
10:46:40.999520 IP (tos 0x0, ttl 64, id 18826, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 5906:7354(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474026>
10:46:40.999541 IP (tos 0x0, ttl 64, id 18827, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 7354:8802(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>
10:46:40.999556 IP (tos 0x0, ttl 64, id 18828, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 8802:10250(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>
10:46:40.999574 IP (tos 0x0, ttl 64, id 18829, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 10250:11698(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>
10:46:40.999590 IP (tos 0x0, ttl 64, id 18830, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: P 11698:13146(1448) ack 1207 win 104 <nop,nop,timestamp 1470697038 1470474027>
10:46:41.002149 IP (tos 0x0, ttl 64, id 18831, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 13146:14594(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>
10:46:41.002174 IP (tos 0x0, ttl 64, id 18832, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 14594:16042(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>
10:46:41.002239 IP (tos 0x0, ttl 64, id 18833, offset 0, flags [DF], proto: TCP (6), length: 1500) 192.168.136.128.7800 > 192.168.136.131.51280: . 16042:17490(1448) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>
10:46:41.002263 IP (tos 0x0, ttl 64, id 18834, offset 0, flags [DF], proto: TCP (6), length: 57) 192.168.136.128.7800 > 192.168.136.131.51280: P, cksum 0xb581 (correct), 17490:17495(5) ack 1207 win 104 <nop,nop,timestamp 1470697041 1470474028>
0x0000: 000c 291e 7876 000c 29fc e9eb 0800 4500 ..).xv..).....E.
0x0010: 0039 4992 4000 4006 5ed8 c0a8 8880 c0a8 .9I.@.@.^.......
0x0020: 8883 1e78 c850 28f9 853b b330 36ac 8018 ...x.P(..;.06...
0x0030: 0068 b581 0000 0101 080a 57a9 0e51 57a5 .h........W..QW.
0x0040: a72c 7465 7265 64 .,tered