My servers don't seem to have problems to communicate with eachother. They form a partition, I can join an un-join nodes.
To be completely sure I ran the tests you reccommended. I also did the JGroupos multicast tests. All completed successfully.
This is the DEBUG log from my cluster services (on the machine that should serve the deployed application)
2007-03-27 09:20:01,866 DEBUG [org.jboss.ha.framework.server.FarmMemberService] farmDeployments request, parentDUMap.size=1
2007-03-27 09:20:01,903 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@78ee7d2a
2007-03-27 09:20:02,023 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@4c85ba73
2007-03-27 09:20:02,287 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:02,288 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:02,289 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:02,290 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:02,291 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:02,292 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:02,293 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:02,294 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:02,294 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
2007-03-27 09:20:02,331 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@58f8a759
2007-03-27 09:20:02,891 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:02,892 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:02,893 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:02,894 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:02,895 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:02,896 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:02,897 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:02,898 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:02,899 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
2007-03-27 09:20:02,931 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@2cb2f1b1
2007-03-27 09:20:03,283 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:03,284 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32806 (own address=192.168.1.105:32790)
2007-03-27 09:20:03,319 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@3f0ecf98
2007-03-27 09:20:03,356 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32806
2007-03-27 09:20:03,503 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@655f247f
2007-03-27 09:20:04,095 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:04,096 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:04,097 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:04,098 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:04,099 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:04,100 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:04,101 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:04,102 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:04,103 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
2007-03-27 09:20:04,139 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@5d8d15f0
2007-03-27 09:20:05,383 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:05,384 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32802 (own address=192.168.1.105:32786)
2007-03-27 09:20:05,419 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@599b6f8b
2007-03-27 09:20:05,456 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32802
2007-03-27 09:20:05,639 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@6bb83ca2
2007-03-27 09:20:06,499 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:06,500 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:06,501 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:06,502 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:06,503 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:06,504 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:06,505 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:06,505 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:06,507 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
2007-03-27 09:20:06,543 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@11afda9
2007-03-27 09:20:07,219 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:07,220 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32809 (own address=192.168.1.105:32793)
2007-03-27 09:20:07,255 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@130362d0
2007-03-27 09:20:07,292 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32809
2007-03-27 09:20:07,427 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@155054f0
2007-03-27 09:20:09,543 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:09,544 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32812 (own address=192.168.1.105:32796)
2007-03-27 09:20:09,546 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32812
2007-03-27 09:20:10,103 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:10,104 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:10,105 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:10,106 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:10,107 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:10,108 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:10,108 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:10,109 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:10,110 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
2007-03-27 09:20:10,144 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@4deb9df0
2007-03-27 09:20:11,243 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@27ce1f87
2007-03-27 09:20:12,248 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@4d77ec7b
2007-03-27 09:20:13,212 DEBUG [org.jgroups.protocols.MERGE2] initial_mbrs=[[own_addr=192.168.1.106:32809, coord_addr=192.168.1.105:32793, is_server=true], [own_addr=192.168.1.105:32793, coord_addr=192.168.1.105:32793, is_server=true]]
2007-03-27 09:20:13,284 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:13,284 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32806 (own address=192.168.1.105:32790)
2007-03-27 09:20:13,319 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@7c46a6f8
2007-03-27 09:20:13,357 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32806
2007-03-27 09:20:13,508 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@27c2386
2007-03-27 09:20:13,708 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:13,708 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:13,710 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:13,711 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:13,711 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:13,712 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:13,713 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:13,714 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:13,715 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
2007-03-27 09:20:13,751 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@6667be00
2007-03-27 09:20:15,388 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:15,388 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32802 (own address=192.168.1.105:32786)
2007-03-27 09:20:15,424 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@4a2e3a59
2007-03-27 09:20:15,461 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32802
2007-03-27 09:20:15,644 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@20f8cf1b
2007-03-27 09:20:17,224 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:17,225 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32809 (own address=192.168.1.105:32793)
2007-03-27 09:20:17,260 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@2ee50686
2007-03-27 09:20:17,297 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32809
2007-03-27 09:20:17,312 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:17,312 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:17,313 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:17,314 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:17,315 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:17,315 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:17,316 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:17,317 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:17,318 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
2007-03-27 09:20:17,352 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@7c217540
2007-03-27 09:20:17,432 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@3e3d101
2007-03-27 09:20:19,548 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:19,548 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32812 (own address=192.168.1.105:32796)
2007-03-27 09:20:19,550 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32812
2007-03-27 09:20:20,916 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:20,917 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:20,919 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:20,920 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:20,921 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:20,922 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:20,922 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:20,923 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:20,924 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
2007-03-27 09:20:20,960 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@52aa1162
2007-03-27 09:20:23,144 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@4ff681ad
2007-03-27 09:20:23,288 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:23,289 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.106:32806 (own address=192.168.1.105:32790)
2007-03-27 09:20:23,324 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@24fcfde
2007-03-27 09:20:23,361 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.106:32806
2007-03-27 09:20:23,512 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@45d7f901
2007-03-27 09:20:24,116 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.pbcast.STABLE$StableTask@79bcfbeb
2007-03-27 09:20:24,148 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@661cd479
2007-03-27 09:20:24,308 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.pbcast.STABLE$StabilitySendTask@6ec1884e
2007-03-27 09:20:24,520 DEBUG [org.jgroups.util.TimeScheduler] Running task 7-7
2007-03-27 09:20:24,521 DEBUG [org.jgroups.util.TimeScheduler] Running task 8-8
2007-03-27 09:20:24,522 DEBUG [org.jgroups.util.TimeScheduler] Running task 10-10
2007-03-27 09:20:24,523 DEBUG [org.jgroups.util.TimeScheduler] Running task 11-11
2007-03-27 09:20:24,524 DEBUG [org.jgroups.util.TimeScheduler] Running task 9-9
2007-03-27 09:20:24,524 DEBUG [org.jgroups.util.TimeScheduler] Running task 12-12
2007-03-27 09:20:24,525 DEBUG [org.jgroups.util.TimeScheduler] Running task 14-14
2007-03-27 09:20:24,526 DEBUG [org.jgroups.util.TimeScheduler] Running task 13-13
2007-03-27 09:20:24,526 DEBUG [org.jgroups.util.TimeScheduler] Running task 15-15
And the log on the requestion machine:
2007-03-27 09:20:01,917 DEBUG [org.jboss.ha.framework.server.FarmMemberService] Found 1 farmDeployments responses
2007-03-27 09:20:01,917 INFO [org.jboss.ha.framework.server.FarmMemberService] **** pullNewDeployments ****
2007-03-27 09:20:01,918 INFO [org.jboss.ha.framework.server.ClusterFileTransfer] Start pull of file kusssdemo.ear from cluster.
2007-03-27 09:20:01,946 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@56b3951d
2007-03-27 09:20:03,354 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@2802cf63
2007-03-27 09:20:03,430 DEBUG [org.jgroups.util.TimeScheduler] Running task true
2007-03-27 09:20:03,431 DEBUG [org.jgroups.protocols.FD] sending are-you-alive msg to 192.168.1.105:32790 (own address=192.168.1.106:32806)
2007-03-27 09:20:03,467 DEBUG [org.jgroups.util.TimeScheduler] Running task org.jgroups.protocols.TP$Bundler$BundlingTimer@507d811a
2007-03-27 09:20:03,505 DEBUG [org.jgroups.protocols.FD] received ack from 192.168.1.105:32790
Fromt his time on, on the sending machine the
Running task
(whatever that is) entries go on and on. The receiving machine has already cancelled the transfer (exactly after 1 minute):
2007-03-27 09:21:01,931 ERROR [org.jboss.ha.framework.server.FarmMemberService] org.jboss.ha.framework.server.ClusterFileTransfer$ClusterFileTransferException: Did not receive response from remote machine trying to open file 'farm/kusssdemo.ear'. Check remote machine error log.