Problem with RPCManager members synchronization
defunkt May 24, 2013 4:11 PMHello,
I'm using the distributed execution framework to distribute tasks across a cluster. Sometimes there occurs a situtation where I send a task to a worker which was 'just' registered to my list of workers by a view changed event. In such an occurance, I usually get an error that the node I tried sending the task to is not yet available.
Looking at the code of DefaultExecutorService, I see that its getMembers() method, which is used to find out which members are available to send tasks to, uses RPCManager.getMembers().
However, it seems that RPCManager.getTransport().getMembers() will display the updated view some time before RPCManager.getMembers() will. This is the result I get from a test I wrote to check this issue:
2013-05-24 23:05:10,621 INFO [JGroupsTransport] (main) ISPN000078: Starting JGroups Channel
2013-05-24 23:05:26,588 INFO [JGroupsTransport] (main) ISPN000094: Received new cluster view: [node1-52269|0] [node1-52269]
2013-05-24 23:05:26,594 INFO [JGroupsTransport] (main) ISPN000079: Cache local address is node1-52269, physical addresses are [X]
LISTENER: after ctor
2013-05-24 23:08:05,744 INFO [JGroupsTransport] (Incoming-1,node1-52269) ISPN000094: Received new cluster view: [node1-52269|1] [node1-52269, node1-6039]
LISTENER: submitting task to address node1-6039
LISTENER: current members from event are: [node1-52269, node1-6039]
Members directly from RPC: [node1-52269]
Members from RPC.transport: [node1-52269, node1-6039]
2013-05-24 23:08:05,837 WARN [CacheManagerNotifierImpl] (notification-thread-0,node1) ISPN000134: Unable to invoke method public void test.exec.ViewChangedTest$MembersListener.handleViewChanged(org.infinispan.notifications.cachemanagerlistener.event.ViewChangedEvent) on Object instance test.exec.ViewChangedTest$MembersListener@19c4c8e - removing this target object from list of listeners!
java.lang.IllegalArgumentException: Target node node1-6039 is not a cluster member, members are [node1-52269]
at org.infinispan.distexec.DefaultExecutorService.submit(DefaultExecutorService.java:453)
at org.infinispan.distexec.DefaultExecutorService.submit(DefaultExecutorService.java:442)
at test.exec.ViewChangedTest$MembersListener.handleViewChanged(ViewChangedTest.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.infinispan.notifications.AbstractListenerImpl$ListenerInvocation$1.run(AbstractListenerImpl.java:212)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Is this intended behaviour?
I ran this test on Infinispan 5.2.1final with only core jars needed. attached are the test code (viewChangedTest sets up the listener and Member simply sets up a new member in the cluster) and config files.
Thanks,
Gilad Ber
-
jgroups-udp.xml.zip 1.6 KB
-
infinispan-config.xml 1.9 KB
-
Member.java.zip 539 bytes
-
ViewChangedTest.java.zip 1.1 KB