3 Replies Latest reply on Jun 19, 2013 10:10 AM by rhusar

deploy-hasinglton does not seem to restart on cluster merge (AS5)

kc7bfi Jun 10, 2013 2:02 PM

I am using AS5 and I have a number of SARs in my deploy-hasinglton directory. We have two nodes. For a test we disconnect the network cable from one of the machines. Both nodes are now running all the services in the deploy-hasinglton directory. However, when I reconnect the network cable both nodes see the cluster merge but the services continue to run on both machines. My deploy-hasingleton-jboss-beans.xml has the restartOnMerge property set to true. Any idea what may be wrong? David

1. Re: deploy-hasinglton does not seem to restart on cluster merge (AS5)

kc7bfi Jun 11, 2013 2:02 PM (in response to kc7bfi)

Here is the logging. The cluster sees the merge command but the topology change never seems to get propogated to the HASingletonSupport$HASingletonService so that it can restart the deploy-hasingleton services. Any ideas?

INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,316 INFO Incoming-4,192.168.25.96:49828 org.jboss.ha.framework.interfaces.HAPartition.VICADS New cluster view for partition VICADS: 3 ([192.168.25.95:1099, 192.168.25.96:1099] delta: 1)
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,316 DEBUG Incoming-4,192.168.25.96:49828 org.jboss.ha.framework.interfaces.HAPartition.VICADS dead members: []
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,316 DEBUG Incoming-4,192.168.25.96:49828 org.jboss.ha.framework.interfaces.HAPartition.VICADS membership changed from 1 to 2
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,316 DEBUG AsynchViewChangeHandler Thread org.jboss.ha.framework.interfaces.HAPartition.VICADS Begin notifyListeners, viewID: 3
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,316 INFO AsynchViewChangeHandler Thread org.jboss.ha.framework.server.DistributedReplicantManagerImpl.VICADS Merging partitions...
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,316 INFO AsynchViewChangeHandler Thread org.jboss.ha.framework.server.DistributedReplicantManagerImpl.VICADS Dead members: 0
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,316 INFO AsynchViewChangeHandler Thread org.jboss.ha.framework.server.DistributedReplicantManagerImpl.VICADS Originating groups: [[192.168.25.95:56959|2] [192.168.25.95:56959], [192.168.25.96:49828|2] [192.168.25.96:49828]]
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,320 DEBUG AsynchViewChangeHandler Thread org.jboss.ha.framework.interfaces.HAPartition.VICADS End notifyListeners, viewID: 3
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,321 DEBUG DRM Async Merger#0 org.jboss.ha.framework.server.DistributedReplicantManagerImpl.VICADS Sleeping for 50ms before mergeMembers
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,372 DEBUG DRM Async Merger#0 org.jboss.ha.framework.server.DistributedReplicantManagerImpl.VICADS Start merging members in DRM service...
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,393 WARN Incoming-8,192.168.25.96:49828 org.jgroups.protocols.pbcast.NAKACK 192.168.25.96:49828] discarded message from non-member 192.168.25.95:56959, my view is [192.168.25.96:49828|2] [192.168.25.96:49828]
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,394 ERROR Incoming-13,192.168.25.96:49828 org.jgroups.protocols.pbcast.NAKACK sender 192.168.25.95:56959 not found in xmit_table
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,394 ERROR Incoming-13,192.168.25.96:49828 org.jgroups.protocols.pbcast.NAKACK range is null
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,395 INFO Incoming-13,192.168.25.96:49828 org.jboss.cache.RPCManagerImpl Received new cluster view: MergeView::[192.168.25.95:56959|3] [192.168.25.95:56959, 192.168.25.96:49828], subgroups=[[192.168.25.95:56959|2] [192.168.25.95:56959], [192.168.25.96:49828|2] [192.168.25.96:49828]]
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,628 DEBUG Incoming-16,192.168.25.96:49828 org.jboss.ha.framework.interfaces.HAPartition.VICADS Unblock processed at 192.168.25.96:1099
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,632 DEBUG DRM Async Merger#0 org.jboss.ha.framework.server.DistributedReplicantManagerImpl.VICADS The DRM merge process has received 1 answers
INFO   | jvm 1    | 2013/06/11 12:34:33 | 2013-06-11 12:34:33,633 DEBUG DRM Async Merger#0 org.jboss.ha.framework.server.DistributedReplicantManagerImpl.VICADS ..Finished merging members in DRM service
INFO   | jvm 1    | 2013/06/11 12:34:42 | 2013-06-11 12:34:41,989 INFO Incoming-6,192.168.25.96:49828 org.jboss.messaging.core.impl.postoffice.GroupMember org.jboss.messaging.core.impl.postoffice.GroupMember$ControlMembershipListener@44d7da87 got new view MergeView::[192.168.25.95:56959|3] [192.168.25.95:56959, 192.168.25.96:49828], subgroups=[[192.168.25.95:56959|2] [192.168.25.95:56959], [192.168.25.96:49828|2] [192.168.25.96:49828]], old view is [192.168.25.96:49828|2] [192.168.25.96:49828]
INFO   | jvm 1    | 2013/06/11 12:34:42 | 2013-06-11 12:34:41,989 INFO Incoming-6,192.168.25.96:49828 org.jboss.messaging.core.impl.postoffice.GroupMember I am (192.168.25.96:49828)
INFO   | jvm 1    | 2013/06/11 12:34:42 | 2013-06-11 12:34:41,989 INFO Incoming-6,192.168.25.96:49828 org.jboss.messaging.core.impl.postoffice.GroupMember New Members : 1 ([192.168.25.95:56959])
INFO   | jvm 1    | 2013/06/11 12:34:42 | 2013-06-11 12:34:41,989 INFO Incoming-6,192.168.25.96:49828 org.jboss.messaging.core.impl.postoffice.GroupMember All Members : 2 ([192.168.25.95:56959, 192.168.25.96:49828])
Actions
2. Re: deploy-hasinglton does not seem to restart on cluster merge (AS5)

kc7bfi Jun 13, 2013 12:02 PM (in response to kc7bfi)

I think I found the problem. DistributedReplicantManagerImpl defines the following function

   /**
    * Add a replicant to the replicants map.
    * @param key replicant key name
    * @param nodeName name of the node that adds this replicant
    * @param replicant Serialized representation of the replica
    * @return true, if this replicant was newly added to the map, false otherwise
    */
   protected boolean addReplicant(String key, String nodeName, Serializable replicant)
   {
      ConcurrentMap<String, Serializable> map = new ConcurrentHashMap<String, Serializable>();
      ConcurrentMap<String, Serializable> existingMap = this.replicants.putIfAbsent(key, map);
      return (((existingMap != null) ? existingMap : map).put(nodeName, replicant) != null);
   }

However, I think the return statement should be

      return (((existingMap != null) ? existingMap : map).put(nodeName, replicant) == null);

Any thoughts? David
Actions
3. Re: deploy-hasinglton does not seem to restart on cluster merge (AS5)

rhusar Jun 19, 2013 10:10 AM (in response to kc7bfi)

I am afraid you are hitting a known issue:

https://issues.jboss.org/browse/JBPAPP-6932
Actions

Go to original post