The one possible thing I saw was that if a user set the -Dorg.jboss.ejb.sfsb.handle.V327 JVM flag, a reference to the Invoker would be stored in the EJB handle and would thus be serialized. Deserializing this could pollute the FamilyClusterInfo, but AFAICT only until getEJBObject() was called on the deserialized handle. Typically, I would expect that call to be made almost immediately after deserializing the handle.

There is no need for any kind of cluster restart to be involved to have this problem. ClusteringTargetsReposity.initTarget() will replace the target list any time its called, irregardless of the value of the viewId.

That's not a trivial problem to fix by adding logic to only update the target list if the new viewId is greater than the old. This is because the viewId is *not* a counter that is incremented as the list of replicants changes. It's a hash of the names of the nodes on which the service is deployed.

If we change this hash-based viewId to a counter, we have to ensure that the counter stays in sync across the cluster (which I'm sure is why the hash approach was chosen :)

I haven't thought this through yet, but my instinct tells me a solution might lie in the fact that there are two paths by which a set of targets are received:

1) As part of an HARMIResponse. This list of targets can be considered authoritative, as HARMIResponse only comes from the server.
2) As part of a serialized proxy. This list is not necessarily authoritative, as there is no way to know where the proxy came from. If no FamilyClusterInfo exists for the proxy's key, the proxy's targets are authoritative. Otherwise, not.

The trick is what to do with the non-authoritative targets. They can be stored in FamilyClusterInfo (and flushed any time a set of authoritative targets comes in), but how to make them available to the load balance policy? Just append them at the end of the target list returned by getTargets()?

I'd like to think a bit about where we want to go on this for 4.0.4. I'd thought through how to implement the GUID concept, but passing around the GUID will affect a lot of classes and will only partly solve one use case -- node A passes bad proxy to node B with cluster restart in between. I say partly solve because when node B uses the bad proxy, the call will still fail, but at least B's own FamilyClusterInfo won't be corrupted.

I agree with Tom's thinking on adding an ability to do discovery after a failure, but adding a full-fledged framework for that for 4.0.4 seems a stretch.

For EJB's at least, we have an existing technology that AFAIK works -- the RetryInterceptor. It has two flaws that I see:

1) To configure the InitialContext, the user has to call a static method on the interceptor class. I think this can be improved by using NamingContextFactory.lastInitialContextEnv.get() like Adrian did in the EJB handle classes.
2) The RetryInterceptor keeps retrying forever, whereas what we want is a version that attempts one JNDI lookup and then gives up.

One possibility for 4.0.4 is to create an OnlyOnceRetryInterceptor and add that too our standard clustered EJB stacks.

This could be an implementation of a more generic DiscoveryInterceptor concept, where the discovery mechanism is pluggable. (Tom, don't know if you were thinking along these kinds of lines).

The "OnlyOnceRetryInterceptor" (renamed "SingleRetryInterceptor") mentioned above is implemented and added to the default EJB client interceptor stacks. I think that reasonably addresses the problem of invalid target lists for EJB2 ejbs.

That leaves EJB3 ejbs (where I assume a similar approach could be used) and other non-EJB clustered proxies.

Did we earlier reach consensus that a "discovery-after-failure" mechanism is the right solution to this problem rather than the GUID? If so, I would like to close JBAS-1476, which specifies the GUID approach, and open a new issue to track adding a discovery process for the other proxy types.

A footnote re: a conversation Bela and I had a couple weeks back.

We were discussing a potential problem if a cluster got restarted, and after restart the viewId recalculated to the same number it had before. This could happen if JGroups were configured with a bind_port, as then all the JGroups IpAddress' would use the same address and port.

In this case, we were concerned there might be problem if a client contacted the cluster, transmitting the old viewId. The server wouldn't know the viewId was from the "old" cluster instance and therefore wouldn't send the new targets with the response. Would this cause a problem that adding a GUID would solve?

I've thought about this some more and have come to the following conclusions:

1) Not an issue w/ JRMPInvokerProxyHA. If the cluster is restarted, none of the old client-side targets will be valid, so the client will not be able to contact the cluster without doing a RetryInterceptor-style lookup. That lookup will bring in the current targets.

2) Not an issue with HttpInvokerProxyHA. The targets are just URLs; even if the targets aren't refreshed, the old ones are still usable. In the very off chance that one of the restarted servers changed it's HttpInvoker URL, that target will be incorrect, but this will be detected as soon as an attempt is made to contact the target. That call will fail, and the client will thus know its target set has an issue and will get fresh targets on the next successful call.

3) AFAICT, not a problem with PooledInvokerProxyHA, for largely the same reason as HttpInvokerProxyHA -- the targets are just address/port combinations.