IsLocalInterceptor not detecting local container

[brian.stansberry]

Some unit tests I'm writing are showing a problem with nested SFSBs where the IsLocalInterceptor doesn't treat a call from a parent SFSB to a nested SFSB as local, but Remoting optimizes it locally. This leads to an exception in the transaction propagation handling. Key points are highlighted:

Caused by: java.lang.RuntimeException: cannot import a transaction context when a transaction is already associated with the thread
 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:62)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:93)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:105)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:333)
 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
 at org.jboss.remoting.transport.local.LocalClientInvoker.invoke(LocalClientInvoker.java:98)
 at org.jboss.remoting.Client.invoke(Client.java:589)
 at org.jboss.remoting.Client.invoke(Client.java:581)
 at org.jboss.aspects.remoting.InvokeRemoteInterceptor.invoke(InvokeRemoteInterceptor.java:62)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.remoting.ClusterChooserInterceptor.invoke(ClusterChooserInterceptor.java:77)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.ClientTxPropagationInterceptor.invoke(ClientTxPropagationInterceptor.java:61)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.SecurityClientInterceptor.invoke(SecurityClientInterceptor.java:53)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.remoting.IsLocalInterceptor.invoke(IsLocalInterceptor.java:85)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulClusteredProxy.invoke(StatefulClusteredProxy.java:105)
 at $Proxy110.increment(Unknown Source)
 at org.jboss.ejb3.test.stateful.nested.base.ParentStatefulBean.increment(ParentStatefulBean.java:58)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeTarget(MethodInvocation.java:121)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:110)
 at org.jboss.ejb3.interceptor.InvocationContextImpl.proceed(InvocationContextImpl.java:166)
 at org.jboss.ejb3.interceptor.EJB3InterceptorsInterceptor.invoke(EJB3InterceptorsInterceptor.java:63)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.cache.StatefulReplicationInterceptor.invoke(StatefulReplicationInterceptor.java:51)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.entity.ExtendedPersistenceContextPropagationInterceptor.invoke(ExtendedPersistenceContextPropagationInterceptor.java:57)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.entity.TransactionScopedEntityManagerInterceptor.invoke(TransactionScopedEntityManagerInterceptor.java:54)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.AllowedOperationsInterceptor.invoke(AllowedOperationsInterceptor.java:47)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.TxPolicy.invokeInOurTx(TxPolicy.java:79)
 at org.jboss.aspects.tx.TxInterceptor$Required.invoke(TxInterceptor.java:193)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.tx.TxPropagationInterceptor.invoke(TxPropagationInterceptor.java:76)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulInstanceInterceptor.invoke(StatefulInstanceInterceptor.java:93)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.remoting.ReplicantsManagerInterceptor.invoke(ReplicantsManagerInterceptor.java:51)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.aspects.security.AuthenticationInterceptor.invoke(AuthenticationInterceptor.java:77)
 at org.jboss.ejb3.security.Ejb3AuthenticationInterceptor.invoke(Ejb3AuthenticationInterceptor.java:105)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.ENCPropagationInterceptor.invoke(ENCPropagationInterceptor.java:46)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.asynchronous.AsynchronousInterceptor.invoke(AsynchronousInterceptor.java:106)
 at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:101)
 at org.jboss.ejb3.stateful.StatefulContainer.dynamicInvoke(StatefulContainer.java:333)
 at org.jboss.aop.Dispatcher.invoke(Dispatcher.java:106)
 at org.jboss.aspects.remoting.AOPRemotingInvocationHandler.invoke(AOPRemotingInvocationHandler.java:82)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:1008)
 at org.jboss.remoting.ServerInvoker.invoke(ServerInvoker.java:857)
 at org.jboss.remoting.transport.socket.ServerThread.processInvocation(ServerThread.java:454)
 at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:527)
 at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:261)

What happens here is the parent bean has an increment() method, which delegates to a nested SFSB's increment() method. The nested bean is @Remote. For some reason the IsLocalInterceptor is not recognizing it should invoke the call locally.

This happens in a unit test where I force the failover of the parent SFSB. This is done with the standard trick of adding a server side interceptor that checks a condition and if met throws GenericClusteringException. The stack trace above is *not* from the call that fails over; that works fine. It's the next call to the parent, and is the first call after failover that the parent delegates delegates to the nested bean.

What's really got me stumped is AFAICT IsLocalInterceptor and AOPRemotingInvocationHandler are both dealing with Dispatcher using the same Invocation and the same metadata. But when IsLocalInterceptor queries Dispatcher, it says the container isn't registered, while when AOPRemotingInvocationHandler asks Dispatcher to handle the invocation, it finds the container just fine.

Any thoughts???

[brian.stansberry]

JIRA for this is http://jira.jboss.com/jira/browse/EJBTHREE-881 .

[brian.stansberry]

I (mostly) understand what's going on here, but need input from the rest of the EJB3 team on how to fix it and resolve EJBTHREE-881. I'll post separately later about the bit I don't understand.

Basic problem is very simple:

1) A invocation is made on a parent bean with REQUIRED tx semantics. Parent delegates to an @Remote nested child; tx is associated with the thread.

2) IsLocalInterceptor will only attempt to directly optimize the call to the local bean if the interceptor instance itself was created in the local VM. So, if an SFSB fails over, the interceptor contained in any nested bean proxy will not have been created in the failover VM and IsLocalInterceptor will not optimize the call.

3) Since the call isn't optimized locally, ClientTxPropagationInterceptor adds transaction metadata to the invocation context.

4) org.jboss.remoting.Client detects that the bean *is* available locally, and executes it locally.

5) TxPropagationInterceptor throws an exception when it finds a tx bound to the thread and tx metadata associated with the invocation that it thinks it should import.


A couple possible solutions come to mind, but I don't know enough to recommend one and certainly won't implement one without someone's OK:

Option A) Make IsLocalInterceptor work more like the EJB3 InvokerInterceptor.
InvokerInterceptor performs the following check when deciding whether to optimize a call locally:

i) check if the interceptor was created in the VM (same as the IsLocalInterceptor).
ii) if not, check if the invocation is for a clustered bean.
iii) if i == true || ii == true, check if the target bean exists locally.

IsLocalInterceptor could be made to work the same way. Odd bit is checking if the invocation is for a clustered bean. That could be done by checking the invocation metadata for any of the cluster-related bindings.


Option B) Does TxPropagationInterceptor need to fail if there is already a tx associated with the thread? Could it just accept the existing tx and not bother importing the tx specified in the metadata? Is there any scenario other than this one where a tx would be associated with the thread?

Me thinks Option B is not good, but if there is a reason Option A is no good, it may be needed.

[brian.stansberry]

Never mind about the bit I didn't understand -- it was why this test doesn't always fail. It's because after the failover FirstAvailable picks a new target for the call to the nested bean. Sometimes it picks the local server and we get a failure; sometimes it picks a remote server and we don't get a failure.

Remoting isn't really "detecting" the bean is locally available; it just recognizes that the targeted Invoker is in VM and skips the over-the-wire transport stuff. Recognizing that the bean is available locally is really the task of IsLocalInterceptor. I'm starting to feel pretty certain Option A in my previous post is the way to go.

[bill.burke]

I thought the EJB3 IsLocalInterceptor checks to see if the EJB container exists in the VM and if it does, routes to that EJB Container. This allows the IsLocalInterceptor to be a singleton, and for the request ot be routed directly to the EJB Container bypassing Remoting entirely.

[brian.stansberry]

No, IsLocalInterceptor only checks whether it itself was created in VM and routes locally if it was. Basically, it's:

private static final long stamp = System.currentTimeMillis();
private long marshalledStamp = stamp;

private boolean isLocal()
{
 return stamp == marshalledStamp;
}

public Object invoke(Invocation invocation) throws Throwable
{
 if (isLocal())
 {
 ... bypass remoting; route directly to Container
 return result;
 }
 else
 {
 // continue on to remoting
 return invocation.invokeNext();
 }
}

Now that I think about it, that's definitely not good enough; you have to check for the local bean before routing locally. What if it's been undeployed locally, but some proxy is stored in-VM somewhere (e.g. in a web session)? The way it is now the call will fail.

[bill.burke]

Hmmm, that's the way I implemented and Bill D changed it. I'll have to ask him why.

[bdecoste]

In the prior implementation, if you had a local bean with the same ejbName as a remote bean and you looked up the remote proxy, you incorrectly invoked the local bean. This issue was raised by several support customers.

Here's the JIRA: http://jira.jboss.com/jira/browse/EJBTHREE-773

[bdecoste]

Looks like we at least need a combination of the old and new implementations - route locally if the proxy was created in the same VM and the client and also make sure the bean is in the local EJB3 registry.

[brian.stansberry]

So the EJBTHREE-873 issue is that people didn't want the EJB2 behavior of forcing the call locally? The EJB2 InvokerInterceptor will do that, but only if the bean is clustered.

 public boolean isLocal(Invocation invocation)
 {
 // No local invoker, it must be remote
 if (localInvoker == null)
 return false;

 // The proxy was downloaded from a remote location
 if (isLocal() == false)
 {
 // It is not clustered so we go remote
 if (isClustered(invocation) == false)
 return false;
 }

 // See whether we have a local target
 return hasLocalTarget(invocation);
 }

[bdecoste]

Brian,

Just went through your proposed solution - the problem with the solution is that if there are 2 clusters, each with @Clustered SFSBs with identical ejbNames, when a bean in cluster A looks up a proxy in cluster B and then invokes on that proxy, we will get a local invocation instead of the remote invocation. Seems like we need a way, other than ejbName, to identify if the container is local.

[bdecoste]

No, EJBTHREE-773 is when people look up an EJB3 via a remote jndi server, they want to invoke the remote bean. If the bean is also deployed locally, the local bean was incorrectly called.

[brian.stansberry]

The use case you just described for EJBTHREE-773 is the typical one I hear about when people complain about the local optimization.

In a way, I like not forcing the call to go locally, as it removes a mechanism whereby an SFSB could end up being invoked on 2 different servers.

(I just opened a related thread on the proper behavior of the FirstAvailable load balance policy, please see http://www.jboss.com/index.html?module=bb&op=viewtopic&t=103967. Bill B, you're listed as original author on FirstAvailable, so any comments on that thread are appreciated.)

Re: the 2 cluster problem, should the partition name be part of the ejbName for a clustered EJB? That doesn't really solve the problem though, as you can have two network-isolated clusters both named DefaultPartition.

[bill.burke]

on First Available

SFSB invocation is supposed to be sticky to the machine it originates on so that you are not having to move state between the machines.

[brian.stansberry]

Agreed. So, IMO FirstAvailable.electedTarget shouldn't be transient.

That also implies IsLocalInterceptor shouldn't try to optimize the call locally (at least not for an SFSB...)

Which leads to this. Assume we fix FirstAvailable so the remote target survives:

1) IsLocalInterceptor precedes ClusterChooserInterceptor
2) IsLocalInterceptor lets the call go through as remote
3) Call fails. Likely. The parent bean failed over, pretty likely child will as well.
4) ClusterChooserInterceptor picks a new target -- the local machine.
5) Call goes through and we get the EJBTHREE-881 failure.

Perhaps the solution here lies more in the direction of my hacky Option B, way back when. Find a way to signal to TxPropagationInterceptor that a call that was thought to be remote has ended up going locally. IsLocalInterceptor adds some transient invocation metadata or something.