Handling soft versus hard pooledInvoker exceptions in JRMPIn

[smarlow]

Before I create a jira for this issue, I wanted to gather some feedback. Let me know if I didn't explain this clearly enough. :-)

I noticed that the PooledInvokerProxy.invoke() handles java.io.EOFException by throwing a java.rmi.ConnectException (this is in a general catch all Exception).

I happened to hit this case and found that the call target (cluster member) is removed as a dead member (in JRMPInvokerProxyHA.invoke). This is not the right thing to do for java.io.EOFException which only means that the pooled connection went bad but the cluster member is still alive (in my case the cluster member was still fine.)

As a quick hack, I'm going to try changing the JRMPInvokerProxyHA.invoke from the following:

 catch (java.rmi.ConnectException e)
 {
 lastException = e;
 }

To:

 catch (java.rmi.ConnectException e)
 {
 lastException = e;
 if(e.getCause() != null && e.getCause() instanceof java.io.EOFException)
 {
 // don't failover as we may of reached the target
 invocationHasReachedAServer (invocation);
 throw e;
 }
 }

I need to do more testing to see if this is the right approach (need to ensure that the real fail-over case still works).

Perhaps there are other exceptions that should also be treated differently.

Thoughts?

[galder.zamarreno]

https://jira.jboss.org/jira/browse/JBAS-4963

I created this JIRA a while back to look into the differences between JRMP and Unified invoker proxies in terms of whether failover would be allowed. You could probably add your stuff in there? Feel free to assign it to yourself :)

[brian.stansberry]

Yes, please group this with JBAS-4963, either as a subtask or as a separate issue that JBAS-4963 depends on.

In general, this pooled code looks hacky. PooledInvokerProxy catching any exception and throwing it as a java.rmi.ConnectException? Basically that seems like a manifestation of a decision to not create a full separate set of classes for the Pooled invokers vs. JRMP, so you end up w/ pooled pretending to be JRMP.

I'm concerned about your fix, because it extends that problem, by introducing a workaround for the pooled case into a class that's also meant to support JRMP. Looking at JRMPInvokerProxyHA, it's probably OK, but I'd like your thoughts on that.

[smarlow]

I agree that the pooled code looks hacky in the exception handling. I would like to explore what the better fix might be and then we can consider if that is safe enough (I don't want to break fail-over for anyone :-).

On the other hand, a hack fix might be safer for fixing earlier releases (creating a proper full set of classes for the Pooled invokers.)

[smarlow]

On the other hand, a hack fix might be safer for fixing earlier releases (creating a proper full set of classes for the Pooled invokers.)

I meant to say that a hack fix might be safer for fixing earlier releases (than creating a proper full set of classes for the Pooled invokers.)

[brian.stansberry]

Agreed about safety in the 4.x releases. These are client-side classes and 4.x is in micro-release mode with possibly older releases trying to communicate with any 4.2.4 this would go into. Adding new classes would break such clients.

It's probably about too late to add new classes to 5.x as well, as the CR2 is about to go out.

I think what you propose looks OK; it basically mimics the behavior of the finally "catch (Exception e)" block in JRMPInvokerProxyHA.invoke(). AFAICT, only side effect it would have would be to prevent failover if java.rmi.ConnectException(cause EOFException) were returned from the server side. And if that happened, preventing failover is probably the right thing to do anyway.

[smarlow]

I created JBAS-5943 and linked it to JBAS-4963. It looks like Jira has been tightened up to not let me assign JBAS-5943 to myself.

[brian.stansberry]

You have two accounts, one is you @ less recent employer, other is you @ more recent employer. You filed it under you @ less recent. I just assigned it to you @ more recent. I'll change it to the other if you like.

[smarlow]

No, that is good. Thanks. Someday I'll go from three userids, down to one. :-)

[smarlow]

I'm having difficulties running the clustered unit tests on my Windows laptop. I added two virtual network (loopback) adapters and can run the JGroups draw program with those but the app servers launched from the clustered unit test don't see each other.

I want to run the unit tests before my change and again with my changes. I've reviewed the various tips for running on Windows (disable media sense, adding virtual ip addresses).

I'll continue to work through this (unless someone can suggest a shorter path?)

In local.properties, I specified the virtual ip addresses:

node0=169.254.25.130
node1=169.254.25.129

[galder.zamarreno]

"ScottMarlowNovell" wrote:
... but the app servers launched from the clustered unit test don't see each other.

How are you launching them the AS instances from the clustered unit test? DId you check one-cluster-test target in testsuite?

[brian.stansberry]

For testing on Windows, I always ended up assigning two static IPs to one of my regular NICs.

Odd, though, that Draw works while the AS doesn't.

[smarlow]

Brian,

Assigning two static IPs to my regular NIC, did the trick!

Thanks for the tip.

Scott

[smarlow]

The clustered unit tests passed. I'll merge the change to trunk + 4_2.

[brian.stansberry]

There's a code freeze in effect for trunk; please wait on that. Dimitris will send out a notice on [e-mail] when lifted (probably 24 to 48 hours).