Fail over isn't working: Connection refused

[hnrk]

I have a failover problem, and I just can't figure it out.

I have two JBoss 4.0.4 servers, with several stateless EJB2 beans. Example jboss.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE jboss PUBLIC "-//JBoss//DTD JBOSS 4.0//EN" "http://www.jboss.org/j2ee/dtd/jboss_4_0.dtd">

<jboss>
 <enterprise-beans>
 <session>
 <ejb-name>MyService</ejb-name>
 <jndi-name>ejb/MyService</jndi-name>
 <configuration-name>MyClusteredSessionBean</configuration-name>

 <clustered>true</clustered>
 <method-attributes>
 </method-attributes>
 </session>
 </enterprise-beans>
</jboss>

This is the container config in standardjboss.xml:

 <container-name>MyClusteredSessionBean</container-name>
 <call-logging>false</call-logging>
 <invoker-proxy-binding-name>clustered-stateless-rmi-invoker</invoker-proxy-binding-name>
 <container-interceptors>
 <interceptor>org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor</interceptor>
 <interceptor>org.jboss.ejb.plugins.CleanShutdownInterceptor</interceptor>
 <interceptor>org.jboss.ejb.plugins.LogInterceptor</interceptor>
 <interceptor>org.jboss.ejb.plugins.SecurityInterceptor</interceptor>
 <!-- CMT -->
 <interceptor transaction="Container">org.jboss.ejb.plugins.TxInterceptorCMT</interceptor>

 <interceptor transaction="Container">org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
 <interceptor transaction="Container">org.jboss.ws.server.ServiceEndpointInterceptor </interceptor>
 <interceptor transaction="Container">org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor</interc
eptor>
 <!-- BMT -->
 <interceptor transaction="Bean">org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor</interceptor
>
 <interceptor transaction="Bean">org.jboss.ejb.plugins.TxInterceptorBMT</interceptor>
 <interceptor transaction="Bean">org.jboss.ejb.plugins.CallValidationInterceptor</interceptor>
 <interceptor transaction="Bean">org.jboss.ws.server.ServiceEndpointInterceptor </interceptor>
 <interceptor>org.jboss.resource.connectionmanager.CachedConnectionInterceptor</interceptor>
 </container-interceptors>
 <instance-pool>org.jboss.ejb.plugins.StatelessSessionInstancePool</instance-pool>
 <instance-cache></instance-cache>
 <persistence-manager></persistence-manager>
 <container-pool-conf>
 <MaximumSize>100</MaximumSize>
 </container-pool-conf>
 <cluster-config>
 <partition-name>${jboss.partition.name:DefaultPartition}</partition-name>
 </cluster-config>
 </container-configuration>

and the the invoker in standardjboss.xml:

 <invoker-proxy-binding>
 <name>clustered-stateless-rmi-invoker</name>
 <invoker-mbean>jboss:service=invoker,type=jrmpha</invoker-mbean>
 <proxy-factory>org.jboss.proxy.ejb.ProxyFactoryHA</proxy-factory>
 <proxy-factory-config>
 <client-interceptors>
 <home>
 <interceptor>org.jboss.proxy.ejb.HomeInterceptor</interceptor>
 <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
 <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
 <interceptor>org.jboss.proxy.ejb.RetryInterceptor</interceptor>
 <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
 </home>
 <bean>
 <interceptor>org.jboss.proxy.ejb.StatelessSessionInterceptor</interceptor>
 <interceptor>org.jboss.proxy.SecurityInterceptor</interceptor>
 <interceptor>org.jboss.proxy.TransactionInterceptor</interceptor>
 <interceptor>org.jboss.proxy.ejb.RetryInterceptor</interceptor>
 <interceptor>org.jboss.invocation.InvokerInterceptor</interceptor>
 </bean>
 </client-interceptors>
 </proxy-factory-config>
 </invoker-proxy-binding>

My test client is simple. I generate a home interface, and do repeated create() while I run a shutdown on one of the servers. This is my argument to InitialContext():

 Hashtable<String, String> env = new Hashtable<String, String>();
 env.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.NamingContextFactory");
 env.put(Context.URL_PKG_PREFIXES, "jboss.naming:org.jboss.naming");
 env.put(Context.PROVIDER_URL, "server1:1100,server2:1100");

When I shutdown the server, I first get a null object Exception:

java.lang.IllegalArgumentException: null object name
 at org.jboss.mx.server.registry.BasicMBeanRegistry.get(BasicMBeanRegistry.java:509)
 at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:653)
 at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:819)
 at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:420)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
 at sun.rmi.transport.Transport$1.run(Transport.java:153)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
 at java.lang.Thread.run(Thread.java:595)
 at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:247)
 at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:223)
 at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:126)
 at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
 at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133)
 at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:331)
 at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:194)
 at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
 at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
 at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:184)
 at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
 at $Proxy0.create(Unknown Source)

.. then I get connection refused:

Connection refused to host: 10.16.202.3; nested exception is:
 java.net.ConnectException: Connection refused
java.rmi.ConnectException: Connection refused to host: 10.16.202.3; nested exception is:
 java.net.ConnectException: Connection refused
 at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:574)
 at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:185)
 at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:171)
 at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:94)
 at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
 at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133)
 at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:331)
 at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:194)
 at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:61)
 at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:70)
 at org.jboss.proxy.ejb.HomeInterceptor.invoke(HomeInterceptor.java:184)
 at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)

I have tried various things, like upgrading the ejb jar to EJB3 Preview 9. This had no effect. I have tried increasing the log level of org.jboss.invocation to TRACE, but no extra logging occur.

It looks like the client doesn't get a home stub with failover capabilities at all.

Repeated use of the client while both servers are running shows that the load-balancing part is working.

Any ideas?

[hnrk]

As a followup, a few observations:

The exception stack traces show that org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke() is used, while I believe JRMPHAInvoker should have been used?

Also, I replaced SingleRetryInvoker with RetryInvoker. Shouldn't my client wait for a connection indefinately instead of issuing a connection refused?

I am starting to suspect that my clustered container isn't used at all. Is there a way to check this?

[brian.stansberry]

You're not getting an HA proxy. Not sure why. Easiest way to tell is to look at the stack trace, like you did. You'll also see that no RetryInterceptor is in the client stack trace.

What happens if you remove clustered=true from jboss.xml and just rely on the fact you've designated a configuration set up for clustering? (Trying that is just a shot in the dark.)

I suspect it appears you're getting load balancing because you're doing multiple JNDI lookups. The JNDI lookups are load balanced, so you're getting different non-HA proxies. If you're not doing multiple JNDI lookups, let me know.

[hnrk]

You're not getting an HA proxy. Not sure why. Easiest way to tell is to look at the stack trace, like you did. You'll also see that no RetryInterceptor is in the client stack trace.

By writing the problem report, I managed to think through and debug the problem properly.

I did all my testing against two of the 20+ beans in our application. As luck would have it, these two (and only these two) had typos in their XDoclet descriptors. This, in turn, caused their generated jboss.xml files to use the standard non-clustered container.

I really should have spotted this earlier.

Anyway, thanks for the input!

[brian.stansberry]

That happens to me about 3 times a week -- I write some long thing asking for help, and by the time I get to the end the process of writing has told me the answer :-).

Glad it's working.