Caching lookups

[dfisher]

I'm running 4 machines with Jboss 3.2.5.
2 machines are web app servers and the other two are for EJBs.
When both instances of JBoss are stopped on the EJB machines and restarted, the web app machines cannot lookup beans on the EJB machines without themselves being restarted.
What is the proper way to lookup EJBs in this sort of environment.
I've tried several different methods of caching and not caching the lookups and the InitialContext, but nothing has worked.

[dfisher]

I've done some testing and here are the results I'm seeing.
I am looking up an entity bean from a remote host which uses hajndi to get the bean from a cluster of 2 machines.

Using the jnp.partitionName produces undesireable results.
When both nodes are down and then one comes back up, the client lookups will fail with "Service Unavailable" exceptions.
The client must be restarted to fix the problem, creating a new InitialContext will not help.

Using the java.naming.provider.url produces better, but not optimal results.
When both nodes are down an exception will be thrown on the client, but creating a new InitialContext will give you a context on the localhost which has recent lookups cached.
Any lookups on immutable data will succeed, but volatile data cannot be found and lookups will return null.
This means the client doesn't have to restart, which is good, but the behavior can give you headaches.
You can never know if your lookup is legitimately returning null, or if your cluster is down.

Does anyone have suggestions for clients surviving when an entire cluster goes down?