Single Client. Multiple Initial contexts not working.

[xavierpayne2]

Hey. I'm really hoping someone can help me out here. We were supposed to release a week ago and this bug cropped up...

We have a Standalone client that grabs a remote remote Stateless Session Bean ref to two different JBoss 4.0.2 application servers (they are not a cluster).

We are setting the InitialContext for each lookup explicitly with the following code:

 Context initial = null;

 Hashtable contextSettings = new Hashtable();
 contextSettings.put("jnp.disableDiscovery", "true");
 contextSettings.put(Context.INITIAL_CONTEXT_FACTORY, "org.jnp.interfaces.NamingContextFactory");
 contextSettings.put(Context.URL_PKG_PREFIXES, "org.jboss.naming:org.jnp.interfaces");
 contextSettings.put(Context.PROVIDER_URL, "jnp://hostname:1099");

 initial = new InitialContext(contextSettings);

Where hostname is set dynamically in the code. We have a system.out.println at this point and verify that when the code gets called the ip address is getting set to the correct value....

However we are noticing weird behavior when we go to use any ejbs. For example we have an application that does the following

1) create InitialContext for FOO
2) lookup and do ejb.create using FOO context.

3) create InitialContext for BAR
4) lookup and do ejb.create using BAR context

5) Invoke method on ejb from FOO context... Works client hits FOO

6) Invoke method on ejb from BAR context... Works client his BAR

7) Invoke method on ejb in FOO context... FAILS! Client hits BAR!!

Why would calls to FOO go to BAR? Is it a bug? I am working on a JUNIT test that doesn't require all of our code. (basically drop the same EJB into two seperate JBoss 4.0.2 app servers and run a client that attempts to do calls on both in mixed order)

Any help on this would be greatly appreciated as we have tried everything we can think of. We've had 3 people working on this for 2 days and our code release is basically at a standstill as connection to multiple servers is a hard and fast requirement.

[achetan]

Hi

i have faced the similar problem in JBOSS 4.0.1 which was having following problem:

http://jira.jboss.com/jira/browse/JBAS-1442

I switched to JBOSS 4.0.2. In this versiob , it should work..

Please let me also know if you are able to figure put the answer as iam also spending lot of time on this.

Thanks
Chetan

[xavierpayne2]

Loading up my client in the Eclipse debugger I can see that its the InvocationContext that is getting confused. So my lookups are indeed being performed correctly but at some point that I haven't pinned down just yet the InvocationContect is getting changed internally.

Attached is a screenshot of my debugger window with just the first three octaves of my ip censored for security reasons. The important things (to me) are the String intendedServerIP ... this is a variable I created to tell me what the object believes is the IP Address the remote ref it has is using.

Notice that further down in the tree under Endpoint ep = TCPEndpoint that it believes the String host is the other ip!

I can continue to dig but it seems like even with DiscoveryDisabled set to true in cluster-service.xml

and "jnp.disableDiscovery" set to true in my client side JNDI lookup and using port 1099 part of the code are still trying to act like each server I jndi to is part of the same cluster family.

Now that this is jacked into the debugger I can also attempt to provide more information if desired but I don't know what's most helpful to you guys.

Something also important to note is that the first few calls on this ejb do in fact work and call the correct server. So it seems as if the InvocationContext is changing on the fly between calls.

[xavierpayne2]

Doh... I can't upload an image so here is a copy paste of the debug tree.

PublisherSequence this= PublisherSequence (id=27)
 String intendedServerIP= "jnp://x.x.155:1099"
 PublisherSequenceSecurityProxyRemote pubSeq= $Proxy3 (id=40)
 InvocationHandler h= ClientContainer (id=86)
 InvocationContext context= InvocationContext (id=90)
 Map context= HashMap (id=111)
 Set entrySet= HashMap$EntrySet (id=260)
 Set keySet= null
 float loadFactor= 0.75
 int modCount= 0
 int size= 5
 HashMap$Entry[] table= HashMap$Entry[16] (id=116)
 HashMap$Entry [0]= null
 HashMap$Entry [1]= HashMap$Entry (id=122)
 HashMap$Entry [2]= HashMap$Entry (id=124)
 HashMap$Entry [3]= HashMap$Entry (id=125)
 int hash= -1368190845
 Object key= InvocationKey (id=138)
 HashMap$Entry next= null
 Object value= JRMPInvokerProxyHA (id=140)
 FamilyClusterInfo familyClusterInfo= FamilyClusterInfoImpl (id=193)
 Object arbitraryObject= null
 long currentViewId= 1560272158
 int cursor= 0
 String familyName= "DefaultPartition/jboss.j2ee:jndiName=PublisherSequenceSecurityProxyBean,service=EJB/R"
 boolean isViewMembersInSyncWithViewId= true
 ArrayList targets= ArrayList (id=201)
 Object[] elementData= Object[1] (id=208)
 Object [0]= JRMPInvoker_Stub (id=213)
 RemoteRef ref= UnicastRef2 (id=220)
 LiveRef ref= LiveRef (id=230)
 Channel ch= TCPChannel (id=233)
 Endpoint ep= TCPEndpoint (id=236)
 RMIClientSocketFactory csf= null
 String host= "x.x.x.87"
 int port= 1872
 RMIServerSocketFactory ssf= null
 TCPTransport transport= null
 ObjID id= ObjID (id=239)
 boolean isLocal= false
 int modCount= 0
 int size= 1
 LoadBalancePolicy loadBalancePolicy= RoundRobin (id=196)
 String proxyFamilyName= "DefaultPartition/jboss.j2ee:jndiName=PublisherSequenceSecurityProxyBean,service=EJB/R"
 Invoker remoteInvoker= null
 boolean trace= false
 HashMap$Entry [4]= null
 HashMap$Entry [5]= null
 HashMap$Entry [6]= HashMap$Entry (id=126)
 HashMap$Entry [7]= null
 HashMap$Entry [8]= null
 HashMap$Entry [9]= null
 HashMap$Entry [10]= null
 HashMap$Entry [11]= null
 HashMap$Entry [12]= null
 HashMap$Entry [13]= null
 HashMap$Entry [14]= null
 HashMap$Entry [15]= HashMap$Entry (id=127)
 int threshold= 12
 Collection values= null
 Interceptor next= StatelessSessionInterceptor (id=92)
 String seqID= "4b6s1o4b-e4z81o-edak9xvu-1-edakrm22-a"
 String version= "1.0"

[xavierpayne2]

We redeployed everything into the default directory (it was in the "All" directory) and connections to multiple (non clustered) JBoss application servers works now. But of course we loose the ability to have JBossAS clusters.

So the answer for us for now seems to be to support multiple connections to different JBoss 4.0.2 application servers as long as none of them belong to a cluster.

:(

[xavierpayne2]

Just for completeness... here is the debug tree from eclipse for the deployment in the default directory which is working...

PublisherSequence this= PublisherSequence (id=28)
 PublisherSequence IOS= PublisherSequence (id=27)
 String intendedServerIP= "jnp://x.x.x.155:1099"
 PublisherSequenceSecurityProxyRemote pubSeq= $Proxy3 (id=40)
 InvocationHandler h= ClientContainer (id=83)
 InvocationContext context= InvocationContext (id=87)
 Map context= HashMap (id=93)
 Set entrySet= null
 Set keySet= null
 float loadFactor= 0.75
 int modCount= 0
 int size= 5
 HashMap$Entry[] table= HashMap$Entry[16] (id=97)
 HashMap$Entry [0]= HashMap$Entry (id=100)
 HashMap$Entry [1]= HashMap$Entry (id=103)
 int hash= -1783084207
 Object key= InvocationKey (id=120)
 HashMap$Entry next= null
 Object value= JRMPInvokerProxy (id=184)
 Invoker remoteInvoker= JRMPInvoker_Stub (id=187)
 RemoteRef ref= UnicastRef2 (id=196)
 LiveRef ref= LiveRef (id=205)
 Channel ch= TCPChannel (id=207)
 Endpoint ep= TCPEndpoint (id=210)
 RMIClientSocketFactory csf= null
 String host= "x.x.x.155"
 int port= 4444
 RMIServerSocketFactory ssf= null
 TCPTransport transport= null
 ObjID id= ObjID (id=213)
 boolean isLocal= false
 HashMap$Entry [2]= null
 HashMap$Entry [3]= null
 HashMap$Entry [4]= null
 HashMap$Entry [5]= null
 HashMap$Entry [6]= HashMap$Entry (id=104)
 HashMap$Entry [7]= null
 HashMap$Entry [8]= null
 HashMap$Entry [9]= null
 HashMap$Entry [10]= null
 HashMap$Entry [11]= null
 HashMap$Entry [12]= null
 HashMap$Entry [13]= null
 HashMap$Entry [14]= null
 HashMap$Entry [15]= HashMap$Entry (id=105)
 int threshold= 12
 Collection values= null
 Interceptor next= StatelessSessionInterceptor (id=89)
 String seqID= "4b6s1o4b-tw3zkv-edbpzjql-1-edbqm7dd-b"
 String version= "1.0"
 int seqState= 0

The difference seems to be that somehow the "all" configuration is signalling the client side to act in clustered fashion... That by itself would not be a problem though... The real problem seems to be that the when using the JRMPInvokerProxyHA it does not appear to be able to distinguish between two seperate clusters... so all the ips get lumped together and seen as one cluster in the client side. I don't see any way around this...

[xavierpayne2]

FOUND A SOLUTION!

Each JBossAS was using DefaultPartition. One of the keys in the hashmap must be based on this. Because as an experiment changing the partition name on one of the servers in the all configuration caused them both to worth concurrently without issue!

-Djboss.partition.name=MyWonderfulCluster

So we can have two clusters now and hit them each seperately and still have ha :). I am off to do more testing.