Is JCA clustered?

[jorgemoralespou_2]

We have developed some JCA connectors for our app, one of them being an LdapJCA connector. Yesterday qe experienced a very strange behaviour that I can`t understand. We have a 2 node cluster accesing an ldap server, this Ldap server went down, and our JCA couldn`t create connections to the Ldap server. So far so good. When the ldap server came up again, we had stuck connections, and the JCA connector couldn`t recreate the Ldap connections. (This is another problem from the one I don`t understand). To let JBoss release those stuck connections, we undeployed the Ldap datasource in one of the nodes of the cluster, and redeployed it again after all connections where released, and everything worked as expected. The thing is it started to work in both nodes of the cluster when we redeployed our datasource in only one of the nodes. Datasource is configured like this:

<?xml version="1.0" encoding="UTF-8"?>
<connection-factories>
 <no-tx-connection-factory>
 <!-- Where to bind the datasource, under the "java:/" prefix -->
 <jndi-name>LdapConnector_Server01</jndi-name>
 <rar-name>ldap-connector.rar</rar-name>
 <connection-definition>com.mycompany.connector.CapabilityConnectionFactory</connection-definition>
 <config-property name="ldapContextFactory" type="java.lang.String">com.sun.jndi.ldap.LdapCtxFactory</config-property>
 <config-property name="providerUrl" type="java.lang.String">ldap://nnssvm02.domaint:389/</config-property>
 <config-property name="securityAuthentication" type="java.lang.String">SIMPLE</config-property>
 <config-property name="securityPrincipal" type="java.lang.String">cn=Directory Manager</config-property>

 <config-property name="securityCredentials" type="java.lang.String">adminxyj</config-property>

 <!-- The minimum number of connections a pool should hold -->
 <min-pool-size>1</min-pool-size>
 <!-- The maximum number of connections for a pool -->
 <max-pool-size>3</max-pool-size>
 <!-- The maximum time in milliseconds to block while waiting for a connection -->
 <!-- before throwing an exception. Note that this blocks only while waiting -->
 <!-- for a permit for a connection, and will never throw an exception if -->
 <!-- creating a new connection takes an inordinately long time -->
 <blocking-timeout-millis>3000</blocking-timeout-millis>
 <!-- The maximum time in minutes a connection may be idle before being closed -->
 <idle-timeout-minutes>0</idle-timeout-minutes>
 <background-validation>true</background-validation>
 <background-validation-minutes>1</background-validation-minutes>
 </no-tx-connection-factory>
</connection-factories>

Can anybody explain why this happened?

[vickyk]

"jorgemoralespou_2" wrote:
When the ldap server came up again, we had stuck connections, and the JCA connector couldn`t recreate the Ldap connections. (This is another problem from the one I don`t understand).

This seems to be logical and it should be handled by providing some configurable property of the RA (ldap-connector.rar) or the MCF .
The pool of connections which is held by the JCA has become stale as your ldap server have been restarted , we have taken care of this scenario by providing check-valid-connection-sql property for the jdbc related RA's .

When the application takes connection from the JCA pool and if we expect the connections to be stale/broken due to restart of the DB while the Jboss AS is running then the JCA closes the stale connection and recreates the new one , all this is transparent to the application .

You will have to design the similar feature for the ldap related connector , I am not the ldap expert so can't tell you the exact way to validate these stale connections .
I am sure the guys who might have written the ldap RA for you should be able to implement this feature .
The validation of the stale/broken connection should be done through the MCF.matchConnection(...) operation .

Here is how we take care of the broken/stale connection in jdbc adapter
http://wiki.jboss.org/wiki/WhatHappensToJBossIfIRestartTheDatabase
http://wiki.jboss.org/wiki/FAQJBossJCA

How about making the ldap ra as the opensource ?
Let me see if we can have the generic ldap RA shipped with our new JCA project .

[jorgemoralespou_2]

Thanks Vickyk for your answer,
But my question was not for the reconnection to the ldap backend. We know where the problem might be. What we don't understand is the problem we have seen where when we fix connections from one node of the cluster to the backend (redeploying the datasource), connections to the backend gets also fixed in the other node of the cluster.

Can you throw some light on this?

[vickyk]

"jorgemoralespou_2" wrote:
What we don't understand is the problem we have seen where when we fix connections from one node of the cluster to the backend (redeploying the datasource), connections to the backend gets also fixed in the other node of the cluster.

Can you outline the sequence of operations which you are performing in the sequential order so that I get an clearer idea about what you are doing ?

[jorgemoralespou_2]

Yes. Here it goes.

1- We have 2 node cluster accesing an Ldap backend via JCA conx. (node-1, node-2)
2- Our Ldap server is restarted (due to maintenance)
3- We have a bug in our Ldap JCA conector, that doesn't detect invalid conections. So no reconnection happens.
4- In this point, bot nodes (node-1 and node-2) don't have valid connections in pool.
5- We undeploy and redeploy ldap datasource in node-1, which makes old connections die, and creates new connections.
6- In node-2 Ldap recreates connections succesfully. (Note we haven't touched this server at all)

This same behaviour has also happened with JMS jca connector, but we have this configured as clustered, so, somehow can be normal.

Is it more clear now?

[vickyk]

"jorgemoralespou_2" wrote:

6- In node-2 Ldap recreates connections succesfully. (Note we haven't touched this server at all)

I am not sure if you are doing the farm deployment , if you are doing so then I would expect the deployment changes to get reflected across nodes .
http://www.wideopen.com/docs/manuals/jboss/jboss-eap-4.2/doc/Server_Configuration_Guide/Introduction-Farming_Deployment.html

Let us see if you can simulate it at your end , you should perform these steps
1) Make min-connection=max-connection=3 .
2) Use prefill tag to prefill the pool at both nodes.
3) Now shutdown the ldap .
4) Go to node 1 and see if application gets the stale connection , it should .
5) Try the [4] with node2 .
6) redeploy the -ds.xml in node1 and see if application can now connect to ldap , it should .
7) check if the application at node2 can connect to ldap now .Let me know if you it .