DataSource failover

[adrian.brock]

This relates to the following task:
http://jira.jboss.com/jira/browse/JBAS-1391

Rather than discuss the design in JIRA we can discuss it here.

[adrian.brock]

For the initial implementation, I would like to see the connectionURL
remain sticky until it needs to failover.
That will reduce the problem of inconsistency where the replicating database
has some latency in its replication or does not fully support distributing locking
(maybe only commit time leading to excessive lazy failures).

[adrian.brock]

For a full implementation, I would like to see this exposed as a policy
such that the deployer can choose the failover policy
(and potentially load balancing policy if the db supports it correctly).

This will require not just checking for errors at connection construction time,
but also tracking failures from matchManagedConnections (checkValidConnection)
and runtime errors checkError/ExceptionSorter.
The ExceptionSorter would need to know the difference between
a FATAL error (the connection is broken) versus a FAILOVER error
(the database is broken).

Extra consistency could also be added by extending the ConnectionEventListener
interface with a propriatory interface, adding

void connectionFailoverOccurred(ConnectionEvent event);

This would allow the adapter to signal that the connection should be closed
once the application has finished with it, as opposed to the
connectionErrorOccurred which signals immediate destruction.

With this extra method, the pluggable policy could choose a "first available"
type policy.
e.g.
1) main db server fails so JBoss fails over to secondary/backup server
2) it remains sticky on that server until the main server is back
3) it then hands out connections from the main server and tells the
listeners (JBoss's connection manager/pool) that the connections to the
backup server should be closed when the applications have finished
with the current requests.

[aloubyansky]

The testcase for this issue is added

package org.jboss.test.jca.test;
public class HAConnectionFactoryUnitTestCase

It's based on the HSQLDB. Also I didn't mention that it requires checking the validity of the connection before it is given to a caller. In the testcase, I use check-valid-connection-sql for that.

[aloubyansky]

What about the introduction of the new ha-local-tx-datasource element?
We could probably do some XSL tricks (like check whether url-delimeter is present and connection-url actually contains more than one url) and avoid the introduction of a new element.

[adrian.brock]

What about the introduction of the new ha-local-tx-datasource element?
We could probably do some XSL tricks (like check whether url-delimeter is present and connection-url actually contains more than one url) and avoid the introduction of a new element.

Yes, but I want it to be experimental for now.
i.e. the user chooses they are going to use it.
It should be documentated as such.

It should be trivial in future releases to make ha-local and local do the same thing,
once we are confident it is working correctly.

[adrian.brock]

Also I didn't mention that it requires checking the validity of the connection before it is given to a caller. In the testcase, I use check-valid-connection-sql for that.

This is true regardless of HA.
JDBC has no mechanism to signal a broken connection asynchronously.
The only way to avoid broken connections being given to the application
(where it will find it is broken on first use) is to have a validity checker
that checks the connecton is not broken before it is handed out from the pool.
http://www.jboss.org/wiki/Wiki.jsp?page=WhatHappensToJBossIfIRestartTheDatabase

[aloubyansky]

Now URLs are selected with sticky round-robin algorithm. This is done in URLSelector which is an inner class of HALocalManagedConnectionFactory.

Should I be warried about thread-safeness of the whole implementation?

[aloubyansky]

It seems to be logical to be able to specify user/password per URL instead of using the same user/pwd for every url in the list?

Then, as for me, it would make sense to still configure the ha datasource as a collection of datasources. Not using all the elements that are usually used to configure a local datasource but a subset (since, e.g. jndi-name would be specified only for the ha one).

The same for XA. If I am going to specify properties per connection, I would collect them under a single parent element.

What do you think?

[adrian.brock]

Should I be warried about thread-safeness of the whole implementation?

Yes, the pool allows multiple threads to be dealing with connection construction/validation
at the same time, upto the max-size which is the basis for the semaphore.

[adrian.brock]

It seems to be logical to be able to specify user/password per URL instead of using the same user/pwd for every url in the list?

Then, as for me, it would make sense to still configure the ha datasource as a collection of datasources. Not using all the elements that are usually used to configure a local datasource but a subset (since, e.g. jndi-name would be specified only for the ha one).

The same for XA. If I am going to specify properties per connection, I would collect them under a single parent element.

What do you think?

Unfortunatley, the JCA spec does not allow for complicated javabean construction

<!--
The element config-property-type contains the fully qualified Java
type of a configuration property as required by ManagedConnection-
Factory instance.

The following are the legal values of config-property-type:
java.lang.Boolean, java.lang.String, java.lang.Integer,
java.lang.Double, java.lang.Byte, java.lang.Short,
java.lang.Long, java.lang.Float, java.lang.Character

Example: <config-property-type>java.lang.String</config-property-type>
-->
<!ELEMENT config-property-type (#PCDATA)>

Like I said before, perhaps a better approach is to plugin a strategy/policy
where this could be better defined?

Don't forget the user/password are only used when there is no Subject,
i.e. the admin has configured JAAS for the logins
In fact, this whole issue is very complicated and ill defined/supported in the spec.

[aloubyansky]

I've got first HA XA tests passing. Here is the ha-xa-datasource configuration I used for testing

 <ha-xa-datasource>
 <jndi-name>XADerbyDS</jndi-name>
 <track-connection-by-tx/>
 <isSameRM-override-value>false</isSameRM-override-value>
<!-- <xa-datasource-class>org.apache.derby.jdbc.EmbeddedXADataSource</xa-datasource-class> -->
 <xa-datasource-class>org.jboss.test.cmp2.advanced.ejb.DerbyXADataSource</xa-datasource-class>
 <xa-datasource-property name="URL">derby/hatest1|derby/hatest2</xa-datasource-property>
 <xa-datasource-property name="User">sa</xa-datasource-property>
 <xa-datasource-property name="Password"></xa-datasource-property>
 <xa-datasource-property name="CreateDatabase">create</xa-datasource-property>

 <url-delimeter>|</url-delimeter>
 <check-valid-connection-sql>VALUES CURRENT TIMESTAMP</check-valid-connection-sql>

 <!-- corresponding type-mapping in the standardjbosscmp-jdbc.xml (optional) -->
 <metadata>
 <type-mapping>Derby</type-mapping>
 </metadata>
 </ha-xa-datasource>

As you can see, it's basically the same as in the local case, i.e. URL contains a list of URLs and url-delimeter specifies the delimeter.

The test is not added to the testsuite since we don't have derby in thirdparty (yet).

Implementation notes. XAManagedConnectionFactory creates XADataSources lazily, i.e. not at deployment time but on the first request. In the HA implementation, I create XADataSources at deployment. Is this ok? Any special reason for lazy instantiation except for "i don't use it, so i don't need it to be created"?

[aloubyansky]

Adrian, what else should be done here for the 4.0.2 release?
The XA HA stuff is only in HEAD currently.

[adrian.brock]

You can't just have

<url-delimiter>

You need to identify which of the xadatasource properties identifies the server,
e.g. for mssql it is "ServerName"

<url-property>ServerName</url-property>
<url-delimiter>|</url-delimiter>

This is not an issue for local since the connection url is always in the same config.

[adrian.brock]

The XADataSource should just be a factory, it should be the connections that are the
heavy weight objects.
I'd guess this is the usual "You don't pay for what you don't use"
philosophy that JBoss normally does.

It looks like the only change you made to the existing stuff was to make
the xa props a protected field, so I don't see why it would be problem backporting
to JBoss4 (assuming you didn't break the stylesheet :-).

Do you have it documentated somewhere on the WIKI (as experimental)?
http://www.jboss.org/wiki/Wiki.jsp?page=JBossJCA
and add a subtask here:
http://jira.jboss.com/jira/browse/JBAS-1597