recovery of multiple, hot deployed xa-datasources

[thomasgo]

Hi,

Our application needs to use two datasources: an application specific one and and a shared one. The shared datasource can, as its name indicates, be shared by multiple instances of our application.

To ensure correct behavior of transactions we'd like to use xa-datasources. However, it seems like JBoss 4.2.3.GA, wich we are using, does not provide working recovery mechanisms out-of-the-box.

So I followed the instructions of this thread: http://management-platform.blogspot.com/2008/11/transaction-recovery-in-jbossas.html and the linked documents. I took the source code of AppServerJDBCXARecovery.java attached to JBTM-441, built a jar and deployed it with the necessary entries in jbossjta-properties.xml.

So far so good, it seems to work as expected.

However, we will be deploying further instances of our application on the same server, which means there will be additional datasources.

Performing a hot deployment would not be possible in this case, since the recovery module is only initialized once during startup and thus wouldn't recognize the additional datasources. Additionally, we'd like to not make our application depend too much on server config, i.e. we'd like no to add application specific properties (like the datasource jndi name) to jbossjta-properties.xml.

So the first question is this: Is there an easy way to provide recovery for all xa-datasources that are deployed or will be deployed in one server instance?

As far as I can see, it should be possible to somehow modify AppServerJDBCXARecovery to be able to handle multiple datasources and connections and detect new datasources. However, I'm not an expert in that area and due to time restrictions it would be very risky to try and see if I can get it working.

Thus my second question: If anyone has already solved that problem or has any suggestions on a solution, would you mind to share?

To work around the problem for the time being, we'll revert to local-tx datasources, wich means we have to create a new (sub) transaction when we work on the second datasource. However, this bears some risks with respect to consistency, e.g. when the inner transaction succeeds but the outer transaction fails.

Here's my third, and slightly unrelated, question: Is using multiple transactions for accessing multiple databases in one logical transaction (i.e. one user operation) a sensible workaround?

Thanks in advance for the endurance of reading my quite lengthy post and even more thanks for your answers.

Regards,

Thomas

[jhalliday]

> Our application needs to use two datasources: an application specificone and and a shared one. The shared datasource can, as its nameindicates, be shared by multiple instances of our application.

From the point of view of the container architecture, datasources don'tbelong to applications. Even when only one app uses a datasource, it'snot considered part of that app. Datasources are deployed at serverlevel and then wired to applications. Hence the normal use case is thatall ds are deployed at server startup, even if apps are deployed orredeployed later. Where different copies of the app needs different datacontent, it's more usual to use different db user accounts or hibernatefilters to achieve that, rather than different datasources.

> Is there an easy way to provide recovery for all xa-datasources that are deployed or will be deployed in one server instance?

No. But there are some hard ways: write a recovery module that will periodically scan the JNDI namespace or AS metadata for new datasources and run recovery for them, or change the datasource deployer so it automatically registers newly deployed datasources with the recovery system. The latter is the approach we are moving to for AS6. It should have been in place years ago, but the JCA guys don't view it as a priority

> If anyone has already solved that problem or has any suggestions on a solution, would you mind to share?

There is assorted discussion on the forums and I have some prototype code for both the above approaches, but nothing good enough to use out the box.

> Is using multiple transactions for accessing multiple databases in one logical transaction (i.e. one user operation) a sensible workaround?

No. As you say, it has different failure semantics.

[adinn]

ThomasGo wrote:

So the first question is this: Is there an easy way to provide recovery for all xa-datasources that are deployed or will be deployed in one server instance?

As far as I can see, it should be possible to somehow modify AppServerJDBCXARecovery to be able to handle multiple datasources and connections and detect new datasources. However, I'm not an expert in that area and due to time restrictions it would be very risky to try and see if I can get it working.

Well, it is definitely possible. The question is how difficult is it to do and what is the best way to achieve the capability you need..

It would probably be best to dynamically register a new instance of AppServerJDBCXARecovery for each new datasource that is deployed. This would only require changing XARecoveryModule so that it allows dynamic registration and  unregistration of XAResourceRecovery instances. The XTS recovery module already provides a mechanism like this allowing XTS web services to register and unregister web service-specific recovery modules at runtime so this model has been proven to work.

In XTS the deployed artefact is usually a war (or an ear containing a war) which uses a listener to do the registration and unregistration. See the XTS demo app for an example of how this works. You would need to provide some way of triggering the register/unregister when you hot-deploy/undeploy your app.

ThomasGo wrote:

Here's my third, and slightly unrelated, question: Is using multiple transactions for accessing multiple databases in one logical transaction (i.e. one user operation) a sensible workaround?

Well, as you say, it risks loss of ACID properties when the local TX works but the global TX fails. It really depends upon whether your applciation can stomach that loss, either temproraily until another transaction compensates it or until human intervention comes to the rescue. There is no answer here which does not depend upon the application semantics.

[thomasgo]

> Our application needs to use two datasources: an application specificone and and a shared one. The shared datasource can, as its nameindicates, be shared by multiple instances of our application.

From the point of view of the container architecture, datasources don'tbelong to applications. Even when only one app uses a datasource, it'snot considered part of that app. Datasources are deployed at serverlevel and then wired to applications. Hence the normal use case is thatall ds are deployed at server startup, even if apps are deployed orredeployed later. Where different copies of the app needs different datacontent, it's more usual to use different db user accounts or hibernatefilters to achieve that, rather than different datasources.

You're right, the datasources are deployed independently of our application. However, if we deploy another "copy" of our application, we need to deploy another datasource as well, although it will be done only once. The need for a different datasource comes from the fact, that the copies of the application are not 100% identical. They might contain slightly different data and thus the database might have a slightly different layout (except for the shared database).

Initially we planned a multi-client system that would support multiple views on the data for the different clients in one application instance. However, since the clients might require bigger changes in the datastructures, we decided to split the app into one instance/copy per client.

> Is there an easy way to provide recovery for all xa-datasources that are deployed or will be deployed in one server instance?

No. But there are some hard ways: write a recovery module that will periodically scan the JNDI namespace or AS metadata for new datasources and run recovery for them, or change the datasource deployer so it automatically registers newly deployed datasources with the recovery system. The latter is the approach we are moving to for AS6. It should have been in place years ago, but the JCA guys don't view it as a priority

> If anyone has already solved that problem or has any suggestions on a solution, would you mind to share?

There is assorted discussion on the forums and I have some prototype code for both the above approaches, but nothing good enough to use out the box.

Glad to hear it will eventually make it into JBossAS. Unfortunately it will take its time until we can use it. Would you mind sharing part of your prototype code for the automatic registering as a base to build upon?

[thomasgo]

adinn schrieb:

ThomasGo wrote:

So the first question is this: Is there an easy way to provide recovery for all xa-datasources that are deployed or will be deployed in one server instance?

As far as I can see, it should be possible to somehow modify AppServerJDBCXARecovery to be able to handle multiple datasources and connections and detect new datasources. However, I'm not an expert in that area and due to time restrictions it would be very risky to try and see if I can get it working.

Well, it is definitely possible. The question is how difficult is it to do and what is the best way to achieve the capability you need..

It would probably be best to dynamically register a new instance of AppServerJDBCXARecovery for each new datasource that is deployed. This would only require changing XARecoveryModule so that it allows dynamic registration and  unregistration of XAResourceRecovery instances. The XTS recovery module already provides a mechanism like this allowing XTS web services to register and unregister web service-specific recovery modules at runtime so this model has been proven to work.

In XTS the deployed artefact is usually a war (or an ear containing a war) which uses a listener to do the registration and unregistration. See the XTS demo app for an example of how this works. You would need to provide some way of triggering the register/unregister when you hot-deploy/undeploy your app.

Thanks for the info, I'll have a look into XTS. I assume it's part of JBossTS, right?

ThomasGo wrote:

Here's my third, and slightly unrelated, question: Is using multiple transactions for accessing multiple databases in one logical transaction (i.e. one user operation) a sensible workaround?

Well, as you say, it risks loss of ACID properties when the local TX works but the global TX fails. It really depends upon whether your applciation can stomach that loss, either temproraily until another transaction compensates it or until human intervention comes to the rescue. There is no answer here which does not depend upon the application semantics.

Well, we could design the application in a way that the risk is minimized and ACID property losses can be coped with. However, this requires additional effort in development, testing and administration. So I'd say going the XA way would be preferable, especially if most the additional effort is required during deployment.

[jhalliday]

> Would you mind sharing part of your prototype code for the automatic registering as a base to build upon?

It wouldn't do you much good, as it relies on the dynamic registration of recovery code and that's not supported in AS 4.2.x  It also patches some app server internal classes, which is tolerable when you work for the app server vendor, but probably not a great approach for end users :-)

For the older version of the app server, your best bet is to write a single generic XAResourceRecovery impl that gets registered at server startup and which exposes an API through which you can dynamically register and unregister new datasources as they are deployed and undeployed. Then you need some sort of lifecycle hook to invoke that registration. If you really are using one ds per app then a servlet app lifecycle listener or such would probably do the trick, or you can take a more AS specific approach and hook into the app server component lifecycle model directly. good luck.

[thomasgo]

Thanks. I'll try that.