Disable two-phase commit

[jbossja]

Hi,

Can someone please provide some help on how to disable two-phase commit?

Thanks.

[jhalliday]

Umm, don't access multiple transactional resource managers in the same transaction? :-)

Seriously, some more context would be helpful.

[jbossja]

I am using JBoss 4.2.1, with Seam 2.0, EJB 3.0, hibernate and mysql as database. I also query the database through jdbc calls in cases where hibernate queries cause performance issues.

The same transaction takes twice as much time as it used to with Seam 1.1CR2 and JBoss 4.0.5, and I read that the way JBoss handles transactions had changed with the upgrade to 4.2.1.

When executing an action which combines both hibernate queries and jdbc calls (inserting data in multiple tables in the database), the application hangs because of the two-phase commit. So, I would like to know how to disable the two-phase commit.

Thanks.

[jhalliday]

ok, so basically you have two reasons:

Your current environment is slower than your old one and your application hangs.

Let's take these one at a time.

What evidence do you have that the 2PC is responsible for the slowdown? Unless you have reconfigured the system in other ways, the old version would have performed a 2PC also, so in that respect nothing has changed. Did you actually profile the code before and after the change to see where the time is going? Since multiple components have changed it seems premature to think the slowdown is due to just one of them.

As for the application hanging because of the two-phase commit, that's very unlikely unless one of the resource managers is blocking. Please post a thread dump from the hung JVM, that should show which part of the 2PC is causing the issue. Which resource managers are involved in the transaction? You mention one (MySQL) but not the other.

[jbossja]

Thank you for your reply. I will try to explain the problem more clearly.

What evidence do you have that the 2PC is responsible for the slowdown?

In the log file, I can see the following:

2008-05-22 01:08:29,624 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - first pass <Thu, 22 May 2008 01:08:29>
2008-05-22 01:08:29,624 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] StatusModule: first pass
2008-05-22 01:08:29,624 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_3] - TORecoveryModule - first pass
2008-05-22 01:08:29,624 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.firstpass] Local XARecoveryModule - first pass
2008-05-22 01:08:39,624 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] Periodic recovery - second pass <Thu, 22 May 2008 01:08:39>
2008-05-22 01:08:39,624 DEBUG [com.arjuna.ats.arjuna.logging.arjLogger] AtomicActionRecoveryModule: Second pass
2008-05-22 01:08:39,624 DEBUG [com.arjuna.ats.txoj.logging.txojLoggerI18N] [com.arjuna.ats.internal.txoj.recovery.TORecoveryModule_6] - TORecoveryModule - second pass
2008-05-22 01:08:39,624 DEBUG [com.arjuna.ats.jta.logging.loggerI18N] [com.arjuna.ats.internal.jta.recovery.info.secondpass] Local XARecoveryModule - second pass

Did you actually profile the code before and after the change to see where the time is going?

No, but the code did not change at all. What the transaction basically does is read data from an XML file, and then insert it into multiple tables. We used the same XML file before and after the upgrade to test the application, and the time taken to complete the transaction doubled after the upgrade for smaller files while for the bigger one (with more than 10000 lines), the transaction could not complete.

My application and database (mysql) are running on the same server, and all transactions involve the same database.

I have set the property "com.arjuna.ats.jta.allowMultipleLastResources" to true in jbossjta-properties file in order to be able to run both jdbc calls and hibernate queries in the same transaction. Otherwise, I was getting transaction not active error.

Could this configuration change be causing this issue? How can I fix it?

Thanks.

[jhalliday]

>> What evidence do you have that the 2PC
>> is responsible for the slowdown?
...
> In the log file, I can see the following:
...
And your point is? You fear the printing of a few log statements is placing unacceptable load on your server? Depending on your configuration, the recovery system will wake up once every two minutes, find there is nothing for it to do and go back to sleep. You can increase the interval via the config file if you like.

> No, but the code did not change at all.

Yours may not have done, but the rest of the app server and seam did. What evidence do you have that its the 2PC *specifically* that is the issue here?

> all transactions involve the same database.

If that's the only resource manager involved you should not need to use XA at all then. With a local-tx datasource the 2PC is not noticeably more expensive than 1PC, since one of the phases is basically a null-op implemented by the LastResource. Not that you will be getting 2PC behaviour anyhow, since with only one resource involved the transaction manager will optimise it out by default.

> Could this configuration change be causing this issue?

That's rather the point, is it not? You are worried about the 2PC performance without any evidence. Based on what you have said about your configuration, it's may not even running a 2PC for the transaction.

Profile your code on the old and new server environments Then profile a microbenchmark of a few thousand basic transactions on each. If you still think the 2PC is the problem after that I'll pay attention then.

[yj4jboss]

Hi,
I have been working on this issue together with jbossja for a while. As a performance benchmark, we have the following:

We have a particular case where a huge XML file of around ten thousands lines is imported via the application. This results in around eight thousand merge and persists into the database.


Our problem is simple, earlier with Seam 1.1.0CR1 and jboss-4.0.5.GA using Seam, Hibernate and EJB3.0 and Mysql database with local-tx-datasource (which btw is still the same with Seam2), this transaction took around 3 minutes at most.

Now that we migrated to the new Seam2.0 with jboss-4.2.1.GA, the same transaction is taking much longer than we can afford. After around, 8 minutes, the system hangs with the logs jbossja mentioned above and we even get an Out of Memory Error (PermGenSpace). We already extended the heap size to 1024MB.


So what we expect generally would be how to fine tune this feature. We hoped that disabling 2PC can help us reach the end of this transaction and hence serve us as a proper benchmarking guideline. How can we ensure that we reach the end of this transaction (whatever time it takes for god sake) without the system hanging and getting OOM errors.


Also, we know whatever you are saying about 2PC makes sense, but would you be kind enough, for our peace of mind ;), to guide us through how to disable 2PC for this application.


Kind Regards,
Jankee Yogesh

[jhalliday]

> I have been working on this issue together with jbossja for a while.

During which time it seems neither of you have really thought much about how the transaction manager and JCA actually works. Pour yourself a very strong coffee and pay attention...

Transactions over database connections can work in one of two ways: directly demarcating the transaction boundaries on the JDBC Connection using the JDBC API, or indirectly using an XAResource obtained from a XAConnection.

Transaction managers work in terms of XAResources. When told to commit a transaction, they make two passes over the list of XAResources. On the first pass they tell each XAResource to prepare and on the second pass they tell them to commit, or to rollback if something has gone wrong. Hence two-phase commit (2PC). In the case of an XAResource that is part of a database driver, this requires two network round trips (prepare,commit) to the database rather than just one (commit) for a non-XA case. It also requires slightly more complex logging on the database server and a file write by the transaction manager. These factors account for the vast majority of the overhead in 2PC vs. 1PC. In return you get the guarantee that all the resource managers (databases) will have the same outcome for the transaction.

But you are not using real XA.

With local-tx datasources, you have a database that is not XA aware, but nevertheless want to drive the transaction using the transaction manager rather than by calling commit/rollback directly on the JDBC Connection(s). This requires introducing an Object, which we shall call a LastResource. It pretends to be a XAResource so that the transaction manager can drive it, but implements prepare by sending a commit to the database and commit as a null-op. This trick is normally only used to allow a single non-XA participant into a transaction with one or more other XA aware players. By ordering the non-XA one last in the sequence of resources, the chances of a problem with its commit leading to an inconsistent transaction outcome is minimized. Hence 'LastResource'.

Your particular use case is a further extension of this idea. You have multiple connections to a non-XA database, but you still want to use the transaction manager rather than calling begin/commit on each Connection one at a time, because that's a bit tedious. So you configure the system to allow multiple LastResources, accepting that this slightly increases the chances of data inconsistency in the event of a crash. Now the transaction manager's 2PC consists of calling prepare on each LastResource, each of which commits that Connection's tx to the database, then calling commit on each, which does nothing. The total number of db network round trips is N rather than 2N and there is no prepare logging in the db. There is still a transaction log write in the transaction manager since we don't currently optimise it out. But that file write is the only significant performance hit relative to walking a list of Connections and calling commit on each directly.

Now let us return to your request: how to disable 2PC. You can't. The transaction manager works only in terms of XAResources, which are inherently two phase. It knows enough to automatically do 1PC if it has only one resource in the transaction. It also has LastResource that allow you to fake 2PC behaviour at very little cost. But that's as far as it goes. If you want to strip out that last bit of overhead the transaction manager is adding, the only option is to go to no-tx datasources and drive the transactions directly using the JDBC Connection Objects.

Before you do that, profile your code. I'm willing to bet the 2PC is not a significant performance issue. It's more likely to be the flushing that takes the time. Maybe try explicitly flushing in stages rather than leaving it to be done automatically in the pre-commit stage.