JBoss AS7 / PostgreSQL and XA transactions

[andhess]

Hi

I've have a problem that I can't figure out since weeks. So this is my last attempt before finally giving up.

In my company we have several JBoss AS7 /EAP6 servers. We always use PostgreSQL as a database. The issue is as follows:

1. Data is read from different tables using JPA and is marked as transferred

2. The entities are added to a JMS / HornetQ queue

3. An MDB picks up the entity

4. Based on the class of the entity a POJO gets loaded (e.g. when the entity class is TestClass -> the entity will be passed to a TestClassHander POJO)

5. The handler class does whatever it does

6. The entity gets updated with em.merge(..)

7. When an error occurred the marker on the original entity gets removed so it will be retried

All works fine except the performance. PostgreSQL always does PREPARE TRANSACTION / COMMIT PREPARED which is slow (approx 100 messages / sec).

I have checked that all EJB/CDI classes that are used during the processing (inside the POJOs) use the same transaction context. No other XA resources are involved as well.

To check the general workflow I have tried to just reset the marker. That is extremely fast (approx 1000 messages per sec) and the database does just a COMMIT.

I'm out of ideas! Any help is appreciated!

Andreas

[jbertram]

I'm not exactly sure what the problem here is, but I'll give you a run-down of what I see from a transaction perspective.

As I see it, you likely have 2 independent, XA transactions executing during this 7 step process.  The first XA TX involves steps 1 & 2.  The second XA TX involves steps 3-7.  As I'm sure you're aware, XA uses a 2-phase commit protocol involving a "prepare" phase and a "commit" phase (just like you're seeing).

1. Data is read from different tables using JPA and is marked as transferred

2. The entities are added to a JMS / HornetQ queue

Most likely your using XA resources for both the database and JMS work which means both will be enlisted into the same transaction and committed atomically via prepare/commit.

3. An MDB picks up the entity

4. Based on the class of the entity a POJO gets loaded (e.g. when the entity class is TestClass -> the entity will be passed to a TestClassHander POJO)

5. The handler class does whatever it does

6. The entity gets updated with em.merge(..)

7. When an error occurred the marker on the original entity gets removed so it will be retried

When the container invokes an MDB's onMessage() by default it will start a new transaction.  Immediately the container will enlist the JMS broker into the transaction so that the consumption of the message by the MDB will be performed atomically with any other transaction work done while processing the message (e.g. database inserts, updates, etc.).  As before, you are likely doing your database work here with an XA resource so that it will be enlisted into the transaction started by the container.  When the MDB is finished executing the transaction will be committed, and since there is > 1 resource enlisted in the transaction the transaction manager will use a 2-phase commit protocol.

Based on the evidence you've provided I believe this is a plausible explanation for the behavior you're seeing.

[andhess]

Hello

You are absolutely right with your explanation, but the application works different. Let me clarify:

Regarding steps 1 & 2:

You are right, but the XA TX is not an issue here. The transaction always moves up to 1000 entities at once. Therefore the "prepare" and "commit" of the 2 phase commit is executed only once.

Regarding steps 3 until 7:

The MDB has been marked as @TransactionManagement(Bean). Each message is passed on to an SLSB with @TransactionAttribute(Requiers_new). Here the new transaction starts. Therefore I believe that there is only 1 resource

involved in the transaction.

Remark: Using a non XA datasource is not an option as well. The processing POJO can theoretically do anything therefore XA TX is possible. Only in the current implementation it is not required (and still that is what is happening).

Can I somehow list the resources inside the transaction and find what makes JBoss behave like this?

[jbertram]

I believe you can activate TRACE or DEBUG logging for "com.arjuna" to see what the transaction manager is doing.  Alternatively you could use a Byteman script to gain additional insight.  I attached an Byteman script that I've used in the past to track down problems like this.

[andhess]

Hello and thank you for your efforts, but I can't find any script attached.

[jbertram]

I'm not sure why you can't see it.  I can see it fine, even if I'm not logged in.  The file is named "xascript.txt.zip."

[andhess]

Hello, I've just wanted to share what we have finally found out about this issue:

A statistic was requested in the processing. Therefore a @Remote EJB was build where statistical information can be send. It turned out that the remote EJB invocation was added to the transaction context

even that the EJB implementation is @Transactionmanagement(BEAN). We had focused so much on the DB/Entitymanager that nobody had a look at this.

Finally after nothing made any difference the @Remote EJB invocation was removed (by a pretty desperate programmer) and the issue disappeared.

Thanks for all the help!