Got the ARJUNA016037 and ARJUNA016038 in every two minutes exactly.

[surajyonzon]

I am using the JBoss AS 7.1.0 in production with postgresql 9.1 as database. I don't know why we are getting this message on logs in every two minutes exactly.

[com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a110115:-312b2506:51132820:25bd, node_name=1, branch_uid=0:ffff0a110115:-312b2506:51132820:25c0, subordinatenodename=null, eis_name=unknown eis name >, heuristic: TwoPhaseOutcome.FINISH_OK com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@21a308a8 >

20:04:40,176 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a110115:-312b2506:51132820:25bd, node_name=1, branch_uid=0:ffff0a110115:-312b2506:51132820:25c0, subordinatenodename=null, eis_name=unknown eis name >

Please assist us about this Warning message.

[nickarls]

What does your datasource definition look like? Is this an XA datasource (and is it supposed to be)?

[surajyonzon]

Thanks for the reply.

My Datasource definition is like this one:

<datasource jndi-name="java:jboss/datasources/MYDS" pool-name="MYDS" enabled="true" use-java-context="true">

                    <connection-url>jdbc:postgresql://192.168.0.190:5432/mydb</connection-url>

                    <driver>postgresql</driver>

                    <pool>

                        <min-pool-size>2</min-pool-size>

                        <max-pool-size>30</max-pool-size>

                        <prefill>true</prefill>

                    </pool>

                    <security>

                        <user-name>username</user-name>

                        <password>password</password>

                    </security>

                    <validation>

                        <check-valid-connection-sql>SELECT 1</check-valid-connection-sql>

                    </validation>

                    <timeout>

                        <idle-timeout-minutes>5</idle-timeout-minutes>

                    </timeout>

                    <statement>

                        <track-statements>TRUE</track-statements>

                    </statement>

                </datasource>

<drivers>

     <driver name="postgresql" module="org.postgresql">

             <xa-datasource-class>org.postgresql.Driver</xa-datasource-class>

    </driver>

</drivers>

I am using PostgreSQL 9.1 with latest JDBC Driver that supports PostgreSQL 9.1.

[jbertram]

My guess is that you're enlisting some other kind of XA resource in your transaction.  Can you enumerate all the resources your using in your transactions?

[surajyonzon]

Thanks for you reply.

Ok I have some 3 datasources where are as follows:

<subsystem xmlns="urn:jboss:domain:datasources:1.0">

            <datasources>

                <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">

                    <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1</connection-url>

                    <driver>h2</driver>

                    <security>

                        <user-name>sa</user-name>

                        <password>sa</password>

                    </security>

                </datasource>

                <datasource jndi-name="java:jboss/datasources/DS_1" pool-name="DS_1" enabled="true" use-java-context="true">

                    <connection-url>jdbc:postgresql://192.168.0.190:5432/ds1</connection-url>

                    <driver>postgresql</driver>

                    <pool>

                        <min-pool-size>2</min-pool-size>

                        <max-pool-size>20</max-pool-size>

                        <prefill>true</prefill>

                    </pool>

                    <security>

                        <user-name>username</user-name>

                        <password>password</password>

                    </security>

                </datasource>

                <datasource jndi-name="java:jboss/datasources/DS_2" pool-name="DS_2" enabled="true" use-java-context="true">

                    <connection-url>jdbc:postgresql://192.168.0.190:5432/ds2</connection-url>

                    <driver>postgresql</driver>

                    <pool>

                        <min-pool-size>2</min-pool-size>

                        <max-pool-size>30</max-pool-size>

                        <prefill>true</prefill>

                    </pool>

                    <security>

                        <user-name>username</user-name>

                        <password>password</password>

                    </security>

                    <validation>

                        <check-valid-connection-sql>SELECT 1</check-valid-connection-sql>

                    </validation>

                    <timeout>

                        <idle-timeout-minutes>5</idle-timeout-minutes>

                    </timeout>

                    <statement>

                        <track-statements>TRUE</track-statements>

                    </statement>

                </datasource>

                <datasource jndi-name="java:jboss/datasources/DS_3" pool-name="DS_3" enabled="true" use-java-context="true">

                    <connection-url>jdbc:jtds:sqlserver://192.168.0.186/ds3</connection-url>

                    <driver>jtds</driver>

                    <pool>

                        <min-pool-size>2</min-pool-size>

                        <max-pool-size>20</max-pool-size>

                        <prefill>true</prefill>

                    </pool>

                    <security>

                        <user-name>username</user-name>

                        <password>password</password>

                    </security>

                </datasource>

                <drivers>

                    <driver name="h2" module="com.h2database.h2">

                        <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>

                    </driver>

                    <driver name="postgresql" module="org.postgresql">

                        <xa-datasource-class>org.postgresql.Driver</xa-datasource-class>

                    </driver>

                    <driver name="jtds" module="net.sourceforge.jtds">

                        <driver-class>net.sourceforge.jtds.jdbc.Driver</driver-class>

                        <xa-datasource-class>net.sourceforge.jtds.jdbcx.JtdsDataSource</xa-datasource-class>

                    </driver>

                </drivers>

            </datasources>

        </subsystem>

I have listed all the datasources and its relative drivers. Please help us.

[surajyonzon]

I think, I have found the solution for this. This is because of the improper shutdown of the application server when accessing the resources like database or anything.

I have a simple solution which worked for me. I found this path

/usr/bin/jboss-as-7.1.0.Final/standalone/data/tx-object-store/ShadowNoFileLockStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction

Inside AtomicAction I found some transaction related files and after removing those files. This issue was automatically solved. Hope it will help someone too

[maxant]

JBoss is telling you that there are transactions that did not complete, and it would like to complete them but cannot (it does not know how).  I don't know exactly why that is in your case.  Deleting the transaction logs like you suggest means that JBoss won't continue to try to complete the transactions, but that isn't ideal, since your system is now in a theoretically inconsistent state.  Next to the ARJUNA016037 log message there might be another log message that contains the JNDI name of the resource which is causing the problem.

[adrianoschmidt]

Hello Suraj Tamang,

I'm having the same problem .. After clearing the "data" folder, the error not happened again?

Kind Regards,

Adriano Schmidt

www.localhost8080.com.br

[surajyonzon]

Hi Adriano, Thanks for the reply, okay I will try it once by clearing the "data" directory.

[mnovak]

Those warnings can show in situation when Transaction Manager calls commit(xid) on XAResource (for DB) and this call does not return. This can happen for example because of shutdown/kill of database or EAP server.

What happens is that this commit() call gets in this case to DB. DB commits its transaction and forgets corresponding xid which was passed by Transaction Manager. Transaction manager does NOT remove transaction with given xid from journal because commit() call has not returned and tries to recover it during periodic recovery every 2 min. Problem is that DB has forgot xid and Transaction Manager can't find the XAResource which would know this xid and prints those warnings.

[kanagurk]

Hello Miroslav Novak,

I am getting the same issue. Can you please suggest me how to fix the issue.?

Mail id : [e-mail]

[ochaloup]

Hi Kanagaraj,

the reason why this warning is shown is explained in Miroslav's answer. The point is to understand how the XA transaction work - jboss transaction manager (TM) communicates with database but both of them are autonomous entities.

One of the reason why you can see this warning could be that  jboss app server was killed/crashred and some transaction was active at that time then it could happen that TM sends commit command to database and database does the commit on its side. The server crashes and the response is not delivered back to TM. TM holds information about  transaction is in progress in its own datastore (transaction object store). When the app server is crashed then those information is left in that store. At the start of app server TM starts special thread named "Periodic Recovery" . It's activated by default each 2 minutes. The task of this service is to recover crashred transactions. The recovery service can see that there are records about unfinished transaction in transaction object store and it tries to find who owns them. It tries to contact all XA resources configured in app server and ask them whether they own transaction with that specific Xid (identifier of xa transaction). If all resources respond that they do not know about that then this message is shown.

In this case data consistency is not threatened in any way as everything was already commited just TM didn't manage to clean its store as it was killed.

If this is the case then it's ok to remove such transaction record from object store. You can do it in "hard" way as it's mentioned in comments above. Or you can you recommended way by using cli commands and remove just that particular transaction.

First refresh/load data from log store to cli

/subsystem=transactions/log-store=log-store/:probe()

You can list all transactions from object store

ls /subsystem=transactions/log-store=log-store/transactions

And finally remove the problematic one from the store (xid could be found in the warning message as tx_uid)

/subsystem=transactions/log-store=log-store/transactions=<XID>:delete()

/subsystem=transactions/log-store=log-store/transactions=0\:ffff0a110115\:-312b2506\:51132820\:25bd:delete()

But!

If you would like to do so on production system be sure that it's this described case. If you can see more warning, errors below those ARJUNA016037 and ARJUNA016038 warnings then the reason is different and should be handled appropriatelly.

A note: the XA resource is not just database but for example JMS queue (topic). It means that the warning could not be touching just database but as well sending messages to a queue. E.g. for the specific warning mentioned at the start of this thread (Got the ARJUNA016037 and ARJUNA016038 in every two minutes exactly.) it seems that the warning happens because of some MDB. I assume so from the format of the warning. if item eis name contains value unknown (eis_name=unknown eis name) then it indicates so (see https://bugzilla.redhat.com/show_bug.cgi?id=1010242). Normally property eis_name would contain JNDI name of the resource to recover.

I hope that my answer is comprehensible. In case give me know and I will try describe better.

[tomjenkinson]

As Ondra says - be very very wary of removing the object store, if you are getting the message due to a misconfiguration then you will have inconsistent state.

[kanagurk]

Hello Ondra,

Thanks a lot for your reply. It was very useful to understand about the issue clearly.

I will keep on post you if i have any concern.