Wildfly Jboss crashes with prepared transactions
a1exander1907 Jul 28, 2014 2:17 AMI new to Wildfly(Jboss) and I have trouble on my new job with it.
Server write a lot of warnings
2014-07-25 13:11:51,151 WARN [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: org.postgresql.xa.PGXAException: Error rolling back prepared transaction
at org.postgresql.xa.PGXAConnection.rollback(PGXAConnection.java:420)
at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.rollback(XAManagedConnection.java:346)
at org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.rollback(XAResourceWrapperImpl.java:186)
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:786) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:692) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:431) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:212) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
Caused by: org.postgresql.util.PSQLException: ERROR: prepared transaction belongs to another database
Hint: Connect to the database where the transaction was prepared to finish it.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:331)
at org.postgresql.xa.PGXAConnection.rollback(PGXAConnection.java:406)
... 8 more
and after some of this warnings it gets error
2014-07-25 13:13:00,599 ERROR [stderr] (EJB default - 4) java.io.FileNotFoundException: /opt/wildfly/standalone/data/tx-object-store/ShadowNoFileLockStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/0_ffff0a0a0944_25a40805_53d20f8e_6dabd (Too many open files)
2014-07-25 13:13:00,600 ERROR [stderr] (EJB default - 4) at java.io.FileOutputStream.open(Native Method)
2014-07-25 13:13:00,601 ERROR [stderr] (EJB default - 4) at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
2014-07-25 13:13:00,601 ERROR [stderr] (EJB default - 4) at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
2014-07-25 13:13:00,601 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.internal.arjuna.objectstore.ShadowingStore.write_state(ShadowingStore.java:560)
2014-07-25 13:13:00,602 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.internal.arjuna.objectstore.FileSystemStore.write_committed(FileSystemStore.java:134)
2014-07-25 13:13:00,602 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2248)
2014-07-25 13:13:00,602 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1481)
2014-07-25 13:13:00,603 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
2014-07-25 13:13:00,603 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
2014-07-25 13:13:00,603 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1166)
2014-07-25 13:13:00,604 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
2014-07-25 13:13:00,604 ERROR [stderr] (EJB default - 4) at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
2014-07-25 13:13:00,604 ERROR [stderr] (EJB default - 4) at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:93)
2014-07-25 13:13:00,605 ERROR [stderr] (EJB default - 4) at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
2014-07-25 13:13:00,605 ERROR [stderr] (EJB default - 4) at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:364)
2014-07-25 13:13:00,606 ERROR [stderr] (EJB default - 4) at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:241)
2014-07-25 13:13:00,606 ERROR [stderr] (EJB default - 4) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
2014-07-25 13:13:00,606 ERROR [stderr] (EJB default - 4) at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
2014-07-25 13:13:00,607 ERROR [stderr] (EJB default - 4) at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
2014-07-25 13:13:00,607 ERROR [stderr] (EJB default - 4) at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)
I read this article: http://www.postgresql.org/docs/9.2/static/sql-prepare-transaction.html
As I understand prepared transactions is transactions wich system write on hard-drive first and then commit or rollback. And there is caution about as it consumes a lot of system resourseces and may cause problems. I think that Server makes a lot of this transactions, but dont commit/rollback them and it holds system resources(as we see in warn message) and then Server fails because there are no more free resources("too many open files"). But why server do this ? Why it makes a lot of this trasnactions that consumes all resourses? And what does it mean "prepared transaction belongs to another database" ? How can it bee ? I have no idea how to solve this problem! I hope for your help ! Thanks!