1 Reply Latest reply on Aug 5, 2014 10:22 AM by Alexander Bykov

    Wildfly Jboss crashes with prepared transactions

    Alexander Bykov Newbie

      I new to Wildfly(Jboss) and I have trouble on my new job with it.

      Server write a lot of warnings

      2014-07-25 13:11:51,151 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMERR: org.postgresql.xa.PGXAException: Error rolling back prepared transaction
        at org
      .postgresql.xa.PGXAConnection.rollback(PGXAConnection.java:420)
        at org
      .jboss.jca.adapters.jdbc.xa.XAManagedConnection.rollback(XAManagedConnection.java:346)
        at org
      .jboss.jca.core.tx.jbossts.XAResourceWrapperImpl.rollback(XAResourceWrapperImpl.java:186)
        at com
      .arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.handleOrphan(XARecoveryModule.java:786) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
        at com
      .arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecoverySecondPass(XARecoveryModule.java:692) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
        at com
      .arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:431) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
        at com
      .arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:212) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
        at com
      .arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:789) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
        at com
      .arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:371) [narayana-jts-jacorb-5.0.0.Final.jar:5.0.0.Final (revision: 9aa71)]
      Caused
      by: org.postgresql.util.PSQLException: ERROR: prepared transaction belongs to another database
        Hint
      : Connect to the database where the transaction was prepared to finish it.
        at org
      .postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2161)
        at org
      .postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1890)
        at org
      .postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
        at org
      .postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:559)
        at org
      .postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:403)
        at org
      .postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:331)
        at org
      .postgresql.xa.PGXAConnection.rollback(PGXAConnection.java:406)
        
      ... 8 more


      and after some of this warnings it gets error


      2014-07-25 13:13:00,599 ERROR [stderr] (EJB default - 4) java.io.FileNotFoundException: /opt/wildfly/standalone/data/tx-object-store/ShadowNoFileLockStore/defaultStore/StateManager/BasicAction/TwoPhaseCoordinator/AtomicAction/0_ffff0a0a0944_25a40805_53d20f8e_6dabd (Too many open files)
      2014-07-25 13:13:00,600 ERROR [stderr] (EJB default - 4)  at java.io.FileOutputStream.open(Native Method)
      2014-07-25 13:13:00,601 ERROR [stderr] (EJB default - 4)  at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
      2014-07-25 13:13:00,601 ERROR [stderr] (EJB default - 4)  at java.io.FileOutputStream.<init>(FileOutputStream.java:171)
      2014-07-25 13:13:00,601 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.internal.arjuna.objectstore.ShadowingStore.write_state(ShadowingStore.java:560)
      2014-07-25 13:13:00,602 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.internal.arjuna.objectstore.FileSystemStore.write_committed(FileSystemStore.java:134)
      2014-07-25 13:13:00,602 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.arjuna.coordinator.BasicAction.prepare(BasicAction.java:2248)
      2014-07-25 13:13:00,602 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.arjuna.coordinator.BasicAction.End(BasicAction.java:1481)
      2014-07-25 13:13:00,603 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:96)
      2014-07-25 13:13:00,603 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:162)
      2014-07-25 13:13:00,603 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1166)
      2014-07-25 13:13:00,604 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:126)
      2014-07-25 13:13:00,604 ERROR [stderr] (EJB default - 4)  at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
      2014-07-25 13:13:00,604 ERROR [stderr] (EJB default - 4)  at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:93)
      2014-07-25 13:13:00,605 ERROR [stderr] (EJB default - 4)  at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:277)
      2014-07-25 13:13:00,605 ERROR [stderr] (EJB default - 4)  at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:364)
      2014-07-25 13:13:00,606 ERROR [stderr] (EJB default - 4)  at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:241)
      2014-07-25 13:13:00,606 ERROR [stderr] (EJB default - 4)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
      2014-07-25 13:13:00,606 ERROR [stderr] (EJB default - 4)  at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
      2014-07-25 13:13:00,607 ERROR [stderr] (EJB default - 4)  at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)
      2014-07-25 13:13:00,607 ERROR [stderr] (EJB default - 4)  at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43)


      I read this article: http://www.postgresql.org/docs/9.2/static/sql-prepare-transaction.html

      As I understand prepared transactions is transactions wich system write on hard-drive first and then commit or rollback. And there is caution about as it consumes a lot of system resourseces and may cause problems. I think that Server makes a lot of this transactions, but dont commit/rollback them and it holds system resources(as we see in warn message) and then Server fails because there are no more free resources("too many open files"). But why server do this ? Why it makes a lot of this trasnactions that consumes all resourses? And what does it mean "prepared transaction belongs to another database" ? How can it bee ? I have no idea how to solve this problem! I hope for your help ! Thanks!