6 Replies Latest reply on Mar 24, 2013 6:01 AM by surajyonzon

    Database connection lost after sometimes and reconnects only after 5 minutes.

    surajyonzon

      12:46:00,001 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a110115:628ddb22:5108b2b5:2aa26 in state  RUN

      12:46:00,008 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a110115:628ddb22:5108b2b5:2aa2c in state  RUN

      12:46:00,010 WARN  [com.arjuna.ats.jta] (Transaction Reaper Worker 9) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a110115:628ddb22:5108b2b5:2aa26, node_name=1, branch_uid=0:ffff0a110115:628ddb22:5108b2b5:2aa29, subordinatenodename=null, eis_name=unknown eis name > (org.jboss.ejb.client.EJBClientManagedTransactionContext$ResourceImpl@4a30c20c) failed with exception code -: java.lang.IllegalStateException: No EJB client context is available

          at org.jboss.ejb.client.EJBClientContext.requireCurrent(EJBClientContext.java:233)

          at org.jboss.ejb.client.EJBClientManagedTransactionContext$ResourceImpl.rollback(EJBClientManagedTransactionContext.java:234)

          at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:345) [jbossjts-4.16.2.Final.jar:]

          at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2874) [jbossjts-4.16.2.Final.jar:]

          at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2853) [jbossjts-4.16.2.Final.jar:]

          at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1616) [jbossjts-4.16.2.Final.jar:]

          at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:118) [jbossjts-4.16.2.Final.jar:]

          at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:217) [jbossjts-4.16.2.Final.jar:]

          at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370) [jbossjts-4.16.2.Final.jar:]

          at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78) [jbossjts-4.16.2.Final.jar:]

       

       

      12:51:30,873 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 6) SQL Error: 0, SQLState: 08006

      12:51:30,878 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 6) An I/O error occured while sending to the backend.

      12:51:30,885 INFO  [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (EJB default - 6) HHH000106: Forcing container resource cleanup on transaction completion

      12:51:30,895 INFO  [CrudService] (EJB default - 6) MESSAGE: org.hibernate.exception.JDBCConnectionException: An I/O error occured while sending to the backend.

      12:51:30,904 SEVERE [CrudService] (EJB default - 6) Error while finding..

      12:51:30,908 WARN  [com.arjuna.ats.arjuna] (EJB default - 6) ARJUNA012077: Abort called on already aborted atomic action 0:ffff0a110115:628ddb22:5108b2b5:2aa2c

      12:51:30,916 ERROR [org.jboss.ejb3.invocation] (EJB default - 6) JBAS014134: EJB Invocation failed on component

        • 1. Re: Database connection lost after sometimes and reconnects only after 5 minutes.
          jaikiran

          Suraj, welcome to the forums.

           

          Your post doesn't have any relevant context information and it's hard to tell what's wrong. Please post the relevant details including the server version, the application packaging and more about when this exception occurs.

          • 2. Re: Database connection lost after sometimes and reconnects only after 5 minutes.
            surajyonzon

            Thanks for your reply.

             

            Actually, I got this warning message after every 40 - 50 minutes.

             

            First Warning:

            12:46:00,001 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a110115:628ddb22:5108b2b5:2aa26 in state  RUN

            12:46:00,008 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0a110115:628ddb22:5108b2b5:2aa2c in state  RUN

            12:46:00,010 WARN  [com.arjuna.ats.jta] (Transaction Reaper Worker 9) ARJUNA016045: attempted rollback of < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a110115:628ddb22:5108b2b5:2aa26, node_name=1, branch_uid=0:ffff0a110115:628ddb22:5108b2b5:2aa29, subordinatenodename=null, eis_name=unknown eis name > (org.jboss.ejb.client.EJBClientManagedTransactionContext$ResourceImpl@4a30c20c) failed with exception code -: java.lang.IllegalStateException: No EJB client context is available

                at org.jboss.ejb.client.EJBClientContext.requireCurrent(EJBClientContext.java:233)

                at org.jboss.ejb.client.EJBClientManagedTransactionContext$ResourceImpl.rollback(EJBClientManagedTransactionContext.java:234)

                at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:345) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2874) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:2853) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1616) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:118) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:217) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:370) [jbossjts-4.16.2.Final.jar:]

                at com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78) [jbossjts-4.16.2.Final.jar:]

             

            After 5 mins I got this error SQL message.

            Disconnection Message:

            12:51:30,873 WARN  [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 6) SQL Error: 0, SQLState: 08006

            12:51:30,878 ERROR [org.hibernate.engine.jdbc.spi.SqlExceptionHelper] (EJB default - 6) An I/O error occured while sending to the backend.

            12:51:30,885 INFO  [org.hibernate.engine.jdbc.internal.LogicalConnectionImpl] (EJB default - 6) HHH000106: Forcing container resource cleanup on transaction completion

            12:51:30,895 INFO  [CrudService] (EJB default - 6) MESSAGE: org.hibernate.exception.JDBCConnectionException: An I/O error occured while sending to the backend.

            12:51:30,904 SEVERE [CrudService] (EJB default - 6) Error while finding..

            12:51:30,908 WARN  [com.arjuna.ats.arjuna] (EJB default - 6) ARJUNA012077: Abort called on already aborted atomic action 0:ffff0a110115:628ddb22:5108b2b5:2aa2c

            12:51:30,916 ERROR [org.jboss.ejb3.invocation] (EJB default - 6) JBAS014134: EJB Invocation failed on component

             

             

            The above warning and error messages cycles in every 30-50 minutes.

             

            The application is using JBoss as 7.1.0.Final with standalone-full.xml and PostgerSQL 9.1 as database.

             

            The application uses EBJ 3.1, Singleton Transaction Schedular of EJB 3.1 as managed by Container and other database tranasction too. JPA API Hibernate using Persistence.XML with JTA value as "true". On the front part we are using JSF 2.1.5 and Prime Faces. This is my standalone-full.xml datasource part.

             

            <datasource jndi-name="java:jboss/datasources/MYDS" pool-name="MYDS" enabled="true" use-java-context="true">

                                <connection-url>jdbc:postgresql://192.168.0.190:5432/mydb</connection-url>

                                <driver>postgresql</driver>

                                <pool>

                                    <min-pool-size>2</min-pool-size>

                                    <max-pool-size>30</max-pool-size>

                                    <prefill>true</prefill>

                                </pool>

                                <security>

                                    <user-name>username</user-name>

                                    <password>password</password>

                                </security>

                                <validation>

                                    <check-valid-connection-sql>SELECT 1</check-valid-connection-sql>

                                </validation>

                                <timeout>

                                    <idle-timeout-minutes>5</idle-timeout-minutes>

                                </timeout>

                                <statement>

                                    <track-statements>TRUE</track-statements>

                                </statement>

                            </datasource>

             

            Please help about this matter.

            • 3. Re: Database connection lost after sometimes and reconnects only after 5 minutes.
              smarlow

              Do you have a firewall between the JBoss application server and database server that could be timing out the database connections?

              • 4. Re: Database connection lost after sometimes and reconnects only after 5 minutes.
                surajyonzon

                No, I don't think so. The application server and database server are in same LAN using local IP.

                • 5. Re: Database connection lost after sometimes and reconnects only after 5 minutes.
                  sfcoy

                  I've used PostgreSQL and JBoss 7.1.x a lot and never experienced this problem.

                   

                  It smells a bit like a hardware problem, like a misbehaving switch or loose cabling. Who hasn't tried to get away with ethernet cables with broken clips?

                   

                  Is it possible to check the error count on the server and database network interfaces?

                   

                  On linux:

                   

                  {code:bash}[steve@newhope ~]$ ifconfig

                  eth0      Link encap:Ethernet  HWaddr 00:30:18:A5:A0:87 

                            inet addr:192.168.12.2  Bcast:192.168.12.255  Mask:255.255.255.0

                            inet6 addr: fe80::230:18ff:fea5:a087/64 Scope:Link

                            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

                            RX packets:2124428 errors:0 dropped:0 overruns:0 frame:0

                            TX packets:438031 errors:0 dropped:0 overruns:0 carrier:0

                            collisions:0 txqueuelen:1000

                            RX bytes:806919298 (769.5 MiB)  TX bytes:207665931 (198.0 MiB)

                            Interrupt:29 Base address:0xa000

                  {code}

                  • 6. Re: Database connection lost after sometimes and reconnects only after 5 minutes.
                    surajyonzon

                    Thanks for the reply.

                     

                    We have a dedicated server for this application. I don't think, there is some kind of hardware related problem, but I would rather say there might be some problem while performing transactions using Java Transaction API. There are lots of Transaction Reaper check  timeout stuffs going around. If you guys have any idea related, please share about it, we all would be glad for your valuable assistance.