11 Replies Latest reply on Jan 7, 2011 2:22 AM by Mads Moelgaard Andersen

    Entity Manager with BMT and Transaction Timeout

    Alexander Hartner Expert

      When running small loads on our application all is working fine, however under heavy concurrent load we consistently get the following exception.

      2010-02-09 12:40:56,773 FATAL [com.abc.IndexOutputStream.deliverJob] failed to lazily initialize a collection of role: com.abc.entities.ChannelJob.m_jobProperties, no session or session was closed
      org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role: com.abc.entities.ChannelJob.m_jobProperties, no session or session was closed

       

      My suspicious were that the underlying JTA transaction has timed-out and that the entity manager session was closed, so I tried increasing the global transaction timeout from 300 to 3600 in jboss-service.xml

       

         <!-- JBoss Transactions JTA -->

         <mbean code="com.arjuna.ats.jbossatx.jta.TransactionManagerService"

            name="jboss:service=TransactionManager">

            <attribute name="TransactionTimeout">3600</attribute>

            <attribute name="ObjectStoreDir">${jboss.server.data.dir}/tx-object-store</attribute>

         </mbean>

       

      However this did not resolve my problem. Shortly (between 5 to 10 minutes) after starting the transaction the same exception occured again. I am injecting a UserTransaction as well as an EntityManager into a MessageDrivenBean and then manage the transaction myself. I found some posts that suggest in this case I have to join the EntityManager to the JTA transaction using entityManger.joinTransaction() which I wasn't doing up to this point. It seemed to have had a positive impact, but I am still testing if this was really the cause of the problem.

       

      What I am looking for is some clarification on if this is really the case that I have to join the transaction and if I don't join where the timeout for the entity manager is configured. I had a look at the Hibernate documentation, but didn't find any reference to a timeout, other then on the entity cache.

       

      Ideally I would like to resolve this issue by changing my environment rather then the application.

       

      Thanks in advance

      Alex

        • 1. Re: Entity Manager with BMT and Transaction Timeout
          Alexander Hartner Expert
          Calling joinTransaction did not make any difference it seems as I am still getting the same exception.
          • 2. Re: Entity Manager with BMT and Transaction Timeout
            Alexander Hartner Expert

            After running more tests I am consistently able to reproduce this problem. Any suggestion on how I could debug the entitymanger session and the JTA transaction would help me out greatly. Are there any specific log context which I can enable to trace the cause of this issue.

             

            Any other suggestions ?

             

            Thanks in advance

            Alex

            • 3. Re: Entity Manager with BMT and Transaction Timeout
              jaikiran pai Master
              Can you post your MDB code where you are doing all this?
              • 4. Re: Entity Manager with BMT and Transaction Timeout
                Alexander Hartner Expert

                Thanks for responsing. Here is my bean code.

                 

                First the MDB which shows the injection of an entityManager, usertransaction and a datasource:

                ...

                @MessageDriven(name = "WorkListener", activationConfig = {

                        @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),

                        @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"),

                        @ActivationConfigProperty(propertyName = "destination", propertyValue = "queue/abc/WorkMessages")

                })

                @TransactionManagement(TransactionManagementType.BEAN)

                public class WorkListener  implements MessageListener {

                 

                    @PersistenceContext(unitName = "entityManager")

                    protected EntityManager entityManager;

                 

                    @Resource(name = "jdbc/abcDSNoTX")

                    DataSource datasource;

                 

                    @Resource

                    private UserTransaction userTransaction;

                 

                    public void onMessage(Message message) {

                        try {          

                            Processor processor = new Processor(entityManager, userTransaction, datasource);

                            processor.process(message);

                ....

                 

                Then further down the stack we begin a new userTransaction and load some entity via the entity manager

                 

                ...

                m_userTransaction.begin();

                Account account = m_entityManager.find(Account.class, accountId);

                ...

                 

                account.getRelatedAccounts();

                 

                However further down the stack we access a collection on the account which loads further linked entities and failes with the LazyInitializationException.

                 

                This problem only occurs under load and is most likely caused by a slow down while accessing the database. However I am not too concerned about performance at this stage, just want to know how I can debug this issue and see what happened to my entity manager session. My suspicion is that since I am using a User Transaction that the global timeout which I set to 3600 (=1 hour) is not applied. If this is the case where do you set the default transaction timeout for a user transaction or the hibernate session. Usually the first time the exception is thrown is between 20 and 30 minutes after submitting a large volume of work.

                 

                PS: I am using this code to illustrate my problem. I know the exception refers to a different class. The application consists of various different classes and posting the actual code would just make things more difficult.

                • 5. Re: Entity Manager with BMT and Transaction Timeout
                  Alexander Hartner Expert

                  I enabled the complete hibernate context at trace level and got the following.

                  2010-02-25 23:09:00,600 TRACE [org.hibernate.transaction.CacheSynchronization] transaction before completion callback

                  2010-02-25 23:09:00,600 TRACE [org.hibernate.jdbc.JDBCContext] before transaction completion

                  2010-02-25 23:09:00,601 TRACE [org.hibernate.impl.SessionImpl] before transaction completion

                  2010-02-25 23:09:00,601 TRACE [org.hibernate.ejb.AbstractEntityManagerImpl] automatically flushing session

                  2010-02-25 23:09:00,601 TRACE [org.hibernate.impl.SessionImpl] automatically flushing session

                  2010-02-25 23:09:00,601 TRACE [org.hibernate.event.def.AbstractFlushingEventListener] flushing session

                  2010-02-25 23:09:00,601 DEBUG [org.hibernate.event.def.AbstractFlushingEventListener] processing flush-time cascades

                  2010-02-25 23:09:00,601 TRACE [org.hibernate.engine.Cascade] processing cascade ACTION_PERSIST_ON_FLUSH for: com.abc.entities.Batch

                  2010-02-25 23:09:00,601 TRACE [org.hibernate.engine.Cascade] done processing cascade ACTION_PERSIST_ON_FLUSH for: com.abc.entities.Batch

                  2010-02-25 23:09:00,601 DEBUG [org.hibernate.event.def.AbstractFlushingEventListener] dirty checking collections

                  2010-02-25 23:09:00,601 TRACE [org.hibernate.event.def.AbstractFlushingEventListener] Flushing entities and processing referenced collections

                  2010-02-25 22:40:12,978 INFO [org.hibernate.ejb.Ejb3Configuration] found EJB3 Entity bean: com.abc.AuditTrailEntry

                  2010-02-25 23:09:00,601 ERROR [org.hibernate.LazyInitializationException] failed to lazily initialize a collection of role: com.abc.entities...

                  Not really sure what to look for.

                  • 6. Re: Entity Manager with BMT and Transaction Timeout
                    jaikiran pai Master
                    I would have expected the global transaction timeout value to apply for UserTransaction too. Can you try explicitly setting the timeout in your code, before starting the transaction? Like this:
                    
                    {code:java}
                    // set timeout (1 hour)
                    m_userTransaction.setTransactionTimeout(3600);
                    m_userTransaction.begin();
                    
                    Account account = m_entityManager.find(Account.class, accountId);
                    
                    ...
                    
                     
                    
                    account.getRelatedAccounts();
                    
                    {code}
                    • 7. Re: Entity Manager with BMT and Transaction Timeout
                      Alexander Hartner Expert
                      This seemed to have resolved the problem although I don't understand at all why. It seems to suggest that the entity manager has some other timeout not determined by the JTA transaction. If this is the case where can I configure it other then inside my application. I know I can set the user transaction timeout via the deployment descriptor was well as suggested in the previous post, but ideally I would like to set it globally for the entire application server, so I don't have to modify the current version of my application.
                      • 8. Re: Entity Manager with BMT and Transaction Timeout
                        Alexander Hartner Expert

                        What I found after further investigation into this issue that every time this problem occurs shortly before the following messages are reported

                        2010-03-09 14:52:58,174 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.BasicAction_58] - Abort of action id -3f57fdae:f7ad:4b965bf5:2855 invoked while multiple threads active within it.

                         

                        2010-03-09 14:52:58,174 WARN  [com.arjuna.ats.arjuna.logging.arjLoggerI18N] [com.arjuna.ats.arjuna.coordinator.CheckedAction_2] - CheckedAction::check - atomic action -3f57fdae:f7ad:4b965bf5:2855 aborting with 1 threads active!

                        They seem to suggest that the same transaction is being accessed by multiple threads, however I don't do any explicit thread creation in the application. Are there any options to diagnose the cause of this issue further. I saw the post on the WIKI relating to a similar issue

                        http://community.jboss.org/wiki/TxMultipleThreads however since I don't create any new threads I expect the cause would be a timeout. Since the error I encounter happens well before the timeout (which I set to 3600) I don't think the global timeout is taken effect to bean managed transaction. This leads me back to my question on where the default timeout setting for user managed transactions are set.

                         

                        thanks in advance

                        Alex

                        • 9. Re: Entity Manager with BMT and Transaction Timeout
                          Mads Moelgaard Andersen Newbie

                          Hi Alexander

                           

                          Did you ever get an answer to your question?

                           

                          I have been facing a similar problem. We have an MDB without transaction support that calls a SLSB with BMT. For a long time we've been struggling with the SLSB just seeming to stop execution without throwing any exceptions or otherwise indicating an error. The method called on the SLSB results in sql that last for more than an hour.

                           

                          At first we had JDBC transactions handle the commit and it seemed to work for the sql but for some reason the SLSB just seemed to stop its execution. Then we read a post indicating that JDBC transacdtions weren't that reliable and this would certainly account for a driver deciding to close a connection without a warning. Consequently we switched to JTA

                           

                          Althout we programatically set TransactionTimeout to zero, equalling no timeout; on commit we caught arjuna exception indicating the connection to be inactive. After fiddling with the global JTA and setting it to 7200 seconds the long running sql statement executes and commits without exceptions.

                           

                          This strongly suggests that the global transaction timeout in jboss-service.xml not only sets the default timeout for all JTA transactions also sets the value as the upper limit for all JTA transactions, regardless of what has been programatically set.

                           

                          There must be someone out there that knows the truth and I would very much like to know what's right and wrong to this matter.

                           

                          Best regards

                          Mads M Andersen

                          • 10. Re: Entity Manager with BMT and Transaction Timeout
                            Alexander Hartner Expert

                            No I haven't been able to figure this one out. I also can't find any reference to 10 minutes, 600 seconds or 600000 milliseconds anywhere in any of the jboss configuration files. I am starting to believe this limit of 10 minutes is hard wired for session beans.

                             

                            Kind regards

                            Alex

                            • 11. Re: Entity Manager with BMT and Transaction Timeout
                              Mads Moelgaard Andersen Newbie

                              Hi

                              Actually it turned out that just raising the transaction timeout didn't solve the problem. Though it supplied a little more information and hinted us in the right direction. It turns out that our firewall terminates the connection when inactive for more than an hour. Althoug the sql was running it seems inactive to the firewall. After turning on the DB's "alive" signalling the problem disappeared.

                               

                              Best regards

                              Mads