8 Replies Latest reply on Nov 3, 2004 8:56 PM by natebowler

    TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA integr

    natebowler Newbie

      I've made several posts on different threads, but none seem to get any attention. I've scoured the docs, hit TopLink groups, and personally emailed posters from many of the dead-end threads on these forums. I'm getting nowhere.

      The fact is, TopLink and JBoss JTS integration does not work. The problem is this.

      TopLink has custom handlers for the two-phase commit protocol. In the beforeCompletion() event, it issues the SQL against the database.

      In the afterCompletion() event, it closes the connection, returns it to the pool, and merges the changes with the cache.

      The problem is, in the afterCompletion() event, JBoss throws this exception when TopLink attempts to close the connection:

      ERROR [ThreadPoolWorker[1]]: Error while closing connection handle!
      org.jboss.resource.JBossResourceException: Error in delist!; - nested throwable: (java.lang.IllegalStateException: Alrea
      dy committed. TransactionImpl:XidImpl[FormatId=257, GlobalId=kungfu/11, BranchQual=, localId=11])
      at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2
      .java:102)
      at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.delist(TxConnectionManager
      .java:487)
      at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.connectionClosed(TxConnect
      ionManager.java:521)
      at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.closeHandle(BaseWrapperManagedConnection.java:30
      0)
      at org.jboss.resource.adapter.jdbc.WrappedConnection.close(WrappedConnection.java:116)
      .....


      I've tried numerous workarounds: closing the connection in beforeCompletion() (results in an even worse error) and not closing the connection (results in errors obtaining new connections).

      I've also looked at the JBoss source code and made changes in hopes that I could fix this and post it to the group. None of these has worked.

      Can someone from the JBoss team comment on this? Is this even an important issue for the JBoss platform? Is there any knowledge of this working successfully anywhere?

      If anyone has gotten this to work (POJO TopLink/SessionBean/JTS), please let me know.

      Nate

        • 1. Re: TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA in
          Scott Stark Master

          We don't have toplink and don't support its usage so it would be better to create a bug report on sourceforge with a service that illustrates the interaction with the JTA interface toplink is attempting. If its integrating via the standard jca contract then we are interested in what the problem is. If its not, then its a question of the allowed operations from within the JTA callbacks.

          http://sourceforge.net/tracker/?group_id=22866&atid=376685

          • 2. Re: TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA in
            Adrian Brock Master

            Did you even "READ THIS FIRST"?

            Posting "ME TOO" will just get you ignored. Unless your post
            helps the original poster nobody is interested.

            Sending private e-mails just makes you menace.

            I recommend you "READ THIS FIRST" so you can see what
            information we need to help you solve this problem.
            A single stacktrace does not a bug report make.

            How many times do I have post this response before it sinks in?

            • 3. Re: TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA in
              Adrian Brock Master

              The actual error message says it is trying to delist the XAResource
              after it has been committed.
              I don't see how this is possible unless there is some bizarre problem
              with the ordering of the transaction synchronizations.
              I did make some changes recently to avoid such a problem.
              These were to remedy the problems you posted "ME TOO" on.
              The fix should be in 3.2.6

              One improvement I can think of by looking at the code is
              TxConnectionManager.TxConnectionEventListener.delist() could have the following change:

              - if (currentTx.getStatus() != Status.STATUS_NO_TRANSCATION)
              +if (org.jboss.tm.TxUtils.isActive(currentTx))
              


              Which will stop it trying to suspend the connection from the transaction
              after the commit process has started.

              This is more of a sanity check than a fix, since it looks like there
              is some other ordering problem in the transaction synchronizations.

              • 4. Re: TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA in
                natebowler Newbie

                This is going to sound funny, but I think you've fixed it.... I made the change you outlined in the source and recompiled JBoss 4.0.0.

                I don't get the error anymore. The whole test suite for my app just passed including the XA tests. Connections appear to be returning to the pool, etc.

                Although the change may seem minor, it seems to directly address the issue I was having getting TopLink to work with JBoss. The transaction status was in conflict with TopLink's action to close the connection.

                I don't understand the JBoss source well enough to know what the overall impact is. I'm obviously going to spend more time with it, but it is very encouraging.

                I think you're onto the problem, is there anything I can do to further test this issue? Any chance of this making it into the next JBoss 4.0.x version?

                Nate

                • 5. Re: TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA in
                  Adrian Brock Master

                  I want to understand why you are getting the problem in
                  the first place.

                  The code change simply sidesteps the issue. It matches
                  the logic in JCA with the check done in JTA.
                  If JTA does not allow it, there is no point JCA trying it.

                  But why do you still have a currentTx when you try to close the
                  connection? What is the relative ordering of the afterCompletions
                  and what is the value of {is/was}trackByTx?

                  • 6. Re: TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA in
                    natebowler Newbie

                    I don't believe there is still a currentTx in afterCompletion(). The incoming status to afterCompletion is 3 (STATUS_COMMITTED).

                    A code snippet from the afterCompletion(int) handler that TopLink provides is as follows (hope this is helpful):

                    public void afterCompletion(int status) {
                    UnitOfWork unitOfWork = getUnitOfWork();
                    if (unitOfWork != null) {

                    if ( status == Status.STATUS_COMMITTED) {
                    // THIS IS WHERE THE ERROR OCCURS!
                    // true=="committed"; true=="jts transaction"
                    unitOfWork.afterTransaction(true, true);

                    if(unitOfWork.isMergePending()) {
                    try {
                    unitOfWork.mergeClonesAfterCompletion();
                    getSession().decrementMergesPending();
                    } catch (RuntimeException rtEx) {
                    // Option to dump exception stack trace
                    if (shouldDumpExceptionsFromCallbacks()) {
                    rtEx.printStackTrace();
                    }
                    throw rtEx;
                    }
                    }
                    } else {
                    // false=="rolled back"; true=="jts transaction"
                    unitOfWork.afterTransaction(false, true);
                    }

                    unitOfWork.release();
                    ((AbstractExternalTransactionController) getSession().getExternalTransactionController()).
                    removeActiveUnitOfWork(this.getGlobalTransaction());
                    }
                    }


                    As I mentioned before,the erroris thrown from JBoss after the TopLink call to UnitOfWork.afterTransaction(boolean, boolean). This is the stack trace:


                    org.jboss.resource.JBossResourceException: Error in delist!; - nested throwable: (java.lang.IllegalStateException: Already committed. TransactionImpl:XidImpl[FormatId=257, GlobalId=kungfu/13, BranchQual=, localId=13])
                    at org.jboss.resource.connectionmanager.BaseConnectionManager2.rethrowAsResourceException(BaseConnectionManager2.java:102)
                    at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.delist(TxConnectionManager.java:487)
                    at org.jboss.resource.connectionmanager.TxConnectionManager$TxConnectionEventListener.connectionClosed(TxConnectionManager.java:521)
                    at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.closeHandle(BaseWrapperManagedConnection.java:300)
                    at org.jboss.resource.adapter.jdbc.WrappedConnection.close(WrappedConnection.java:116)
                    at oracle.toplink.internal.databaseaccess.DatabaseAccessor.closeDatasourceConnection(DatabaseAccessor.java:326)
                    at oracle.toplink.internal.databaseaccess.DatasourceAccessor.closeConnection(DatasourceAccessor.java:346)
                    at oracle.toplink.internal.databaseaccess.DatasourceAccessor.afterJTSTransaction(DatasourceAccessor.java:73)
                    at oracle.toplink.threetier.ClientSession.afterTransaction(ClientSession.java:89)
                    at oracle.toplink.publicinterface.UnitOfWork.afterTransaction(UnitOfWork.java:1765)

                    This is close-source, but it appears all that is happening is the connection is simply being closed and returned to the connection pool. After this operation, TopLink merges changes with the cache and continues cleanup.

                    If I remove this line of code, I begin to get a different exception in beginTransaction about Connections not being closed correctly..then connections run out. I'm not sure what options I have to make this work correctly.

                    My hope was that this is a case where the JTS spec is not clear and assumptions being made on the Toplink/JBoss side are in conflict.

                    Would more code samples and/or a simply binary/ear file help?

                    Nate

                    • 7. Re: TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA in
                      natebowler Newbie

                      BTW,

                      The last thread I started in the TopLink support group can be found here.

                      http://forums.oracle.com/forums/thread.jsp?forum=48&thread=273031&tstart=0&trange=30

                      Their initial response from the Oracle group is that a close() operation should not thrown an exception if already closed per the JTS spec.

                      Let me know if there is additional information I need to provide.

                      Nate

                      • 8. Re: TopLink (9.0.4-10.1.3) and JBoss (3.2.3-4.0.0) JTS XA in
                        natebowler Newbie

                        I think I have found a fix. After looking through the source, I noticed another way to avoid that call was if isTrackByTx() is true.

                        So I added:

                        <track-connection-by-tx>true</track-connection-by-tx>

                        to my mssql-xa-ds.xml file. And, it appears to have addressed the issue.

                        From reading the docs, the only thing I can gather this does is hand control of returning connections back to the pool over to the app server. And that the app server will do this after the transaction is completed.

                        Is there anything else I should be aware of with this setting?