8 Replies Latest reply on Jul 8, 2002 10:46 PM by dsundstrom

    cuncurrency problem in CMP

    mostofizadeh

      mostofizadeh:
      I run the same scenario under weblogic 7.0 and did not get the exception.

      mostofizadeh:

      I verified that Progress does support TRANSACTION_SERIALIZABLE. I also run some more tests on Progress to verify the behavior of the levels. I also run the same test scenarios against Oracle 8 and still got the same exception.

      Here are some thoughts;

      I am calling the CMP findAll (which I have declared this eql for in the descriptor, SELECT OBJECT(o) FROM Organization o), and iterating through the CMP local references and calling the getter methods in a method of the session facade. Both the session facade and CMP beans have transaction demarcation 'Required' on all of their methods. My question is, does the container execute the findAll method inside the transaction, which would make me believe that in a Serializable transaction, the remove call in thread-1 must fail since findAll is already has the lock.

      davidjencks:

      Are you sure Progress actually supports transaction isolation? For
      instance, Hypersonic does not object if you call setTxIsolation(any legal
      value) but the only isolation level supported is ReadUncommitted. I can't
      see how this is could be a problem with JBoss, but I could be missing a
      lot. If your db actually does support tx isolation levels please post this
      on the User mailing list or Database forum.

      mostofizadeh:

      I have come across an issue with jboss (the GA version) running a concurrency test. The problem is rather easy to explain.
      The test scenario is performed as a JUnit test case. I use JUnitPP to run
      the JUnit test case in iterations and concurrent threads.

      Note: I am not sure if this matters, the entity below (organization) is in a 1:m CMR. 1(organization):m(users)

      thread-1
      -----------
      method1-create organization A
      method2-read organization A
      method3-readAll (SELECT OBJECT(o) FROM Organization o), look for
      organization A in the collection and call a getter method on the reference( getId() )
      method4-find organization A
      method5-remove organization A

      thread-2
      -----------
      method1-create organization A
      method2-read organization A
      method3-readAll (SELECT OBJECT(o) FROM Organization o), look for organization A in the collection.
      note: Before calling the getter method on the reference, thread-1 calls remove (added timestamps and noticed the race condition),
      once call the getter is called the container throws:
      javax.ejb.NoSuchEntityException: Entity not found: primaryKey=A
      method4-find organization A
      method5-remove organization A

      A few notes about the transaction demarcation and isolation levels. I am
      using Progress 1.9.d database and its JDBC drivers.

      -All these method calls are made from a JNDI client to a stateless
      session
      bean (session facade)
      -The session methods all have <trans-attribute>Required</trans-attribute>
      -The session methods intern call the CMP entity methods which also have
      <trans-attribute>Required</trans-attribute>
      so the transaction starts when calling the session method.
      -At first I was not setting the transaction isolation level (what is the
      default isolation level?) Later I set it to TRANSACTION_SERIALIZABLE and still got the same
      exception. My understanding was if the findAll method and the getter methods on the references in the collection
      were called in the same transaction (with a higher isolation level), there should be not a race condition.





        • 1. Re: concurrency problem in CMP
          mostofizadeh

          Fixed a spelling

          • 2. Re: concurrency problem in CMP
            mostofizadeh

            Here is a zip file containing both source and ear files that expose the bug. I also included sql file containing the schema. I run the test in the following stack:

            Windows 2000
            SDK 1.4
            JBoss 3.0.0 (GA)
            Any database

            • 3. Re: cuncurrency problem in CMP
              dsundstrom

              I don't know much about the JCA stuff, but I throw in my $0.02 anyway. The second thread does not have to fail. I could wait for the first to finish and then remove the record. Also if you are using CMP you can turn on row-locking which will cause the cmp engine to use the SELECT FOR UPDATE syntax.

              • 4. Re: cuncurrency problem in CMP
                mostofizadeh

                Hi Dain

                I tried turning on the row-locking and still got the same result. Let me ask a few questions.

                What does the CMP engine do when I call the abstract findAll method is called? I assume it execute some sort of a 'Select * from xxtable' statement or something similar to what I declared in ejb-jar.xml for the findAll method.

                what does the CMP engine do when I call the getter methods on the references that are returned from the findAll method?

                If I call findAll and the getter methods in one method from a session bean which has a transaction demarcation of 'required', does the CMP engine in fact will execute these SQL statements within the transaction? Is this why you suggested to turn on row-locking to force the container to use 'select for update' in order to lock all rows? Does the container use 'select for update' for the findAll method or just the getters or both?

                Thanks in advance for any feedback.

                • 5. Re: cuncurrency problem in CMP
                  mostofizadeh

                  Hi Dain

                  I tried turning on the row-locking and still got the same result. Let me ask a few questions.

                  What does the CMP engine do when I call the abstract findAll method is called? I assume it execute some sort of a 'Select * from xxtable' statement or something similar to what I declared in ejb-jar.xml for the findAll method.

                  What does the CMP engine do when I call the getter methods on the references that are returned from the findAll method?

                  If I call findAll and the getter methods from one method in a session bean which has a transaction demarcation of 'required', does the CMP engine in fact execute the SQL statements within the transaction? Is this why you suggested to turn on row-locking to lock all row while calling the getter methods? Does the CMP engine use ‘select for update’ only for the findAll method or just the getter methods or both?

                  Thanks in advance for any feedback.

                  • 6. Re: cuncurrency problem in CMP
                    dsundstrom

                    > Hi Dain
                    >
                    > What does the CMP engine do when I call the abstract
                    > findAll method is called? I assume it execute some
                    > sort of a 'Select * from xxtable' statement or
                    > something similar to what I declared in ejb-jar.xml
                    > for the findAll method.

                    Not exactally, JBoss does not use Select *. The executed SQL is logged in the server.log file. You may have to increase the log level of the org.jboss.ejb.plugins.cmp category to DEBUG in the log4j.xml file.

                    > What does the CMP engine do when I call the getter
                    > methods on the references that are returned from the
                    > findAll method?

                    Depends if the data has been preloaded, which is optional. You should read the Optimized Loading chapter of the JBossCMP documentation.

                    > If I call findAll and the getter methods from one
                    > method in a session bean which has a transaction
                    > demarcation of 'required', does the CMP engine in
                    > fact execute the SQL statements within the
                    > transaction?

                    Yes

                    > Is this why you suggested to turn on
                    > row-locking to lock all row while calling the getter
                    > methods?

                    You only need to lock if you are changing the results in another thread.

                    > Does the CMP engine use ?select for update?
                    > only for the findAll method or just the getter
                    > methods or both?

                    It should. If it doesn't it is a bug.

                    • 7. Re: concurrency problem in CMP
                      mostofizadeh

                      Thanks Dain for the reply. I had read the JBossCMP document, and read it again. I only have the primary key setup to eager-load. After I sent the last post, I checked the database log and was able to see the queries, thanks for the pointer.

                      Anyway here are some thoughts and results from various tests. First of all let me say that we are very determined to go with JBoss, therefore I am very anxious to resolve this issue.

                      1- I don’t think we would want to turn row-locking on during a findAll call and iterating through the local references. Obviously we do not want to lock the entire table for a ‘select *’. As you mentioned we should only use ‘select for update’ when updating. The method that calls findAll and iterates, is a read-only method. On the other hand, if we do not use ‘select for update’ the transaction demarcation on that method is a no-op.
                      2- I tried different configurations of eager-load (pre-load all fields and not pre-load any fields), either case I get the exception.
                      3- Here is the stack trace.

                      09:25:22,992 ERROR [LogInterceptor] TransactionRolledbackException, causedBy:
                      javax.ejb.NoSuchEntityException: Entity not found: primaryKey=9977
                      at org.jboss.ejb.plugins.cmp.jdbc.JDBCLoadEntityCommand.execute(JDBCLoadEntityCommand.java:165)
                      at org.jboss.ejb.plugins.cmp.jdbc.JDBCLoadEntityCommand.execute(JDBCLoadEntityCommand.java:62)
                      at org.jboss.ejb.plugins.cmp.jdbc.JDBCStoreManager.loadEntity(JDBCStoreManager.java:495)
                      at org.jboss.ejb.plugins.CMPPersistenceManager.loadEntity(CMPPersistenceManager.java:410)
                      at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.loadEntity(CachedConnectionInterceptor.java:353)
                      at org.jboss.ejb.plugins.EntitySynchronizationInterceptor.invoke(EntitySynchronizationInterceptor.java:310)

                      From examining the SQL traces, I do not think this is an issue with transactions and isolation levels. It is somewhat similar though.

                      Jboss’s CMP behavior seems normal:

                      - The container maintains a list (of all keys?) for given entity.
                      - One thread requests the list from findAll(),
                      - Another thread removes an item from the list.
                      - The first thread tries to access the deleted item via the reference retrieved from the list, gets the NoSuchEntityException.

                      The behavior that I expect to see is, when I call findAll() and then a getter method on a reference from the returned collection, all in the same transaction, NOT to get a NoSuchEntityException.

                      In a very non-suggestive way, here is my thought:

                      The CMP container could handle this situation like databases do in a dirty-read scenario. Since the CMP engine is an extension of the database, it should apply the transaction isolation level semantics to its cached data. I do understand the fact the container does not have a context to determine when the getter calls are coming from interfaces retrieved from a findAll method, unless it imposes the transaction demarcation and isolation to its cache. For example, when we turn the row-locking on, we do get the behavior we want if we were updating. But in the case of read-only (Not turning row-locking on), the database is not imposing any concurrency transaction isolation.

                      I have been running the same multi-threaded test against Weblogic 7.0, and do get any of such exceptions and imagine this is how they do it, frankly I cannot think of any other way.

                      The test that I have attached reproduces this issue very quickly. I have included a Junit test, OrganizationTest.java, and use JunitPP to run it in iterations inside multiple threads.

                      java junit.extensions.PPTestRunner -r 6 -s 0 -w 3 bug.OrganizationTest

                      Please let me know if you can do anything about this issue or if I you need me provide more information. Thanks.

                      • 8. Re: concurrency problem in CMP
                        dsundstrom

                        Caches are not my area of expertise, and I realy don't understand the problem you are having. I suggest you packages this as a bug report an post it to SourceForge, and the guy who workes on the cache will look at it.