12 Replies Latest reply on Jan 23, 2009 12:57 PM by camunda

    Any possibility of persistence-API?

      Hi.
      I usually develop enterprise systems with DI container such as Spring, Guice, etc..
      Recently, I ve chosen iBatis or another light persistent API as a persistent / data access framework.
      So, I'd like to use jBPM with above oss frameworks, not hibernate.
      If you guys prepare the persistence API for it, jBPM would be much more used widely, and some of developers can help to contribute it.

      Any possibility of persistence-API? No chance?

        • 1. Re: Any possibility of persistence-API?
          kukeltje

           

          jBPM would be much more used widely, a
          Hmmm... serious? ;-)

          You mean by swithching to using JPA so libs below are pluggable?

          A 'good' practice is to not store domain data in your workflow engine and just store references. It is possible then (afaik) to use both frameworks (hibernate and iBatis) side by side then by using XA datasources... and a JTA transactionmanager. So the real need is not that high imo.

          • 2. Re: Any possibility of persistence-API?
            brittm

             

            A 'good' practice is to not store domain data in your workflow engine and just store references. It is possible then (afaik) to use both frameworks (hibernate and iBatis) side by side then by using XA datasources... and a JTA transactionmanager. So the real need is not that high imo.

            The issue with this (and for us its been a big issue) is the subsequent inability to update then read the same table row in one XA when the row is updated by one persistence mechanism and then a read attempt is made by the other. (This generates a deadlock in the DB.)

            Scenario:
            * A Business domain uses the container's JPA. jBPM uses Hibernate, so we're using two different persistence architectures.
            * User completes a task via an application in the business domain. The action updates a series of domain objects based on data recorded from the screen and then calls a jBPM service to advance the process.
            * A jBPM process executes and at some point needs to read data from one of the domain tables that were updated earlier. (A jBPM process may traverse multiple decision trees and launch multiple points of integration and/or child processes, so that it may be impractical to try to consider exactly what all might be part of this big XA transaction.)
            * The database will deadlock since the original update to the table won't commit till the entire XA commits. However, the XA won't commit till the read is complete, and the read is waiting for the update to commit.


            If both domains were using the same persistence architecture (the same persistence unit), we'd be using the same JTA transaction and the same DB connection, and there would be no deadlock.

            This problem can be overcome in one of three ways that I can see:
            * Use a single persistence unit on a single server for both the domain and jBPM.
            * Enlist jBPM (Hibernate) in the same JTA transaction that was started via JPA. Anyone know how to readily do this?
            * Put the domain application and jBPM on separate servers, using a JMS (or similar) interface between the two. This ensures (mostly) that the domain transaction can complete ahead of any jBPM transaction and vice-versa. Of course, if your process does lots of back and forth with your domain objects, this is going to get pretty heavy pretty fast--don't forget to make provisions for rolling back each call. You had better be good at writing a solid framework to handle it, or you're going to lose your sanity.


            These solutions really reflect two simple strategies. When it comes to persistence, either get your domain and jBPM completely together, or keep them completely apart. Any in-between is a recipe for eventual trouble. Unfortunately, in its current incarnation, jBPM doesn't make either of these solutions readily available--you can't readily switch out the persistence mechanism, and jBPM doesn't provide a viable JMS interface.

            To date, we've been able to side-step, hurdle, or otherwise work around these deadlock issues as they arise, but it's truly been a mine field for the designer (me, in this case). Am I just missing something really simple here?

            Of course, the impetus to break out jBPM's persistence mechanism is lessened since future versions will be based on standard JPA, but the above concerns are applicable to anyone using the 3.x branch--which is basically every jBPM user at this point.

            • 3. Re: Any possibility of persistence-API?

              Britt,

              Nice post! This area can always use more thought.

              The one pushback I see (and it's major), is that when a domain business method fails and rolls back, it's painful for that to roll back the workflow persistence operations as well. The amount that is rolled back is essentially unpredictable, since it depends on details of the engine's implementation that probably should not be specified or relied on. For instance, in JBPM3, you could have a rollback that would obliterate some or all of the activity on other "tines" of a fork. Everything is in sync, but the behavior is probably unexpected.

              Of course, in the two-transaction model, you can have the JBPM transaction roll back, while the business methods have already committed - the inverse of the problem described above. But I think this is a lesser problem.

              -Ed Staub

              • 4. Re: Any possibility of persistence-API?
                brittm

                 

                For instance, in JBPM3, you could have a rollback that would obliterate some or all of the activity on other "tines" of a fork. Everything is in sync, but the behavior is probably unexpected.

                One way to mitigate the "roll back all 30 branches of execution" is to place async "save" points in your process--whether by jBPM's own async feature, or your own JMS functionality. This way, the process would persist itself before continuing. If you had a roll back on one branch, you'd just roll back to the save point for that branch. This way, the process has a 'plan' for managing transaction scope.

                Since we should all be catching reasonably expected exceptions, a failure to commit should be an extraordinary event in which there is no certainty of what went wrong. In such a case, the only safe response is to roll back the entire transaction to some manageable point--preferably where you started. Sure, some JMS integration messages may have gone out the door, but they're all tagged with a business key/id that would allow them to be resent without causing trouble (right?) So in my line of thinking, I would definitely want everything rolled back and in sync.


                Unfortunately, many of our use cases aren't limited to two transactions. As the signalled process moves back and forth gathering or updating data in the domain, the XA can wrap several transactions. For instance

                1) Domain updates data and signals process
                2) Process execution sets status at some point on domain objects (e.g. inProgress)
                3) Process execution continues
                4) Process reads domain objects for decision making, forking, etc.
                5) Process updates status on domain objects (e.g. inService)
                6) XA finally commits

                These issues became large enough that we no longer start processes directly from the domain application--all initial signalling of new process instances is now done via JMS. (The domain creates and prepares the process instance, but uses JMS to signal out of the Start state.) But this only addresses the first hand-off between transactions; if, in the same execution, the process itself updates the domain then later tries to read the same domain table row (for example, from two different ActionHandlers), the we have the deadlock again because we're utilizing different DB connections.


                • 5. Re: Any possibility of persistence-API?
                  tom.baeyens

                   

                  "wkzk" wrote:
                  Hi.
                  I ve chosen iBatis or another light persistent API as a persistent / data access framework.
                  So, I'd like to use jBPM with above oss frameworks, not hibernate.


                  can you elaborate on this motivation. the only motivation that i have seen in this context is that from a coding perspective, this looks cleaner.

                  but in fact the downsides are much bigger.

                  1) the api would have to be reduced to the intersection of what hibernate, jpa, ibatis and all other potential persistent stores would offer.

                  another way to make this point is that we say: there is such an api! the interface is called org.hibernate.Session. go and implement it with iBatis. probably you realize that is not going to be a trivial job. that is the same work that you ask us to do when you ask us for a persistence SPI.

                  2) it would be impossible to do the QA on a matrix of JDK, DB and persistence framework

                  3) we provide nice integrations with hibernate, but you can always ignore those and consider hibernate a jbpm implementation detail. in which case, the only thing that comes out of jbpm is a JDBC connection.

                  • 6. Re: Any possibility of persistence-API?
                    camunda

                     

                    brittm wrote:

                    * Enlist jBPM (Hibernate) in the same JTA transaction that was started via JPA. Anyone know how to readily do this?


                    Maybe I get you wrong, but this is exactly what jbpm can do if you use the JtaDbPersistenceServiceFactory, or where is the problem with that solution?

                    Then Hibernate uses the current JTA-Transaction and everything works as expected in one transaction (even JMS and such stuff, not tied to a database).

                    wkzk wrote:

                    Ive chosen iBatis or another light persistent API as a persistent / data access framework.
                    So, I'd like to use jBPM with above oss frameworks, not hibernate.


                    Motivation would be also very interessting for me. The only real reason I heared till now was politics, that Hibernate is not a standard and JPA has to be used. There is no technical reason behind it, pure politics.

                    But with iBatis you don't have a standard either, where is the value to exchange jbpm persistence mechanism? You coul deven write a integration to persist domain objects as process variables with iBatis instead of Hibernate, why do you care what jbpm itself uses?

                    Cheers
                    Bernd

                    • 7. Re: Any possibility of persistence-API?

                      Thank you guys.

                      "tom.baeyens@jboss.com" wrote:

                      can you elaborate on this motivation. the only motivation that i have seen in this context is that from a coding perspective, this looks cleaner.

                      As a coding perspective, Yes, its one of the reason.But the main motivations are, which is not smart , sorry,;-)
                      1. My team developers are Not familiar with Hibernate.(I know that much Hibernate-knowledge is not needed to use jBPM.but if anything happen in hibernate,it would be panic...)
                      2. I don't know well about passing the jdbc connection from iBatis to the hibernate Session on the same transaction...

                      That's it. I apologize that you guys expected much more smart things.

                      I appreciate lots of responces, anyway!

                      btw, standard is not big deal around me recently. JEE6 doesn't show up yet, I can't wait. ;-)


                      • 8. Re: Any possibility of persistence-API?
                        brittm

                         

                        Maybe I get you wrong, but this is exactly what jbpm can do if you use the JtaDbPersistenceServiceFactory, or where is the problem with that solution?

                        My apologies, I wasn't very clear. With JtaDbPersistenceServiceFactory the Hibernate transaction is enlisted in the greater XA transaction that was started by JPA, but the Hibernate transaction is still a new "sub" transaction in a new thread with a new DB connection. However, to work, the solution would have to ensure that jBPM's persistence mechanism is using the actual DB connection that was opened from JPA, so that Hibernate and JPA are using the same DB connection. Using the same DB connection is actually what enables the DB to avoid the deadlock scenario.

                        • 9. Re: Any possibility of persistence-API?
                          aguizar

                           

                          the api would have to be reduced to the intersection of what hibernate, jpa, ibatis and all other potential persistent stores would offer.

                          Tom, do we really need anything else than the intersection in jBPM 4? Even Seam manages to do with JPA only...
                          it would be impossible to do the QA on a matrix of JDK, DB and persistence framework [/quote
                          Totally true, but we open the door for other folks to do their QA on something other than Hibernate. In jBPM 3 it is just impossible to replace Hibernate without rewriting half of the product.


                          • 10. Re: Any possibility of persistence-API?
                            camunda

                            But it should be JPA, not a own abstraction...

                            And I think it was agreed to use JPA in jbpm 4 unless we find any real showstopper?

                            • 11. Re: Any possibility of persistence-API?
                              aguizar

                              Indeed. JPA should be the intersection. I missed the Hibernate, JPA, iBatis bit. An abstraction layer on top of JPA would be a crime :-)

                              We agreed on JPA back in Antwerp if I recall correctly. No mention was made about who or when, tough. I would not mind, but the short term priorities are elsewhere. In any case it has to be there before GA, otherwise it will be impossible to switch later.

                              • 12. Re: Any possibility of persistence-API?
                                camunda

                                +1