1 2 Previous Next 19 Replies Latest reply on Sep 19, 2010 2:15 AM by abdulrahman

    jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update

    jedizippy

       

      We have a subprocess that essentially is composed to two activities. The first calls a <custom> java class which in turn moves to the second <custom> activity which performs a wait until a signal is recieved. The main process is composed of a number of steps that basically call this subprocess with continue="async". We also have a version of this subprocess that just does the calls the custom java class amd another one that only does the custom wait state.

       

      We are now running into an issue with a contraint violation which is the primary key of the JBPM4_EXECUTION table. This seems to happen at varying points throughout the workflow. Somtimes just after the main process is persisted (upon calling the first sub-process invocation).

       

      Sometimes it is in one of the sub processes after they have executed the the wait (so is presumably persisted to the database).

       

      In fact it seems completely random. Has anyone ran into this ? It seems (cannot be sure) that this could have been caused by removing continue="async" from some of the steps in the subprocesses however we cannot be sure !.

       

      It seems strange to be getting unique key constraint violations on the execution table !?

       

      We are using the JobExecutor with the default configuration (we have not changed it just imported it).

       

      17:06:16.106 | DEBUG | pool-1-thread-2 | d.o.s.p.c.BusinessServiceResponseHandler | Order Id [Ord2717322722] Waiting For

      Response - Start Time [1264089976104]

      17:06:16.110 | DEBUG | pool-1-thread-2 | d.o.s.p.c.BusinessServiceResponseHandler | Order Id [Ord2717322722] Wait Execut

      ed !

      17:06:16.245 | WARN | pool-1-thread-2 | o.h.u.JDBCExceptionReporter | SQL Error: 1, SQLState: 23000

      17:06:16.245 | WARN | pool-1-thread-2 | o.h.u.JDBCExceptionReporter | SQL Error: 1, SQLState: 23000

      17:06:16.249 | ERROR | pool-1-thread-2 | o.h.u.JDBCExceptionReporter | ORA-00001: Unique Constraint (WIEDMANNJ_DEV.SYS_C

      0017598) verletzt

      17:06:16.249 | ERROR | pool-1-thread-2 | o.h.u.JDBCExceptionReporter | ORA-00001: Unique Constraint (WIEDMANNJ_DEV.SYS_C

      0017598) verletzt

      17:06:16.261 | WARN | pool-1-thread-2 | o.h.u.JDBCExceptionReporter | SQL Error: 1, SQLState: 23000

      17:06:16.261 | WARN | pool-1-thread-2 | o.h.u.JDBCExceptionReporter | SQL Error: 1, SQLState: 23000

      17:06:16.268 | ERROR | pool-1-thread-2 | o.h.u.JDBCExceptionReporter | ORA-00001: Unique Constraint (WIEDMANNJ_DEV.SYS_C

      0017598) verletzt

      17:06:16.268 | ERROR | pool-1-thread-2 | o.h.u.JDBCExceptionReporter | ORA-00001: Unique Constraint (WIEDMANNJ_DEV.SYS_C

      0017598) verletzt

      17:06:16.327 | ERROR | pool-1-thread-2 | o.h.e.d.AbstractFlushingEventListener | Could not synchronize database state wi

      th session

      org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update

      at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:94) [hibernate-core-3.3.1.GA.jar:3.3

      .1.GA]

      at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66) [hibernate-core-3.3.1.GA.jar

      :3.3.1.GA]

      at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:275) [hibernate-core-3.3.1.GA.jar:3.3.1.

      GA]

      at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:114) [hibernate-core-3.3.1.GA.jar:3.

      3.1.GA]

      at org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:109) [hibernate-core-3.3.1.GA.jar:3.

      3.1.GA]

      at org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:244) [hibernate-core-3.3.1.GA.j

      ar:3.3.1.GA]

      at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2252) [hibernate-c

      ore-3.3.1.GA.jar:3.3.1.GA]

      at org.hibernate.persister.entity.AbstractEntityPersister.insert(AbstractEntityPersister.java:2688) [hibernate-c

      ore-3.3.1.GA.jar:3.3.1.GA]

      at org.hibernate.action.EntityInsertAction.execute(EntityInsertAction.java:79) [hibernate-core-3.3.1.GA.jar:3.3.

      1.GA]

      at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279) [hibernate-core-3.3.1.GA.jar:3.3.1.GA]

      at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263) [hibernate-core-3.3.1.GA.jar:3.3.1.GA]

      at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:167) [hibernate-core-3.3.1.GA.jar:3.3.1.GA]

      at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:32

      1) [hibernate-core-3.3.1.GA.jar:3.3.1.GA]

      at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50) [hibernate-core-

      3.3.1.GA.jar:3.3.1.GA]

      at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1027) [hibernate-core-3.3.1.GA.jar:3.3.1.GA]

      at org.jbpm.pvm.internal.tx.HibernateSessionResource.prepare(HibernateSessionResource.java:56) [jbpm-pvm-4.3-pat

      ched.jar:na]

      at org.jbpm.pvm.internal.tx.StandardTransaction.commit(StandardTransaction.java:107) [jbpm-pvm-4.3-patched.jar:n

      a]

      at org.jbpm.pvm.internal.tx.StandardTransaction.complete(StandardTransaction.java:64) [jbpm-pvm-4.3-patched.jar:

      na]

      at org.jbpm.pvm.internal.tx.StandardTransactionInterceptor.execute(StandardTransactionInterceptor.java:61) [jbpm

      -pvm-4.3-patched.jar:na]

      at org.jbpm.pvm.internal.svc.EnvironmentInterceptor.executeInNewEnvironment(EnvironmentInterceptor.java:53) [jbp

      m-pvm-4.3-patched.jar:na]

      at org.jbpm.pvm.internal.svc.EnvironmentInterceptor.execute(EnvironmentInterceptor.java:40) [jbpm-pvm-4.3-patche

      d.jar:na]

      at org.jbpm.pvm.internal.svc.RetryInterceptor.execute(RetryInterceptor.java:55) [jbpm-pvm-4.3-patched.jar:na]

      at org.jbpm.pvm.internal.svc.SkipInterceptor.execute(SkipInterceptor.java:43) [jbpm-pvm-4.3-patched.jar:na]

      at org.jbpm.pvm.internal.jobexecutor.JobParcel.run(JobParcel.java:48) [jbpm-pvm-4.3-patched.jar:na]

      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [na:1.6.0_11]

      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) [na:1.6.0_11]

      at java.util.concurrent.FutureTask.run(FutureTask.java:138) [na:1.6.0_11]

      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_11]

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_11]

      at java.lang.Thread.run(Thread.java:619) [na:1.6.0_11]

        • 1. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update

          Hi Martin

           

          Think we have similar issues. Have read through your issue reported in: https://jira.jboss.org/jira/browse/JBPM-2706. Think they're more or less the same underlying issue.

           

          In our case, we have may process which has several sub-processes and the sub-process actually executes some java/decision activities. THe sub-processes and the java activies are marked as 'async'.

           

          In our test we have 4 job executors.

           

          when using on HSQL, we run into violation of unique constraint, eg,

           

          Caused by: java.sql.SQLException: Violation of unique constraint $$: duplicate value  for column  $$: SYS_CT_107322 in statement [update JBPM4_EXECUTION set DBVERSION_=?, ACTIVITYNAME_=?, PROCDEFID_=?, HASVARS_=?, NAME_=?, KEY_=?, ID_=?, STATE_=?, SUSPHISTSTATE_=?, PRIORITY_=?, HISACTINST_=?, PARENT_=?, INSTANCE_=?, SUPEREXEC_=?, SUBPROCINST_=? where DBID_=? and DBVERSION_=?]

          if we run it on DB2, it gives deadlock issue(which i posted in this forum yesterday):

           

          ### EXCEPTION ###########################################
          19:57:51,203 SEV                 | [AbstractFlushingEventListener] Could not synchronize database state with session
          org.hibernate.exception.LockAcquisitionException: pool-1-thread-5: could not update: [org.jbpm.pvm.internal.model.ExecutionImpl#166]
          at org.hibernate.exception.SQLStateConverter.convert(SQLStateConverter.java:105)
          at org.hibernate.exception.JDBCExceptionHelper.convert(JDBCExceptionHelper.java:66)
          at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2466)
          at org.hibernate.persister.entity.AbstractEntityPersister.updateOrInsert(AbstractEntityPersister.java:2340)
          at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2653)
          at org.hibernate.action.EntityUpdateAction.execute(EntityUpdateAction.java:115)
          at org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
          at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
          at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:168)
          at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
          at org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:50)
          at org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1028)
          at org.jbpm.pvm.internal.tx.HibernateSessionResource.prepare(HibernateSessionResource.java:56)
          at org.jbpm.pvm.internal.tx.StandardTransaction.commit(StandardTransaction.java:107)
          at org.jbpm.pvm.internal.tx.StandardTransaction.complete(StandardTransaction.java:64)
          at org.jbpm.pvm.internal.tx.StandardTransactionInterceptor.execute(StandardTransactionInterceptor.java:61)
          at org.jbpm.pvm.internal.svc.EnvironmentInterceptor.executeInNewEnvironment(EnvironmentInterceptor.java:53)
          at org.jbpm.pvm.internal.svc.EnvironmentInterceptor.execute(EnvironmentInterceptor.java:40)
          at org.jbpm.pvm.internal.svc.RetryInterceptor.execute(RetryInterceptor.java:55)
          at org.jbpm.pvm.internal.svc.SkipInterceptor.execute(SkipInterceptor.java:43)
          at org.jbpm.pvm.internal.jobexecutor.JobParcel.run(JobParcel.java:48)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
          at java.lang.Thread.run(Thread.java:619)
          Caused by: com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -911, SQLSTATE: 40001, SQLERRMC: 2
          at com.ibm.db2.jcc.c.kh.c(kh.java:1660)
          at com.ibm.db2.jcc.b.db.s(db.java:875)
          at com.ibm.db2.jcc.b.db.k(db.java:387)
          at com.ibm.db2.jcc.b.db.a(db.java:60)
          at com.ibm.db2.jcc.b.t.a(t.java:52)
          at com.ibm.db2.jcc.b.tb.b(tb.java:202)
          at com.ibm.db2.jcc.c.lh.X(lh.java:1842)
          at com.ibm.db2.jcc.c.lh.d(lh.java:2411)
          at com.ibm.db2.jcc.c.lh.T(lh.java:465)
          at com.ibm.db2.jcc.c.lh.executeUpdate(lh.java:448)
          at org.hibernate.persister.entity.AbstractEntityPersister.update(AbstractEntityPersister.java:2448)
          ... 24 more

          • 2. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
            kukeltje
            4 Job executors? Running where? Different machines? Or at least bound to different IP adresses? If nothave a look at that first, or try to reproduce running with one JobExcutor
            • 3. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
              kukeltje

              Martin,

               

              Is this on a clustered system or just one node? There have been multiple reports lately on constraint violations which from what I see now could be related to the ID Generator, just not sure. Can you check what happens if you e.g. use a DBMS native id generator and not the one that comes with jBPM (since 4.2 afaik). Altough I'm not sure if that can be replaced in just the hibernate config file.

              • 4. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
                jedizippy

                Hi,

                 

                This is basic plain vanilla jBPM 4.3 deployed as an EAR to a single standalone instance of WLS (not clustered). Our jbpm.cfg.xml is just importing the Job Executor so we have not changed any of the threads or lock values so it should be using the default 3 threads according to the docs-

                 

                <import resource="jbpm.jobexecutor.cfg.xml"/>

                 

                We are running in a JTA enviroment and reading the latest 4.3 docs we should be importing the jta config:-

                 

                <import resource="jbpm.tx.jta.cfg.xml"/>

                 

                I will try this as currently we have been using the basic hibernate tx config:-

                 

                <import resource="jbpm.tx.hibernate.cfg.xml"/>

                 

                I am not sure if this would have any bearing on this problem or not. I will look into it further today and see if this has any bearing.

                 

                We are running against Oracle 11g so its seems then that its not related to the DB according to Tians post.It would seem as you say the problem lies in the unique ID generation for the execution id.

                 

                Regards

                Martin

                • 5. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
                  jedizippy
                  btw we are already using the DatabaseIdComposer (or should be) as we are importing jbpm.default.cfg.xml and in there is has:-
                  <object class="org.jbpm.pvm.internal.id.DatabaseIdComposer" init="eager" />
                  • 6. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update

                    we're running 5 jobexecutors(sorry, it's not 4) the same machine. we're trying to strip down the logic to remove the sub-process and it seems fine. Now we're looking to add subprocess and running in just one jobexecutor.

                     

                    Thanks

                    • 7. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
                      jedizippy

                      Hi,

                       

                      Did you manage to get any further with this and one job executor ?

                       

                      Regards

                      Martin

                      • 8. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update

                        While we strip down the logic to remove the subprocess. It seems running fine with multiple job executors(haven't seen issue so far).

                         

                        When we check the sql statements to create database tables for db2, it doesn't define the unique constrain of ID_ on the JBPM4_EXECUTION table.  But that field is unique in the sql statements for HSQL and MySQL at least. the Unique constrain is defined in the mapping file. Think that's why we see different issues in db2.

                         

                        We also did some analysis on the deadlock issue we experienced on db2.

                        Have identified the statements to lock tables and release them.  Have captured the stack trace and the java classes(the beauty of open sources ) below.

                         

                        Deadlock in Hibernate

                         

                        From our observation , in Hibernate a group of tables are inserted/updated in one Transaction and all tables involved(not the whole database as we thought on Friday night, thought we just happened to check the tables were involved in the lock) are LOCKED before the commit on the jdbc connection was called.

                        In details once the statement.executeUpdate() is called on a query, the table involves in the query is locked until the commit statement is called. So just before the commit statement all tables involved are actually locked. At that moment if another thread involves in any operation(eg. read or update) on any of the tables, it will cause a DEADLOCK in the default isolation level(think it’s repeatable read in DB2Dialect)

                        In our case it involves multiple tables to create an jbpm instance, they are:

                        Jbpm4_execution

                        Jbpm4_hist_procinst

                        Jbpm4_job

                        Jbpm4_variable

                        Jbpm4_lob

                        From Hibernate’s perspective, I  think that might be expected behaviour as a group of tables may have some relations, eg. 1-m, m-m relations, so it’s nature to deal with them in one transaction. If not lock them together( I have tested by change the DB2Dialect to update with NC) it will have some wired behaviours – later on jbpm cannot find some rows of data it expects them to be there.

                         

                        Behaviour of JBPM

                         

                        On the jbpm side, we know:

                        1)      If an instance key isn’t unique, it will definitely cause problem. We observe this situation while we have subprocess (all subprocesses are created as an instance which has no key. Given the limitation in the subprocess key field that we cannot use expression, this is an issue). The Custom activity is trying to create subprocess as well so have this similar issue

                        2)      A standard jbpm process without subprocess seems running fine with multiple job executors(??? We just observe this from our test with the strip down process definition ). That means the tables involving in executing one activity( not include subprocess activity of course ) are properly isolated in the jbpm level.

                        If both 1) and 2) are true, think we have no problem to use it in our environment. We just need to take the inconvenience of exception handling in the DF CARMI call, and be creative to control the 1-m relations between an instance and its sub-tasks….

                         

                        A stack trace to identify the statement which lock the tables:

                        jbpm_db2_lock_tables.png

                         

                        A stack trace to identify the unlock statement:

                        jbpm_db2_unlock_tables.png

                        • 9. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update

                          Hi Martin

                           

                          How's thing with you? are you running ok with the subprocess on multiple job executors?

                           

                          Thanks

                          ZT

                          • 10. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
                            jedizippy

                            Hi ZT,

                             

                            Well the current status is that we still do not know what is causing the issue. There is without any doubt a bug in the id generation somewhere but we have not been able to nail down where the root cause is. We get integrity constraint violations in random places throughout the workflow and its not reproducable. Also our workflow is quite complex but it is just a sequence of calls to a generic subprocess that handles retry and manual error handling processing and these subprocesses all have continue="async" on.

                             

                            We found that removing continue=async on our custom and task nodes in that subprocess caused the Constraint Violation to be reproducable every time. Adding continue=async solved the issue so presumably forcing the subprocess state to persist to the DB somehow prevents the duplicate ID generation. But its flaky.

                             

                            However just this morning for example l ran our test and it failed at the very start with this same Constraint violation again. The process is started and the first step is a <java> with continue="async". We had to add this so the caller will get immediate return and not wait for the process. We tried adding continue="async" on the <start> but we get NullPointerException then (its been like that since 4.1) so we have this Java activity which just logs a message and then moves onto the first subprocess call. At this first java activity we got our Constraint Violation so it happened when jbpm attempted to persist the main process (not subprocess). We invoke (create) our processes using MDB's and in this case the second time the MDB was fired and tried to create the process it worked.......

                             

                            We are using out of the box job executor with no changes so according to the docs that is 3 threads (but I have not had time to verify that). We are running on WLS 11g but I am unable at the moment to verify which JTA configuration we should be using. As soon as you mention transactions on this forum it all goes quiet. All the questions remain unanswered. I added a previous (unanswered) post about this a few days ago.

                             

                            We are due to deliver into end to end testing next Friday but to be honest given the current status that seems increasingly unlikey as that environment is clustered etc etc and we cant even get this to run properly on a single WLS instance at the moment.

                             

                            Will update with any further findings !.

                             

                            One option might be to write our own ID Generator and use an Oracle sequence but not sure how much effort that will be. But it might be an option we have to consider.

                             

                            Cheers

                            MP

                            • 11. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update

                              Thanks for sharing Martin

                               

                              Interesting to know that you have done the test to add continue="async" on the Start activity. We did the similar test and found the same issue. Hope someone from the JBPM team could explain whether it's a known problem which will be fixed in the future release.

                               

                              In our case we use DB2 9.1.7 and running JBPM as a standalone server(wrap up JBPM in a java process ) with out of box job executors.The issue we experienced frequently is db2 SQL exception of deadlock(SQL Error code -911) on JBPM4_EXECUTION, and sometimes on JBPM4_JOB as well. The different errors we saw could be due to the differences between Oracle and DB2 on the default constrains of the tables and the isolation levels. The cause could be the same underlyer issue or issues. As you pointed out, the ID generator could be a point of issue. We also doubt about the way JBPM handling the subprocess activity as while as it seems running fine if we strip off the subprocess.

                               

                              We have pushed our QA for two weeks because of this issue. We'll give a shot to the id generator to see if we can do anything. Will keep you posted.

                               

                              Thanks

                              ZT

                              • 12. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
                                jedizippy

                                HI,

                                 

                                As you say I am sure if we coded everything without subprocesses. We have been having issues with subprocesses since 4.1 to be honest. But we thought we had solved the issues. However it would seem that it was onlyt short lived. One theory we have regarding this is that it could be caused due to a XA transaction rollback. We have calls from our <custom> code in the subprocess that call EJBs which send JMS messages and wait for a JMS response. If this times out as no response is recieved then the XA transaction timeout is thrown. Maybe under this circumstance it causes the ID generation issue as sometime later (even during another process instance) the constraint violation could appear. Again as I said we havre no confirmed this. It could all be related to the JTA configuration but I cant get an answer to my questions on this......maybe its all related.

                                 

                                I have a great fear at this point in time that we will not progress past end to end testing as there are so many issues appearing now during unit and load tests.But hopefully I am wrong !.

                                 

                                Regards

                                Martin

                                • 13. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
                                  jedizippy
                                  I created JBPM-2771 for the continue=async on the <start> activity.Feel free to vote for it if you want it fixed !.
                                  • 14. Re: jbpm 4.3 - org.hibernate.exception.ConstraintViolationException: Could not execute JDBC batch update
                                    kukeltje

                                    Let me first clear some things up. This forum does not get silent when transactions are mentioned, it just does not get active. Keep in mind that this is a user forum. User forum in more sense then one. It is for users, by users. So it is a peer forum. If you run a less commonly used application server (WLS 11g), there is a bigger chance of not getting help. I myself used WLS up to 8.1SP4 (or was it 5) so have no options to even try to help out and suggesting things. So I keep quiet to. This is probably true for many other users as well. (using db2 is comparable as there are also less people using it, at least as WLS in combination with jBPM)

                                     

                                    Secondly, as long as there is no *minimal* testcase where things can be reproduced on the (currently) supported configurations (supported in the sense that there is qa cycle for it, not that it will only run on these), it is even harder to start helping out.

                                     

                                    What I further get from this topic is that I hear things like table locking... that is scary

                                     

                                    Further more, if things seem related to 'async', (dead) locks etc, read 'Async continuations' again and pay special attention to the third possible value of the continue attribute, called exclusive.... Maybe it helps...

                                     

                                    Cheers,

                                     

                                    Ronald

                                    1 2 Previous Next