Version 1

    Riftsaw / ODE Performance Comparison

    In the tests I used:

    • a web service NewWebService that concats "Hello " with an input string
    • a bpel process SimpleInvoke of this structure:



         <assign />

         <foreach 1 to x>

              <invoke NewWebService-sayHello />


         <assign />

    <reply />

    So I was able to test performance on these scenarios:
    • Direct call of NewWebService
    • Call of SimpleInvoke BPEL with 0 WS calls (just receive - reply)
    • Call of SimpleInvoke BPEL with 1 WS call (x = 1)
    • Call of SimpleInvoke BPEL with 2 WS calls (x = 2)
    • Call of SimpleInvoke BPEL with 3 WS calls (x = 3)

    I tested on these platforms:
    • JBoss AS 5.1.0.GA + Riftsaw-2.0-CR1
    • Apache Tomcat 6.0.24 + ODE 1.3.3
    • Apache Tomcat 6.0.24 + ODE 2.0beta2

    The results can be seen in the table below.
    Database used were default, so HSQL for Riftsaw and Derby for ODE 1.3.3 & 2.0.

    Some other remarks:
    • Riftsaw 2.0 CR1 - When testing higher number of messages I got errors in the console: org.h2.jdbc.JdbcSQLException: Concurrent update in table ODE_JOB: another transaction has updated or deleted the same row [90131-68]
    • ODE 1.3.3 - When testing higher number of messages I got errors in the console: Server returned HTTP response code: 500 for URL: http://localhost:8080/ode/processes/simpleInvoke/sayHelloTo
    • ODE 2.0 beta2 -  No errors even for higher message count, but e.g. after testing 1000 msgs a few times, the results got worse, for BPEL 1xWS from 4.5 msgs/s to around 1 msgs/s and didn't get back up when trying with lower amount of messages (1, 10, 100)

    Possible conclusions:
    • ODE 2.0beta2 is about 2x slower then ODE 1.3.3
    • Riftsaw 2.0 CR1 is about 2x slower then ODE 1.3.3
    • ODE 2.0beta2 is more stable then ODE 1.3.3


    Other results

    Riftsaw 2.0 M1 BPEL WS 1x = 6 msgs/s

    Riftsaw 2.0 CR1 with Derby database BPEL WS 1x = 9 msgs/s

    Riftsaw 2.0 CR1 with in-memory persistence BPEL WS 1x = 90 msgs/s


    In-memory tests of all platforms

    In the next test I tried all the 4 platforms, this time without database usage and also logging turned off. All the test are run with BPEL that calls the web service 1x.


    Riftsaw 2.0-CR1 on JBoss 5.1.0.GA90 msgs/s
    Riftsaw 2.0-M1 on JBoss 5.1.0.GA115 msgs/s
    ODE 1.3.3 on Apache Tomcat 6.0.24190 msgs/s
    ODE 2.0-beta2 on Apache Tomcat 6.0.24170 msgs/s

    In-memory tests with simplest possible BPEL

    For this test I simplyfied the BPEL as much as possible, leaving only <receive> and <reply> elements and removed every reference to the external web service. So this test shows results of pure BPEL calls without any interaction with database or external web service or any funcionality inside the process itself.


    I tested on these setups:

    • ODE 1.3.3 on Lab machine
    • ODE 2.0beta2 on Lab machine
    • Riftsaw M1 on Lab machine
    • Riftsaw CR1 on Lab machine
    • ODE 1.3.3 on my Notebook
    • Riftsaw CR1 on my Notebook


    I've done the same sequence of tests for each setup and I'm attaching the result table. There are 14 tests with different amount of messages from 1.000 to 100k. In the white fields there is msgs/s value for that test.

    Possible conclusions:

    • ODE 1.3.3 is 20% faster as ODE 2.0beta2 (further slowdown for 2.0 seams to be due to the interaction with database)
    • ODE 1.3.3 is about 2x faster then Riftsaw CR1
    • M1 seams to be pretty close to 2.0beta2 but has performance issues for higher message counts