Riftsaw / ODE Performance Comparison
In the tests I used:
- a web service NewWebService that concats "Hello " with an input string
- a bpel process SimpleInvoke of this structure:
<receive>
<assign />
<foreach 1 to x>
<invoke NewWebService-sayHello />
</foreach>
<assign />
<reply />
- Direct call of NewWebService
- Call of SimpleInvoke BPEL with 0 WS calls (just receive - reply)
- Call of SimpleInvoke BPEL with 1 WS call (x = 1)
- Call of SimpleInvoke BPEL with 2 WS calls (x = 2)
- Call of SimpleInvoke BPEL with 3 WS calls (x = 3)
- JBoss AS 5.1.0.GA + Riftsaw-2.0-CR1
- Apache Tomcat 6.0.24 + ODE 1.3.3
- Apache Tomcat 6.0.24 + ODE 2.0beta2
- Riftsaw 2.0 CR1 - When testing higher number of messages I got errors in the console: org.h2.jdbc.JdbcSQLException: Concurrent update in table ODE_JOB: another transaction has updated or deleted the same row [90131-68]
- ODE 1.3.3 - When testing higher number of messages I got errors in the console: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/ode/processes/simpleInvoke/sayHelloTo
- ODE 2.0 beta2 - No errors even for higher message count, but e.g. after testing 1000 msgs a few times, the results got worse, for BPEL 1xWS from 4.5 msgs/s to around 1 msgs/s and didn't get back up when trying with lower amount of messages (1, 10, 100)
- ODE 2.0beta2 is about 2x slower then ODE 1.3.3
- Riftsaw 2.0 CR1 is about 2x slower then ODE 1.3.3
- ODE 2.0beta2 is more stable then ODE 1.3.3
Other results
Riftsaw 2.0 M1 BPEL WS 1x = 6 msgs/s
Riftsaw 2.0 CR1 with Derby database BPEL WS 1x = 9 msgs/s
Riftsaw 2.0 CR1 with in-memory persistence BPEL WS 1x = 90 msgs/s
In-memory tests of all platforms
In the next test I tried all the 4 platforms, this time without database usage and also logging turned off. All the test are run with BPEL that calls the web service 1x.
| Platform | Result |
|---|---|
| Riftsaw 2.0-CR1 on JBoss 5.1.0.GA | 90 msgs/s |
| Riftsaw 2.0-M1 on JBoss 5.1.0.GA | 115 msgs/s |
| ODE 1.3.3 on Apache Tomcat 6.0.24 | 190 msgs/s |
| ODE 2.0-beta2 on Apache Tomcat 6.0.24 | 170 msgs/s |
In-memory tests with simplest possible BPEL
For this test I simplyfied the BPEL as much as possible, leaving only <receive> and <reply> elements and removed every reference to the external web service. So this test shows results of pure BPEL calls without any interaction with database or external web service or any funcionality inside the process itself.
I tested on these setups:
- ODE 1.3.3 on Lab machine
- ODE 2.0beta2 on Lab machine
- Riftsaw M1 on Lab machine
- Riftsaw CR1 on Lab machine
- ODE 1.3.3 on my Notebook
- Riftsaw CR1 on my Notebook
I've done the same sequence of tests for each setup and I'm attaching the result table. There are 14 tests with different amount of messages from 1.000 to 100k. In the white fields there is msgs/s value for that test.
Possible conclusions:
- ODE 1.3.3 is 20% faster as ODE 2.0beta2 (further slowdown for 2.0 seams to be due to the interaction with database)
- ODE 1.3.3 is about 2x faster then Riftsaw CR1
- M1 seams to be pretty close to 2.0beta2 but has performance issues for higher message counts
Comments