JBoss vs WebLogic --- findings

[svante]

Guys,

I compared the speed of our J2EE-based statistics/polling application on WebLogic and JBoss. My findings:

Setup:
Server machine: Dell Optiplex GX150 (930MHz Pentium III, 512MB RAM), RedHat Linux 7.3
Remote database server: Sun Enterprise450, Oracle 8.1.7i
Client machines: Dell Optiplex GX150, Sun Ultra 10

The client machines posted votes in the form of http requests to a Servlet.

There were a total of 30 concurrent client processes.

The application server used thin JDBC drivers from Oracle. Prior to any timing the test cases were run four times to ensure pools have been set up, beans instantiated and the JVM has done whatever optimisations it can.

Tests runs:
1. WebLogic 7.0 SP1, HotSpot 1.3.1_03: 401 req/s
2. WebLogic 7.0 SP1, HotSpot 1.3.1_03: 401 req/s
3. WebLogic 7.0 SP1, JRockit 7.0: 557 req/s
4. WebLogic 7.0 SP1, JRockit 7.0: 583 req/s

5. JBoss 3.0.2, HotSpot 1.3.1_04: 255 req/s
6. JBoss 3.0.2, HotSpot 1.3.1_04: 245 req/s
7. JBoss 3.0.2, JRockit 7.0: 263 req/s
8. JBoss 3.0.2, JRockit 7.0: 274 req/s

JRockit seems to be more of a benefit when running WebLogic than when running JBoss. The speed difference between WebLogic and JBoss is significant, at least as far as to the components of this test: parsing http requests, managing transactions, security checks on EJB calls, etc. The database access is based on Oracle's JDBC classes in both cases so there should be little difference.

Without getting into the architecture of the application, is this roughly what to expect in general speed difference between the application servers? I was surprised to see such difference.

Comments?


Cheers,
Svante

[pbhandar1]

Just wanted to give an alternate view of EJBs. I’ve developed a multi-transactional web application (an online community) using J2EE, EJBs, & JBoss and I’m quite happy with the technology and the performance. The performance numbers and architecture are posted in the message ‘Informal Performance Numbers For JBoss 3.0/Jetty’. When one person can develop a mutli-transactional web application that is capable of handling 60 to 100 simultaneous users on an entry level server, I would say the technology to develop that application has a lot of practical use to many business applications.
The biggest advantage I get from using EJBs is the ability to handle multiple transactions. Getting a software system to handle multiple transactions reliably is no small feat. I’ve witnessed knowledgeable and competent teams of software engineers unable to successfully accomplish this task in a reasonable time frame for client server applications.
I run my web app on Linux with JBoss but I’ve also tested it on Windows 2000 with Weblogic. I had very good portability between the two platforms. I haven’t tested with WebSphere however.

[moraelin]

This more or less mirrors my findings when comparing JBoss to IBM Websphere. JBoss is slow.

In my case, there was (practically) no database access in the mix, either. Beyond the very first call in 10,000 calls, the data was cached and the actual work in the call ammounted to a simple HashMap lookup. There was no HTTP access involved, either. I was just calling a stateless session bean's method over and over again.

The computer is a 2260 MHz Pentium 4, IBM JDK 1.3.0 used for both WebSphere and JBoss. JBoss 2.4.4 with Catalina. Testing locally, but from different JVM's:

- When keeping a reference to the bean (i.e., a single JNDI lookup, then keep calling the methods):

WebSphere 4.01: 9.313s (~ 1074 calls / sec)
JBoss: 19.239s (~ 520 / Sec)

- With a JNDI lookup each time (which is closer to what our clients will do.)

WebSphere 4.01: 22.766s (~ 439 / Sec)
JBoss: 70.297s (~ 142 / Sec)

Just for curiosity sake, I also tested calling the EJB's over the network, on an Ultra 10 machine. (Which is even closer to the experience our clients will have.) Sadly, I don't have a WebSphere installed on that machine at the moment, so this is more to illustrate the effect of having a network in between.

With only 1 JNDI lookup: 77.641s (~ 131 / Sec)
With JNDI lookup each time: 298.391s (~ 33.5 / Sec)

Sorry, IMHO that is a bit too slow for an enterprise environment. Even at the rate of one EJB call per web page view, 33.5 calls per second might just be enough for someone's home page. And even then, only as long as it doesn't get too popular. For something that's supposed to be the core system called by all of a corporation's web services... nope.

Doubly so when a more realistic scenario involves 3-5 EJB calls per page.

[moraelin]

OK, just for comparison sake, I found an identical Ultra with a WebSphere 4.04 installed. So when calling the same test EJB over the network:

With only 1 JNDI lookup: 33,359 s (~ 300 / Sec)
With JNDI lookup each time: 55,938 s (~ 179 / Sek)

[joelvogt]

You would expect jrockit to be better for web logic.
It seems your test was run on different hotspot vms. I would be interested to know if there is any difference between _03 and _04.

As for the second test, jboss 2.4.4 isn't the best version to be testing with. What results do you get for a recent version?

[moraelin]

No idea. At the moment 2.4.4 is what we use for development, and I doubt anyone would let me install a 3.0 on those ultras.

Still, on the whole it kinda makes me wonder why the heck are we using EJB's at all. Even with WebSphere, it's ridiculous. Even WebSphere's 1000 local calls per second (for a method that only does a string lookup in a HashMap with 1 element, and returns true if found) on a 2.26 GHz machine with 1 GB DDR-RAM is just plain wrong. As an ex-assembly programmer, it hurts. It really hurts. I can't help but do the simple maths that that's more than 2 _million_ CPU cycles just for a short function call. It's nothing short of turning a modern computer into a Commodore 64 or ZX Spectrum.

So what problem were EJB's supposed to solve?

- Scalability? Well, nope. Web apps were always scalable to start with. That's what clustering and load balancing was for. Plus now we have, what? A scalable way to need 30 computers to do the work of 1 computer without EJB's?

- Distributed computing? Well, nope. If you look at the results with a network between client and server, they're a sight to make your eyes sore. The preached architecture with the app-server on a different machine than the web server, and a firewall in between, is just impractical at that kind of performance. (Or lack of performance.) And sure enough, everyone is either having their web server on the same machine, or inventing funny ways (like "astral projection") to _avoid_ calling over the network.

- Portability? Nope, either. Servlets and JSP's were portable to start with. EJB's only introduces the various proprietary quirks of IBM, BEA, Sun, JBoss and others into the mix. After having to develop something that ought to run on both JBoss and WebSphere, I can honestly say that EJB's just give you portability headaches you wouldn't have had without them.

- Security? Well, nope. After putting a nightmare couple of weeks of discovering exactly how JBoss's JAAS implementation works, and more importantly how JAAS doesn't even start to solve the kind of authorizing that we have to do... then came the realization that WebSphere does it in a completely different way. Which doesn't solve our problems either.

- Rapid development? Heck, that definitely not. At each step we're running into one more EJB container quirk that doesn't work in an obvious way. The various ClassLoader problems being at the top of the list, and singlehandedly responsible for a few man-months wasted so far. Plus we're wasting even more time to optimize this to get at least half-way decent speed, trying to work around the limitations and abysmal performance of EJB architecture.

So, ok, I'm dead curious now. Can anyone tell me exactly what _is_ the benefit of EJB's? I mean, other than allowing IBM and Sun to sell more expensive hardware? (Which I'm sure is a noble goal in and by itself.)

[jasonbrome]

A brief question - in these performance evaluations, what were the logging level settings configured for the different application servers?

[moraelin]

I'll have to check that out on monday, when I get back to the office.

Not sure why logging would make that huge an impact, though. Then again, one of our EJB's did mysteriously double in performance when the author removed a line which was causing about a dozen System.err.println. Was a bit weird that those were taking about as much time as building a DOM tree.

[jstrohm]

I agree, I'm currently working a small app just trying to learn EJB and it seems VERY slow. I'll write the whole thing and get it working with EJB but I may end up rewriting it just to get speed out of the thing.

Maybe I'm not thinking straight, but if I have a million row database and want to do a little processing with each row (entity bean), like update a field, stepping thru each record will take FOREVER. Or I could forgo EJB and simply do an update and be done pretty quickly.

[moraelin]

Oh, Entity Beans are a whole other dish as slow processing goes. If you use entity beans to, say, go through 10,000 records and print those whose date is in August and satisfy some other criteria, that's 10,001 SQL statements sent to the database. (And I'm not even talking about what happens when you have a couple of _million_ records, like one of our clients' databases had.) And even when you deal with just one row, unless you do unnatural tricks you end up doing one EJB call for each of its getter and setter methods, which go at the 1000 calls per second rate I talked about above.

Basically forget about entity beans. Whoever designed _that_ architecture, had one severe shortage of clue.

What I was more surprised about, though, were stateless session beans. I mean, that's basically more or less like a DLL when used locally.

[joelvogt]

well hang on don't go overboard ;) As far as logging goes, I seem to remeber that there is a very large performance boost from turning it down or off in jboss. Maybe as good as ten times(??). So give this a go.
As far as ejb's go, yes they can be slow for retrieving data. What your best to do in those situations is use jdbc as fast lane readers to get the data out (one sql query) and then use ejb's on the way in for the transactions and other goodies. You might consider this a failure of ejbs, but it works, which is the only real important thing.

[moraelin]

Hmm... Where do I find the logging settings, please? I found the logging settings for the data sources, and those are off, but that's not much of a difference anyway. I must confess I'm not much of an expert in configuring either JBoss or WebSphere.

Then again, the called function doesn't print anything, except on the very first of those 10,000 calls (the one that doesn't find the result in the HashMap.) I've looked on both the JBoss console and in the log files, and there is no output produced in this interval. (I think I would notice a block of 10,000 lines in a log file.)

On the other hand, WebSphere with the default settings seems to log everything but the kitchen sink. No, make that: everything _and_ the kitchen sink.

What _could_ make a difference is that my test is run through a JUnit task in an Ant script in Cygwin. (The test itself does print a counter, once every 128 calls.) Still, it's run through JUnit for both app servers, so I'd guess it's still apples-to-apples. Still, I'll try to make a short stand-alone program and run it directly in a DOS box, just to see how much of a difference does it make.

As for taking a fast lane with SQL, well, that's why we don't have any Entity Beans in this project. But, yes, whenever I have to spend more time coding work-arounds because a technology doesn't work as advertised, I do consider it a major failure of that technology. (E.g., having to code Session Bean facades to an Entity Bean.) Doubly so when the recoding isn't even a work-around, but a way to _avoid_ using that technology. (E.g., taking an SQL fast lane around it.)

[l.g.]

Are you using loading optimization?

[moraelin]

OK, turned out that I wasn't comparing as apples to apples as I thought. The local JBoss was configured to use Sun's JDK 1.3.1_04 _and_ to have JPDA enabled. Turned out those were the big performance killers.

So I measured again. This time directly in a DOS box, and to give the JIT a chance to optimize it, I ran each test 5 times in a loop in main().

With 1 JNDI lookup:

JBoss, local (Sun JDK 1.3.1_04, debug):
18.860 s
17.875 s
17.985 s
17.875 s
17.844 s
JBoss, local (Sun JDK 1.3.1_04, non-debug):
14.329 s
10.609 s
10.329 s
10.250 s
10.281 s
JBoss, local (IBM JDK 1.3.0):
10.531 s
9.641 s
9.641 s
9.641 s
9.625 s
WebSphere, local:
9.547 s
8.750 s
8.765 s
8.797 s
8.797 s

With 1 lookup per call:

JBoss, local (Sun JDK 1.3.1_04, debug):
68.437 s
66.547 s
66.484 s
66.500 s
66.516 s
JBoss, local (Sun JDK 1.3.1_04, non-debug):
45.688 s
39.859 s
39.516 s
39.438 s
39.344 s
JBoss, local (IBM JDK 1.3.0):
38.047 s
35.984 s
36.047 s
36.031 s
36.031 s
WebSphere, local:
20.844 s
19.250 s
19.187 s
19.218 s
19.265 s

The over-the-network calls were apples to apples, though, since both ran on a Sun Ultra, both with Sun's JDK (IBM has no JDK for Sun machines), and the JBoss there was not configured to use JPDA. So I didn't run those again.

It seems that having JPDA on was the biggest performance killer. It also seems that IBM's JDK is quite a bit faster than Sun's.

[moraelin]

Just wanted to add something: I'm still disappointed in the whole EJB architecture nevertheless. While JBoss does get closer to WebSphere in one unnatural case (get the EJB remote only once, call one of its methods again and again), I wasn't impressed with WebSphere's performance either. Over 2 million CPU cycles just to call an EJB method is one major waste of CPU, memory, time and money.

To get a comparison, I've benchmarked a lookup in the exact same cache object as in the test above, only this time pre-populated with 20,000 values. (In the test it only had one value in it: the one we were looking up.) And without the EJB interface.

The result? It ran 2,000,000 synchronized lookups in 0.86 seconds. That's about 1500 times faster than the same operation over an EJB remote interface. 'Nuff said.