The Middleware Co. J2EE vs .NET benchmark

[cjohan]

The Middleware Company compared two unnamed J2EE application servers with Microsoft .NET using newly optimized versions of the Sun Java Petstore implemented in Java and C#. They report hugely better performance for the .NET implementation. The study doesn't say whether or not Microsoft paid for the study. Overall, they seemingly tried to give J2EE a good chance to perform well. I would have to look at the source code to see how well they did with the optimization. Haven't done that yet.

Let's try to understand why the performance was so different. The optimized Pet Shop source code is available for download.

Possibly, the number of threads allowed in the JVM or other configuration parameters are at least partially responsible. One interesting result was that they found the Sun 1.4 JRE much faster, which they attributed to concurrent garbage collection. They threw distributed transactions into the mix, which may have confused the basic issue of which technology is better. Authentication was done differently, which may have given some advantage to .NET.

http://www.middleware-company.com/j2eedotnetbench

http://www2.theserverside.com/home/thread.jsp?thread_id=16149

[kirkross]

Some responses to their conclusions can be found here:
http://dreambean.com/petstore.html

[juha]

Also should note Cameron's comments at the server side on the XA issue.

[marc.fleury]

what were they?

From what I understand of this benchmark this is a funny cache bench. Good for the microsoft guys for optimizing everything from cache. That is the way you really build applications. Everything else is wanking. Microsoft proves that and everyeone is crying foul. Good for microsoft and shame on the TSS crew for not optimizing everything to be in cache, that is how you build real architecture.

Enough righteous ranting. I just knew it from the beginning.

[juha]

Apparently the original petstore application was modified to use distributed, recoverable transactions. Each order from the store required an update to two separate databases within a single transaction.

So distributed transaction manager with XA was used for J2EE. NET used COM+ distributed transactions to achieve the same. XA is vendor neutral whereas COM+ is MS proprietary protocol and works only with MTS + SQL server instances.

So it would look like it was really a performance test where most time was spent comparing J2EE vendor's DTM implementation using XA with Microsoft MTS and SQL server proprietary distributed transactions.

So the latter ends up being quite a bit faster. Which is really not that surprising. It's really not that common scenario either so it doesn't say much about the true performance differences that might exist between J2EE app servers and MS .NET server.

[marc.fleury]

I would expect the DTM sync to take some time as there is dramatic serialization going on, but frankly the implementation can only be optimized a little bit. Meaning that the same amount of serious serialization must still go on. I will buy that a MS only world would be optimized there. I don't believe that it accounts for wide variations. Wide variations come when one is serializing the other not.

In this case it seems that there is serialization in the J2EE design with enforced web/ejb serialization where there is none with the MS bench and also more serialization in the lack of cache. 2 accounts where JBoss can help since the first one is optimized (but I can't believe BEA doesn't optimize this by default) and proper cache design is done by consulting. DTM is certainly a difference but seems present in both designs.

Do the numbers justify such a wide suckage?

If so I must say I am less that it smells like a rat.
1- either the server side guys are stupid as consultants. I mean this is beginner stuff.
2- either the server side guys are doing it on purpose and then it was a stupid move.

That being said, I am a big proponent of any publicity is good publicity so this move is maybe the smartest on their part.

Business is fucked up that way. You can put your ethics in the drain and you will be rewarded. Sometimes I feel silly for doing Free Software and wearing my ideals on my sleeve. We end up being taken either as communist by the business guys and as greedy bastards by the communists.

If it wasn't because I got the 2 babies and I can't sleep at night, I would say I would have problems sleeping at night if I was in the serverside shoes