3 Replies Latest reply on May 6, 2003 10:39 PM by jonlee

    Discussion about results of Web Services Benchmark

    tobias

      Hi!
      There is a benchmark about Web Services at:
      http://www.doculabs.com/Downloads/WebServicePerformance_04-03.pdf

      I don't have the time to read all 120 pages. But several things struck me when browsing the report.

      On page 28 you see a chart of J2EE and .NET performance results. It's nice that JBoss is the fastest J2EE app server in this chart. But two things look odd to me: .NET performance increases by 28 % by changing the OPERATING SYSTEM. And then again it increases again by 38% by changing the database in the background.
      a) What do these performance tests test ? The OS ? The database performance ? Or the Web Service performance ?
      b) It's also interesting that they did not let J2EE use a faster database such as PostgreSQL. Why is that ?

      Then on page 37 look at the percentage in the "CPU Usage" column. It is at 76% for JBoss and 100% for the fast .NET results.
      c) If they really try to push the server to the limit, how come the CPU is not fully saturated ? What are they measuring here ? What is the bottle neck here ?

      I stopped reading then. Can someone explain me what all that means ?

      Cheers,
      Tobias

        • 1. Re: Discussion about results of Web Services Benchmark

          Caching was not allowed in this benchmark

          I think because it would make .NET look bad.

          m$ wouldn't allow the results to be published
          if they weren't the best.
          Read the windows EULA.

          Regards,
          Adrian

          • 2. Re: Discussion about results of Web Services Benchmark
            jonlee

            They indicate in the report that they understand that Web services are a facade to the underlying business process transaction, but then measure the totality of the system, rather than just the Web services layer.

            It is an interesting report because it tends to imply that if you move to a proprietary system, you get better results. Or conforming to open standards causes performance problems?

            • 3. Re: Discussion about results of Web Services Benchmark
              jonlee

              I was thinking of letting things lie with this topic. But I have to be fair. Benchmarking can be made to justify any particular spin someone wants to deliver. However, in the enterprise world that .NET and J2EE live in, there are factors that lie beyond the manufactured realm a benchmark can conjure. Connectivity with business IT policy, interaction with legacy systems, operation in a distributed environment and security are some of the challenging issues. And certain approaches are going to yield better results for one technology over the other.

              Web services are going to be one part of the business integration puzzle. But Web services are not producers of data, they are connection and transport services for accessing data. Data producers will in most businesses, still be the ERP and legacy systems that automate and perform the business processes. They will ultimately determine how fast you can transport data through web services, and the manner in which data is consumed will determine how fast and how much data you must actually deliver. If a particular company were to generate the number of bills represented by the transactions in the report, then that would be a good problem to have.