2 Replies Latest reply on Sep 12, 2005 11:14 AM by Clebert Suconic

    JMS automated perf. testing framework

    Tim Fox Master

      I've been working on a JMS performance testing framework for the last few days - it's about 75% complete.

      I had a look at what was available on the market:

      IBM provide a simple tool, but it's quite limited in functionality and has no ability for co-ordinating multiple clients.
      Also, Sonic provide one, but again limited functionality.
      JMeter also, but again limited functionality and no co-ordination of multiple clients.

      The one I've knocked together works something like the following:

      A simple server runs at all the machines that you want to use as JMS clients. This server uses JBoss remoting to listen for requests.

      Each server can run multiple jobs.

      A job is one of:

      Sender
      ------

      A sender send messages to a destination.
      For a sender the following params are configurable:

      Number of jms connections to use
      Number of jms sessions to use (connections are allocated to sessions on a round robin basis)
      Ramp delay (delay between starting threads when there is more than one thread)
      Run length - how long to run the sender for
      Destination - the jms destination
      Transacted - whether the sending session is transacted
      Transaction size - if transacted then how many sends per tx
      Anon - Use an anonymous producer?
      Message Size- message size
      Message type - any of the JBoss message types, plus a foreign message
      Delivery mode - persistent or non persistent

      Receiver
      --------

      A receiver receives messages from a destination
      For a receiver the following params are configurable:

      Number of jms connections to use
      Number of jms sessions to use (connections are allocated to sessions on a round robin basis)
      Ramp delay (delay between starting threads when there is more than one thread)
      Run length - how long to run the sender for
      Destination - the jms destination
      Transacted - whether the sending session is transacted
      Transaction size - if transacted then how many acks per tx
      Acknowledgement Mode
      Durable subscription name
      Message Selector
      No Local?
      Use message listener rather than synch. receive?

      Browser
      -------

      A browser job repeatedly browses messages in a destination (TODO)


      A server can run any number of jobs at any one time

      The server measures the throughput in msgs/s for each job at predetermined intervals.

      Another application, the controller is responsible for sending requests to the servers.

      Requests are as follows:

      Start job on server
      Stop job on server
      Get current data for job on server (only throughput currently supported).
      Throttle a send job.


      Depending on the particular test to be done, the controller communicates remotely with the servers telling them to start or stop jobs.

      E.g. the controller could tell servers A, B and C to start send jobs to queue Q, and tell server D to start a receive job on queue Q.

      The controller would then intermittently communicate with the servers to get current throughput (msgs sent/received) for each server.

      If we want to measure max. throughput of the server for a particular configuration of senders and receivers, then we really need to reach a steady state, i.e. messages aren't being sent faster then they are being received causing them to build up on the server.

      Although this isn't currently implemented, this could be achieved by the server noticing that the msgs received per sec from the receivers is lower then the msgs sent per sec from the senders and telling the senders to slow down (throttle them). (and vice versa).

      In such a way we should home in on the steady state.

      Another important test IMHO is to test peak send rate. This is the max. number of sends the system can cope with when the sender is suddenly unthrottled from a steady state. This tests the systems ability to deal with spikes.

      (What other metrics are we interested in?)

      We need to think about what tests and configurations we're really interested in.

      Also, currently data is just persisted to a CSV file.

      It would be nice to store it in a database somewhere so a reporting tool can generate graphs and reports from it.


      -Tim

        • 1. Re: JMS automated perf. testing framework
          Adrian Brock Master

           

          "timfox" wrote:

          It would be nice to store it in a database somewhere so a reporting tool can generate graphs and reports from it.


          This is what hypersonic was designed for.

          Everybody else except Gavin would tell you to use Hibernate :-)

          • 2. Re: JMS automated perf. testing framework
            Clebert Suconic Master

            We are designing JRunit to capture information from Benchmarks, and save them to a simple output report or to the database.


            (http://labs.jboss.com/portal/index.html?ctrl:id=page.default.info&project=jrunit)


            Basically you have an API where you could start capturing the benchmark, and where you would stop capturing.


            Underneath is just capturing System.currentTimeMilliseconds(), and stabilishing the differences between your open and close statements, but the nice part is we are providing a single API in such way we could save data to database.

            We will also have a Web application which will provide ad-hoc queries over historical data as you described.

            We are writting documentation now, and we should have the documentation done by JBossWorld Barcelona.

            A good example you could check now, is the performance tests I (and Manik) have written to JBossCache. You could just copy/paste based on those examples.

            If you have any questions on how to use jrunit then you could post question on the Internal Benchmark Forum.


            Clebert