5 Replies Latest reply on Nov 17, 2004 2:03 AM by genman

    UIL2 loadtesting and threads

    amayingenta

      Hi,
      I've been trying to run some load tests against a remote server, and I'm finding that JBoss is using a lot of threads (and eventually can't create any more).

      I have a MDB that's listening on a local queue, and replying to messages via the destination I'm setting as the JMSReplyTo on the Message. The work the MDB does is usually taking about 100ms, and both the request and reply are ObjectMessages.

      My client application loops around sending a message, then waiting for the reply, so there is never more than one active MDB. The intention is that the ReplyTo destination would be a TemporaryQueue used for a single call, but I've tested with using a statically declared Queue on the remote machine and I get the same problem.

      Each time I send and receive a Message there seem to be two threads created on the JBoss server. These threads seem to disappear after a while (30 seconds or something of that order). If I send the messages slowly this isn't a problem, but if I keep sending another message as soon as I get a reply then I soon have hundreds of threads, and eventually I run out of threads somewhere > 4000.

      Looking at thread dumps the threads are doing:

      "Thread-296" daemon prio=1 tid=0x089252d8 nid=0x4b36 in Object.wait() [60b9c000..60b9c87c]
       at java.lang.Object.wait(Native Method)
       - waiting on <0x475baa48> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
       at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.poll(SynchronousChannel.java:353)
       - locked <0x475baa48> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.getTask(PooledExecutor.java:723)
       at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:747)
       at java.lang.Thread.run(Thread.java:534)
      


      I'm using JBoss 3.2.5 (same problem with 3.2.3) on Mandrake 10.0 with Java 1.4.2_05. There's nothing else running on the server.

      Here's a portion of my test case (in this version I'm using a statically declared Queue for the replies instead of a TemporaryQueue):
       QueueConnection con = factory.createQueueConnection();
       con.start();
       QueueSession session = con.createQueueSession(false, QueueSession.AUTO_ACKNOWLEDGE);
       QueueSender sender = session.createSender(queue);
       sender.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
      
       QueueReceiver receiver = session.createReceiver(replyQueue);
      
       int i = 0;
       while (System.currentTimeMillis() - start < duration)
       {
       Query query = (Query)searches.get(i);
       Message message = session.createObjectMessage(query);
       message.setJMSReplyTo(replyQueue);
       long searchStart = System.currentTimeMillis();
       sender.send(message);
       ObjectMessage reply = (ObjectMessage)receiver.receive(60000);
       Long time = new Long(System.currentTimeMillis() - searchStart);
       }
      


      I had a similar test working with messaging inside the same JBoss server (between a session bean and the MDB using the JVM-IL) and I did not have this problem.

      I'd appreciate any suggestions you can offer.

      Thanks,

      Andrew

        • 1. Re: UIL2 loadtesting and threads
          genman


          Do you close your connections when done?

          • 2. Re: UIL2 loadtesting and threads
            amayingenta

            The connection gets closed after the loop finishes - when I'm running this test with a single thread there's only ever one connection open.

            The duration of the test is usually a minute - in which time it can send and receive about 2000 messages. And each time it sends/receives a message over the UIL2 IL it's creating 2 temporary threads which seem to last for a while - so it's often not finishing the loop because that can be up to about 4000 threads it's trying to create.

            -Andrew

            • 3. Re: UIL2 loadtesting and threads
              nzmalik

              How about closing the session and creating a new one periodically? Just a guess on my end, no experience with it, but it might help if it doesn't add too much overhead.

              • 4. Re: UIL2 loadtesting and threads

                For every message that comes into the JMS Server via UIL2, the SocketManager seems to be starting two threads: One ReadTask and another WriteTask. So, you are seeing two threads for every message.

                Your test is a good one. But for every run, you are creating the following:
                1) Connection
                2) Session
                3) Sender
                4) Receiver

                This is causing in a proliferation of the number of threads that have to be created by the JMS Server.

                Try caching the above 4 objects (for a given queue) once you have created them the first time. From the second time onwards, keep using the cached objects. This might alleviate your problem.

                For ex:

                class ConnectionInfoHolder
                {
                 private Connection conn = null;
                 private Session session = null;
                 ....
                }
                
                class Client
                {
                 Map map = new Hashtable();
                 public void test()
                 {
                 //create all of them
                 map.put("queueName",connInfoHolder);
                
                 if(map.contains("queueName"))
                 //do your testing
                 }
                }


                HTH

                Thanks,
                Kalyan

                • 5. Re: UIL2 loadtesting and threads
                  genman


                  I think what is suggested by the JMS architecture is this:

                  1. Create a single connection to JBoss from the connection factory per Java process
                  2. For each thread that you use, create a separate Session
                  3. For each session, you can create as many queue reader and senders as you like. Typically, there is no reason to create more than one of each per queue (per thread).