4 Replies Latest reply on May 24, 2008 2:09 AM by Tim Fox

    Scaling UP tests

    Clebert Suconic Master

      I have been trying to scale up few tests with nonTransactionSends/Persistent (that means.. every send requires transaction), and this scaled pretty nicely with AIO.

      With NIO I've got about 350 sends / second with a single perfSender, because of the syncs on every send.

      With AIO I've got 1200 sends /s econd with a single perfSender.
      I've also tested with 5 perfSender, and I've got the same rate on every perfSender. That means.. it scaled without any problem.


      *However* when I added a listener, things were much different.


      The rate on the send was terrible, and I got a bunch of threads waiting on this condition:


       [java] "pool-7-thread-4" prio=1 tid=0x00007fb8e96d56c0 nid=0x3d44 waiting for monitor entry [0x000000004396f000..0x000000004396fb20]
       [java] at org.jboss.messaging.core.server.impl.QueueImpl.addLast(QueueImpl.java:159)
       [java] - waiting to lock <0x00007fb8f0a62968> (a org.jboss.messaging.core.server.impl.QueueImpl)
       [java] at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:321)
       [java] at org.jboss.messaging.core.server.impl.ServerProducerImpl.send(ServerProducerImpl.java:103)
       [java] at org.jboss.messaging.core.server.impl.ServerProducerPacketHandler.doHandle(ServerProducerPacketHandler.java:61)
       [java] at org.jboss.messaging.core.server.impl.ServerPacketHandlerSupport.handle(ServerPacketHandlerSupport.java:48)
       [java] at org.jboss.messaging.core.remoting.impl.PacketDispatcherImpl.dispatch(PacketDispatcherImpl.java:139)
       [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.messageReceivedInternal(MinaHandler.java:283)
       [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.access$000(MinaHandler.java:32)
       [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$1.run(MinaHandler.java:165)
       [java] at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:70)
       [java] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       [java] at java.lang.Thread.run(Thread.java:595)
       [java]
      



      While this thread was holding the lock:


      [java] "pool-7-thread-1" prio=1 tid=0x00007fb8e9870bc0 nid=0x3d33 in Object.wait() [0x00000000413a4000..0x00000000413a4da0]
       [java] at java.lang.Object.wait(Native Method)
       [java] - waiting on <0x00007fb8f09b0528> (a org.jboss.messaging.core.remoting.impl.mina.MinaHandler)
       [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.checkWrite(MinaHandler.java:210)
       [java] - locked <0x00007fb8f09b0528> (a org.jboss.messaging.core.remoting.impl.mina.MinaHandler)
       [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$2.send(MinaHandler.java:251)
       [java] at org.jboss.messaging.core.server.impl.DeliveryImpl.deliver(DeliveryImpl.java:80)
       [java] at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleDelivery(ServerSessionImpl.java:222)
       [java] at org.jboss.messaging.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:217)
       [java] - locked <0x00007fb8f0a62a90> (a java.lang.Object)
       [java] at org.jboss.messaging.core.server.impl.QueueImpl.deliver(QueueImpl.java:631)
       [java] at org.jboss.messaging.core.server.impl.QueueImpl.add(QueueImpl.java:528)
       [java] at org.jboss.messaging.core.server.impl.QueueImpl.addLast(QueueImpl.java:159)
       [java] - locked <0x00007fb8f0a62968> (a org.jboss.messaging.core.server.impl.QueueImpl)
       [java] at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:321)
       [java] at org.jboss.messaging.core.server.impl.ServerProducerImpl.send(ServerProducerImpl.java:103)
       [java] at org.jboss.messaging.core.server.impl.ServerProducerPacketHandler.doHandle(ServerProducerPacketHandler.java:61)
       [java] at org.jboss.messaging.core.server.impl.ServerPacketHandlerSupport.handle(ServerPacketHandlerSupport.java:48)
       [java] at org.jboss.messaging.core.remoting.impl.PacketDispatcherImpl.dispatch(PacketDispatcherImpl.java:139)
       [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.messageReceivedInternal(MinaHandler.java:283)
       [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.access$000(MinaHandler.java:32)
       [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$1.run(MinaHandler.java:165)
       [java] at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:70)
       [java] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
       [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
       [java] at java.lang.Thread.run(Thread.java:595)
       [java]
      



      I haven't taken any conclusions about what's wrong as I haven't looked on the code yet. These are just figures from my test.

        • 1. Re: Scaling UP tests
          Clebert Suconic Master

          to complete the thread:


          5 perfSenders running as:


          ant -Ddelivery.mode=PERSISTENT -Dsess.trans=false perfSender
          


          and 1 perfListener running as:

          ant -Dsess.trans=true -Dsess.trans.size=1000 -Dmessage.count=1000000 perfListener
          


          That is... 5 non Transactioned senders.. and 1 transactioned listener.

          • 2. Re: Scaling UP tests
            Clebert Suconic Master

            As we have talked on the IRC, there is a task to improve concurrency on QueueImpl. Running the test with a single Sender and a single Listener was okay.

            • 3. Re: Scaling UP tests
              Clebert Suconic Master

              Another thing.. about the 300 msg/second with NIO.

              I have a workstation with a SCSI (or as they call it now.. SAS) disk... and that is very fast at write, and it seems slow at sync. (maybe something on the OS?) That's why I got this rate of 350 messages / second. On the laptop where I have a regular disk I got about 1000 messages / second.

              • 4. Re: Scaling UP tests
                Tim Fox Master

                 

                "clebert.suconic@jboss.com" wrote:
                Another thing.. about the 300 msg/second with NIO.

                I have a workstation with a SCSI (or as they call it now.. SAS) disk... and that is very fast at write, and it seems slow at sync. (maybe something on the OS?) That's why I got this rate of 350 messages / second. On the laptop where I have a regular disk I got about 1000 messages / second.


                Maybe because it actually syncs! On some cheaper disks/OS combinations when you sync it doesn't actually sync, just writes to the disks cache (note not the OS cache) for performance:

                http://www.sr5tech.com/write_back_cache_experiments.htm