Scaling UP tests

[clebert.suconic]

I have been trying to scale up few tests with nonTransactionSends/Persistent (that means.. every send requires transaction), and this scaled pretty nicely with AIO.

With NIO I've got about 350 sends / second with a single perfSender, because of the syncs on every send.

With AIO I've got 1200 sends /s econd with a single perfSender.
I've also tested with 5 perfSender, and I've got the same rate on every perfSender. That means.. it scaled without any problem.


*However* when I added a listener, things were much different.


The rate on the send was terrible, and I got a bunch of threads waiting on this condition:

 [java] "pool-7-thread-4" prio=1 tid=0x00007fb8e96d56c0 nid=0x3d44 waiting for monitor entry [0x000000004396f000..0x000000004396fb20]
 [java] at org.jboss.messaging.core.server.impl.QueueImpl.addLast(QueueImpl.java:159)
 [java] - waiting to lock <0x00007fb8f0a62968> (a org.jboss.messaging.core.server.impl.QueueImpl)
 [java] at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:321)
 [java] at org.jboss.messaging.core.server.impl.ServerProducerImpl.send(ServerProducerImpl.java:103)
 [java] at org.jboss.messaging.core.server.impl.ServerProducerPacketHandler.doHandle(ServerProducerPacketHandler.java:61)
 [java] at org.jboss.messaging.core.server.impl.ServerPacketHandlerSupport.handle(ServerPacketHandlerSupport.java:48)
 [java] at org.jboss.messaging.core.remoting.impl.PacketDispatcherImpl.dispatch(PacketDispatcherImpl.java:139)
 [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.messageReceivedInternal(MinaHandler.java:283)
 [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.access$000(MinaHandler.java:32)
 [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$1.run(MinaHandler.java:165)
 [java] at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:70)
 [java] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [java] at java.lang.Thread.run(Thread.java:595)
 [java]

While this thread was holding the lock:

[java] "pool-7-thread-1" prio=1 tid=0x00007fb8e9870bc0 nid=0x3d33 in Object.wait() [0x00000000413a4000..0x00000000413a4da0]
 [java] at java.lang.Object.wait(Native Method)
 [java] - waiting on <0x00007fb8f09b0528> (a org.jboss.messaging.core.remoting.impl.mina.MinaHandler)
 [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.checkWrite(MinaHandler.java:210)
 [java] - locked <0x00007fb8f09b0528> (a org.jboss.messaging.core.remoting.impl.mina.MinaHandler)
 [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$2.send(MinaHandler.java:251)
 [java] at org.jboss.messaging.core.server.impl.DeliveryImpl.deliver(DeliveryImpl.java:80)
 [java] at org.jboss.messaging.core.server.impl.ServerSessionImpl.handleDelivery(ServerSessionImpl.java:222)
 [java] at org.jboss.messaging.core.server.impl.ServerConsumerImpl.handle(ServerConsumerImpl.java:217)
 [java] - locked <0x00007fb8f0a62a90> (a java.lang.Object)
 [java] at org.jboss.messaging.core.server.impl.QueueImpl.deliver(QueueImpl.java:631)
 [java] at org.jboss.messaging.core.server.impl.QueueImpl.add(QueueImpl.java:528)
 [java] at org.jboss.messaging.core.server.impl.QueueImpl.addLast(QueueImpl.java:159)
 [java] - locked <0x00007fb8f0a62968> (a org.jboss.messaging.core.server.impl.QueueImpl)
 [java] at org.jboss.messaging.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:321)
 [java] at org.jboss.messaging.core.server.impl.ServerProducerImpl.send(ServerProducerImpl.java:103)
 [java] at org.jboss.messaging.core.server.impl.ServerProducerPacketHandler.doHandle(ServerProducerPacketHandler.java:61)
 [java] at org.jboss.messaging.core.server.impl.ServerPacketHandlerSupport.handle(ServerPacketHandlerSupport.java:48)
 [java] at org.jboss.messaging.core.remoting.impl.PacketDispatcherImpl.dispatch(PacketDispatcherImpl.java:139)
 [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.messageReceivedInternal(MinaHandler.java:283)
 [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler.access$000(MinaHandler.java:32)
 [java] at org.jboss.messaging.core.remoting.impl.mina.MinaHandler$1.run(MinaHandler.java:165)
 [java] at org.jboss.messaging.util.OrderedExecutorFactory$ChildExecutor.run(OrderedExecutorFactory.java:70)
 [java] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
 [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
 [java] at java.lang.Thread.run(Thread.java:595)
 [java]

I haven't taken any conclusions about what's wrong as I haven't looked on the code yet. These are just figures from my test.

[clebert.suconic]

to complete the thread:


5 perfSenders running as:

ant -Ddelivery.mode=PERSISTENT -Dsess.trans=false perfSender

and 1 perfListener running as:

ant -Dsess.trans=true -Dsess.trans.size=1000 -Dmessage.count=1000000 perfListener

That is... 5 non Transactioned senders.. and 1 transactioned listener.

[clebert.suconic]

As we have talked on the IRC, there is a task to improve concurrency on QueueImpl. Running the test with a single Sender and a single Listener was okay.

[clebert.suconic]

Another thing.. about the 300 msg/second with NIO.

I have a workstation with a SCSI (or as they call it now.. SAS) disk... and that is very fast at write, and it seems slow at sync. (maybe something on the OS?) That's why I got this rate of 350 messages / second. On the laptop where I have a regular disk I got about 1000 messages / second.

[timfox]

Another thing.. about the 300 msg/second with NIO.

I have a workstation with a SCSI (or as they call it now.. SAS) disk... and that is very fast at write, and it seems slow at sync. (maybe something on the OS?) That's why I got this rate of 350 messages / second. On the laptop where I have a regular disk I got about 1000 messages / second.

Maybe because it actually syncs! On some cheaper disks/OS combinations when you sync it doesn't actually sync, just writes to the disks cache (note not the OS cache) for performance:

http://www.sr5tech.com/write_back_cache_experiments.htm