HornetQ slow when using filters?
ebuddy May 31, 2010 10:54 AMHi,
I have build an application that runs a distributed HornetQ server. Initially my plan was to have one queue / address where clients can send messages to and then have different consumers read from this queue applying filters to only obtain the data that they want.
The message I am sending has a string payload and two properties: type (String) and success (boolean). These are the properties I want to filter on for my consumers.
The queues I'm using are all non-durable
When I run my setup with the filters on the consumers I get a throughput of roughly 800 messages per second on my machine (4 core / 8 gig, running the app with a 2 Gig Heap on SUN jvm 1.6.0_18).
Not satisfied with this performance I tried to optimize some things and it turns out that if I split the messages on the client (i.e. have one queue per consumer without using filters) I get 10.000+ messages per second with the same setup!
Now I cannot believe that applying a filter would have such a big impact on the HornetQ. Can anybody shed some light on this. Also in the future I would like to use HornetQ in a more elaborate setup where I would really need to rely on filters to work and perform correctly.
Please find attached my hornetQConfig (I wire it up with spring). This is the one with the good performance:
<bean id="hornetQConfig"><property name="acceptorConfigurations"><set value-type="org.hornetq.api.core.TransportConfiguration"><bean><constructor-arg index="0" value="org.hornetq.integration.transports.netty.NettyAcceptorFactory"/><constructor-arg index="1"><map key-type="java.lang.String" value-type="java.lang.Object"><entry key="host" value="${hornetq.acceptor.host}"/><entry key="port" value="${hornetq.acceptor.port}"/></map></constructor-arg></bean><bean><constructor-arg index="0" value="org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory"/></bean></set></property><property name="threadPoolMaxSize" value="8"/><property name="JMXManagementEnabled" value="true"/><property name="securityEnabled" value="false"/><property name="queueConfigurations"><list value-type="org.hornetq.core.server.cluster.QueueConfiguration"><bean><constructor-arg index="0" value="eBuddy.events.SECURITY.login.success"/><constructor-arg index="1" value="eBuddy.events.SECURITY.login.success"/><constructor-arg index="2" value=""/><constructor-arg index="3" value="false"/></bean><bean><constructor-arg index="0" value="eBuddy.events.SECURITY.login.failure"/><constructor-arg index="1" value="eBuddy.events.SECURITY.login.failure"/><constructor-arg index="2" value=""/><constructor-arg index="3" value="false"/></bean><bean><constructor-arg index="0" value="eBuddy.events.TEST"/><constructor-arg index="1" value="eBuddy.events.TEST"/><constructor-arg index="2" value=""/><constructor-arg index="3" value="false"/></bean></list></property></bean>
And this is the one with the filters and the subsequent bad performance (actually I get and OutOfMemory on this one while processing my same testset of 500.000 messages)
<bean id="hornetQConfig" class="org.hornetq.core.config.impl.ConfigurationImpl"> <property name="acceptorConfigurations"> <set value-type="org.hornetq.api.core.TransportConfiguration"> <bean class="org.hornetq.api.core.TransportConfiguration"> <constructor-arg index="0" value="org.hornetq.integration.transports.netty.NettyAcceptorFactory"/> <constructor-arg index="1"> <map key-type="java.lang.String" value-type="java.lang.Object"> <entry key="host" value="${hornetq.acceptor.host}"/> <entry key="port" value="${hornetq.acceptor.port}"/> </map> </constructor-arg> </bean> <bean class="org.hornetq.api.core.TransportConfiguration"> <constructor-arg index="0" value="org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory"/> </bean> </set> </property> <property name="threadPoolMaxSize" value="8"/> <property name="JMXManagementEnabled" value="true"/> <property name="securityEnabled" value="false"/> <property name="queueConfigurations"> <list value-type="org.hornetq.core.server.cluster.QueueConfiguration"> <bean class="org.hornetq.core.server.cluster.QueueConfiguration"> <constructor-arg index="0" value="eBuddy.events"/> <constructor-arg index="1" value="eBuddy.events"/> <constructor-arg index="2" value=""/> <constructor-arg index="3" value="false"/> </bean> <bean class="org.hornetq.core.server.cluster.QueueConfiguration"> <constructor-arg index="0" value="eBuddy.events"/> <constructor-arg index="1" value="eBuddy.events.SECURITY.login.success"/> <constructor-arg index="2" value="type = 'SECURITY' AND success = true"/> <constructor-arg index="3" value="false"/> </bean> <bean class="org.hornetq.core.server.cluster.QueueConfiguration"> <constructor-arg index="0" value="eBuddy.events"/> <constructor-arg index="1" value="eBuddy.events.SECURITY.login.failure"/> <constructor-arg index="2" value="type = 'SECURITY' AND success = false"/> <constructor-arg index="3" value="false"/> </bean> <bean class="org.hornetq.core.server.cluster.QueueConfiguration"> <constructor-arg index="0" value="eBuddy.events"/> <constructor-arg index="1" value="eBuddy.events.TEST"/> <constructor-arg index="2" value="type = 'TEST'"/> <constructor-arg index="3" value="false"/> </bean> </list> </property> </bean>
Any help / insights would be appreciated.
Thanx
Joost