Messages get lost from the queue!!
janvandeklok Jan 14, 2013 5:40 AMHello there,
We have been using hornetq messaging for over a year now without any problems. We are using the embedded version in JBoss 6.0.1_final.
Up to now we had about 120 queues that had 1 consumer each and that are capable of paging. We have tested some of these queues to contain over 140.000 messages with no problem at all.
Now we have change it to about 700 queues using paging. We have loaded a lot of messages in the system and ran into a problem that we can't figure out.
Our pages on disk are about 2mb (each page). On several queue's the messageCounter states that there are still a number of messages on the queue, however there is just 1 diskpage on disk the has a size of 0 bytes and the listMessages call on the queue Bean using the jboss console has 0 messages!!!!!!!
In our database log we still see that the number of messages that are not processed equals the number of the messageCounter value. In other words we have lost a numer of messages.
We checked the logging of the jboss app server but could not find errors that could indicate such a failure. Evenso, We process each message in a seperate transaction an do commit / rollbacks depending on the processing result.
- What is happening here?
- Has someone else ran into the same problems?
- Could someone give me a clue where to look for the source of the problem?
- What will happen when the app server will hit its Xmx limit due to the in-memory messages for each of the 700 queues for instance??
- We do see an HornetQException[errorCode=3 message=Timed out waiting for response when sending packet 43 error after wich we do a roll back for that single transaction by the way, but IMHO this should not cause loss of messages.
Our appserver runs with these memory settings : -Xms3072M -Xmx4096M -XX:MaxPermSize=256m
The Jboss appserver itself and the deployed apps do not use more than 1.2 GB so that will leave at least 1.8 GB for the message in memory. Will this be enough for 700 queue's where the disk page size ??
Any help is greatly appreciated!!!!
Jan van de Klok
Her is our queue configuration:
<configuration xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">
<!--
Don't change this name.
This is used by the dependency framework on the deployers,
to make sure this deployment is done before any other deployment
-->
<name>HornetQ.main.config</name>
−
<log-delegate-factory-class-name>
org.hornetq.integration.logging.Log4jLogDelegateFactory
</log-delegate-factory-class-name>
<bindings-directory>${jboss.server.data.dir}/hornetq/bindings</bindings-directory>
<journal-directory>${jboss.server.data.dir}/hornetq/journal</journal-directory>
−
<journal-file-size>${hornetq.journal.file.size:1048576}</journal-file-size>
−
<journal-min-files>${hornetq.journal.min.files:2}</journal-min-files>
<large-messages-directory>${jboss.server.data.dir}/hornetq/largemessages</large-messages-directory>
<paging-directory>${jboss.server.data.dir}/hornetq/paging</paging-directory>
<!-- true to expose HornetQ resources through JMX -->
<jmx-management-enabled>true</jmx-management-enabled>
−
<connectors>
−
<connector name="netty">
−
<factory-class>
org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
</factory-class>
<param key="host" value="${jboss.bind.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.port:5445}"/>
</connector>
−
<connector name="netty-throughput">
−
<factory-class>
org.hornetq.core.remoting.impl.netty.NettyConnectorFactory
</factory-class>
<param key="host" value="${jboss.bind.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.batch.port:5455}"/>
<param key="batch-delay" value="50"/>
</connector>
−
<connector name="in-vm">
−
<factory-class>
org.hornetq.core.remoting.impl.invm.InVMConnectorFactory
</factory-class>
<param key="server-id" value="${hornetq.server-id:0}"/>
</connector>
</connectors>
−
<acceptors>
−
<acceptor name="netty">
−
<factory-class>
org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory
</factory-class>
<param key="host" value="${jboss.bind.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.port:5445}"/>
</acceptor>
−
<acceptor name="netty-throughput">
−
<factory-class>
org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory
</factory-class>
<param key="host" value="${jboss.bind.address:localhost}"/>
<param key="port" value="${hornetq.remoting.netty.batch.port:5455}"/>
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</acceptor>
−
<acceptor name="in-vm">
−
<factory-class>
org.hornetq.core.remoting.impl.invm.InVMAcceptorFactory
</factory-class>
<param key="server-id" value="0"/>
</acceptor>
</acceptors>
−
<security-settings>
−
<security-setting match="#">
<permission type="createDurableQueue" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="send" roles="guest"/>
<permission type="manage" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="jms.queue.studielink.delivery.#">
<max-delivery-attempts>-1</max-delivery-attempts>
<!-- -1 means no re-delivery-attempt limit -->
<page-size-bytes>2097152</page-size-bytes>
<!-- size of disk pages 2Mb -->
<max-size-bytes>3145728</max-size-bytes>
<address-full-policy>PAGE</address-full-policy>
</address-setting>
</address-settings>
</configuration>