Journaling errors
ronnys Jul 8, 2010 11:01 AMHi,
I've encountered a few strange errors in the HornetQ (2.1.1.final) log, such as
[pool-4-thread-1] 15:53:38,513 SEVERE [org.hornetq.core.journal.impl.JournalImpl] Inconsistency during compacting: Delete record being read on an existent record
java.lang.IllegalStateException: Inconsistency during compacting: Delete record being read on an existent record
at org.hornetq.core.journal.impl.JournalCompactor.onReadDeleteRecord(JournalCompactor.java:305)
at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:629)
at org.hornetq.core.journal.impl.JournalImpl.compact(JournalImpl.java:1517)
at org.hornetq.core.journal.impl.JournalImpl$5.run(JournalImpl.java:2315)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
[Thread-6 (group:HornetQ-server-threads1485988078-970926436)] 15:56:53,925 WARNING [org.hornetq.core.server.impl.QueueImpl] Unable to remove message id = 2201750 please remove manually
java.lang.IllegalStateException: Cannot find add info 2201750
at org.hornetq.core.journal.impl.JournalImpl.appendDeleteRecord(JournalImpl.java:913)
at org.hornetq.core.persistence.impl.journal.JournalStorageManager.deleteMessage(JournalStorageManager.java:514)
at org.hornetq.core.server.impl.QueueImpl.postAcknowledge(QueueImpl.java:1415)
at org.hornetq.core.server.impl.QueueImpl.access$400(QueueImpl.java:65)
at org.hornetq.core.server.impl.QueueImpl$RefsOperation.afterCommit(QueueImpl.java:1510)
at org.hornetq.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:294)
at org.hornetq.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:207)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
[Old I/O server worker (parentId: 2071272094, channelId: 1292454077, null => ttwsrvdev02.dev.ttw/10.115.151.40:61445)] 15:59:59,093 WARNING [org.hornetq.core.server.impl.QueueImpl] Error on checkDLQ
java.lang.IllegalStateException: Cannot find add info 3963098
at org.hornetq.core.journal.impl.JournalImpl.appendUpdateRecord(JournalImpl.java:841)
at org.hornetq.core.persistence.impl.journal.JournalStorageManager.updateDeliveryCount(JournalStorageManager.java:689)
at org.hornetq.core.server.impl.QueueImpl.checkDLQ(QueueImpl.java:946)
at org.hornetq.core.server.impl.QueueImpl$RefsOperation.afterRollback(QueueImpl.java:1471)
at org.hornetq.core.transaction.impl.TransactionImpl$3.done(TransactionImpl.java:362)
at org.hornetq.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:158)
at org.hornetq.core.persistence.impl.journal.JournalStorageManager.afterCompleteOperations(JournalStorageManager.java:393)
at org.hornetq.core.transaction.impl.TransactionImpl.rollback(TransactionImpl.java:344)
at org.hornetq.core.server.impl.ServerConsumerImpl.close(ServerConsumerImpl.java:296)
at org.hornetq.core.server.impl.ServerSessionImpl.closeConsumer(ServerSessionImpl.java:931)
at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:447)
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:471)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:451)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:412)
at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)
at org.hornetq.core.remoting.impl.netty.HornetQChannelHandler.messageReceived(HornetQChannelHandler.java:67)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:100)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:754)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:287)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.decode(HornetQFrameDecoder2.java:169)
at org.hornetq.core.remoting.impl.netty.HornetQFrameDecoder2.messageReceived(HornetQFrameDecoder2.java:134)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:545)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:540)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.oio.OioWorker.run(OioWorker.java:90)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.IoWorkerRunnable.run(IoWorkerRunnable.java:46)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
The log file contains hundreds of these errors. HornetQ is running on Linux with enabled AIO; standalone, unclustered
installation, data directory on local ext3 FS. A separate load test (source attached) can be used to reproduce them
within a few minutes when running it on a fresh HornetQ installation with empty data directory. Issues appears with
NIO journal as well. Config files are attached.
In order to reproduce the issue start the load test with the following options
org.jnp.interfaces.NamingContextFactory jnp://yourserver:yourport ConnectionFactory queue/inbound_messages 3600000 100000 25000 5 25 10 2048 5000 1000000
Keep it running for 5min or so.
Best regards,
Ronny
-
LoadTest.java.zip 3.1 KB
-
hornetq-users.xml 341 bytes
-
hornetq-jms.xml 2.1 KB
-
hornetq-configuration.xml 4.2 KB
-
hornetq-beans.xml 2.0 KB