Failed to fill batch | Could not locate data file.
surajsathyapal Nov 23, 2010 6:46 PMHi Team,
We are running load test on a apache-activemq-5.4.1-fuse-01-00 message broker with master/slave shared file system on NFS. The load is around 150+ messages/sec. The messages are 100Kb messages. We have producers and consumer configured through Spring. Yesterday we got the following error.
2010-11-23 08:57:41,977 | ERROR | Looking for key 32672 but not found in fileMap: {33051=db-33051.log number = 33051 , length = 33617685, 33050=db-33050.log number = 33050 , length = 33638462, 33049=db-33049.log number = 33049 , length = 33638530, 33048=db-33048.log number = 33048 , length = 33638602, 33055=db-33055.log number = 33055 , length = 33617686, 33054=db-33054.log number = 33054 , length = 33638528, 33053=db-33053.log number = 33053 , length = 33617703,??.
2010-11-23 08:57:41,993 | ERROR | Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:com.cisco.ecp.vdl.jms.load.queuejava.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) at org.apache.kahadb.journal.Journal.read(Journal.java:592) at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:788) at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:988) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) at org.apache.kahadb.page.Transaction.execute(Transaction.java:728) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110) at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678) at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)2010-11-23 08:57:41,994 | ERROR | Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:com.cisco.ecp.vdl.jms.load.queue java.lang.RuntimeException: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110) at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678) at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)Caused by: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) at org.apache.kahadb.journal.Journal.read(Journal.java:592) at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:788) at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:988) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) at org.apache.kahadb.page.Transaction.execute(Transaction.java:728) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) ... 7 more
2010-11-23 08:57:41,994 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:com.cisco.ecp.vdl.jms.load.queuejava.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113) at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157) at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678) at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110) ... 6 moreCaused by: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) at org.apache.kahadb.journal.Journal.read(Journal.java:592) at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:788) at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:988) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) at org.apache.kahadb.page.Transaction.execute(Transaction.java:728) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) ... 7 more
Once this error happened we noticed the log files were not getting cleaned. The broker will not restart unless the kahadb folder is cleaned.
The broker tag in activemq.xml is as below. (activemq.xml attached along)
This issue looks similar to https://issues.apache.org/activemq/browse/AMQ-1795 which seems to be resolved. Please let me know how to fix for this issue.
The test has been running for more than 10hrs when this error occurred.
Thanks,
Suraj.
Edited by: surajsathyapal on Nov 23, 2010 11:08 PM
Edited by: surajsathyapal on Nov 23, 2010 11:10 PM
Edited by: surajsathyapal on Nov 23, 2010 11:45 PM
-
activemq.xml 3.0 KB