2 Replies Latest reply on Dec 2, 2010 11:49 AM by basheeruddin

    Failed to fill batch | Could not locate data file.

    surajsathyapal
    surajsathyapal Nov 23, 2010 6:46 PM

      Hi Team,

       

      We are running load test on a apache-activemq-5.4.1-fuse-01-00 message broker with master/slave shared file system on NFS. The load is around 150+ messages/sec. The messages are 100Kb messages. We have producers and consumer configured through Spring. Yesterday we got the following error.

       

       

      2010-11-23 08:57:41,977 | ERROR | Looking for key 32672 but not found in fileMap: {33051=db-33051.log number = 33051 , length = 33617685, 33050=db-33050.log number = 33050 , length = 33638462, 33049=db-33049.log number = 33049 , length = 33638530, 33048=db-33048.log number = 33048 , length = 33638602, 33055=db-33055.log number = 33055 , length = 33617686, 33054=db-33054.log number = 33054 , length = 33638528, 33053=db-33053.log number = 33053 , length = 33617703,??.

      2010-11-23 08:57:41,993 | ERROR | Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:com.cisco.ecp.vdl.jms.load.queuejava.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)        at org.apache.kahadb.journal.Journal.read(Journal.java:592)        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:788)        at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:988)        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)        at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)         at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)         at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678)        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)         at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)2010-11-23 08:57:41,994 | ERROR | Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:com.cisco.ecp.vdl.jms.load.queue java.lang.RuntimeException: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678)         at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)Caused by: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)        at org.apache.kahadb.journal.Journal.read(Journal.java:592)        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:788)        at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:988)        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)         at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)         at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)        at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)        ... 7 more

      2010-11-23 08:57:41,994 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue | Queue:com.cisco.ecp.vdl.jms.load.queuejava.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)        at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1678)        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)        ... 6 moreCaused by: java.io.IOException: Could not locate data file /appdata/jms/data/kahadb/db-32672.log        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)        at org.apache.kahadb.journal.Journal.read(Journal.java:592)        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:788)        at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:988)        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)        at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)        at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)        at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)        at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)        ... 7 more

       

      Once this error happened we noticed the log files were not getting cleaned. The broker will not restart unless the kahadb folder is cleaned.

      The broker tag in activemq.xml is as below. (activemq.xml attached along)

       

       

       

      This issue looks similar to https://issues.apache.org/activemq/browse/AMQ-1795 which seems to be resolved. Please let me know how to fix for this issue.

       

      The test has been running for more than 10hrs when this error occurred.

       

      Thanks,

      Suraj.

       

      Edited by: surajsathyapal on Nov 23, 2010 11:08 PM

       

      Edited by: surajsathyapal on Nov 23, 2010 11:10 PM

       

      Edited by: surajsathyapal on Nov 23, 2010 11:45 PM

      • 1051 Views
      • Tags: