7 Replies Latest reply on Aug 24, 2011 5:18 AM by fgynnild

Could not locate data file

fgynnild Aug 17, 2011 2:33 PM

Hi everyone,

with the latest 5.5.0-00-43 of FuseMB I get an error

similar to this MB-873

when running stability tests (constantly producing and consuming messages).

Has anyone run into this issue?

2011-08-17 20:15:14,774 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@444c742:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=105596,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2011-08-17 20:15:14,775 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@444c742:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=105596,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     ... 7 more
2011-08-17 20:15:14,775 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue | Queue:TestQueue
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     ... 6 more
Caused by: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     ... 7 more
2011-08-17 20:15:14,780 | ERROR | Looking for key 682 but not found in fileMap: {687=db-687.log number = 687 , length = 33663821, 686=db-686.log number = 686 , length = 33672993, 685=db-685.log number = 685 , length = 33788068, 684=db-684.log number = 684 , length = 33726449, 683=db-683.log number = 683 , length = 33568210, 702=db-702.log number = 702 , length = 33647665, 703=db-703.log number = 703 , length = 33808723, 700=db-700.log number = 700 , length = 33644222, 701=db-701.log number = 701 , length = 33806891, 698=db-698.log number = 698 , length = 33601212, 699=db-699.log number = 699 , length = 33703938, 696=db-696.log number = 696 , length = 33810835, 697=db-697.log number = 697 , length = 33646169, 694=db-694.log number = 694 , length = 33817575, 695=db-695.log number = 695 , length = 33555466, 692=db-692.log number = 692 , length = 33585277, 693=db-693.log number = 693 , length = 33621574, 690=db-690.log number = 690 , length = 33671671, 691=db-691.log number = 691 , length = 33790940, 688=db-688.l

When trying to restart the message bus, I get this error:

2011-08-17 20:30:01,462 | INFO  | KahaDB is version 3 | org.apache.activemq.store.kahadb.MessageDatabase | main
2011-08-17 20:30:01,534 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
2011-08-17 20:30:01,535 | INFO  | Recovery replayed 1 operations from the journal in 0.049 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
2011-08-17 20:30:01,876 | INFO  | Some journal files are missing: [682] | org.apache.activemq.store.kahadb.MessageDatabase | main
2011-08-17 20:30:01,886 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Detected missing/corrupt journal files. 1 messages affected. | org.apache.activemq.broker.BrokerService | main
java.io.IOException: Detected missing/corrupt journal files. 1 messages affected.
     at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:660)
     at org.apache.activemq.store.kahadb.MessageDatabase$6.execute(MessageDatabase.java:479)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:477)

The only way to deal with this seem to be to delete the data store (and loose all messages).

Best regards,

Frank

Edited by: fgynnild on Aug 17, 2011 6:31 PM

1. Re: Could not locate data file

fgynnild Aug 18, 2011 3:36 AM (in response to fgynnild)

Happened again today after sending approximately 9 000 000 messages:

2011-08-18 03:48:37,141 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@69bcfc26:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=1446040,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2011-08-18 03:48:37,141 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@69bcfc26:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=1446040,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     ... 7 more
2011-08-18 03:48:37,141 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue | Queue:TestQueue
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     ... 6 more

2. Re: Could not locate data file

garytully Aug 18, 2011 11:08 AM (in response to fgynnild)

Are there any other errors in the log? Can you raise a jira issue and attach your test code? This is not the same issue as MB-873.
Are you by any chance running out of disk space?
Actions
3. Re: Could not locate data file

fgynnild Aug 18, 2011 3:50 PM (in response to garytully)

When it happens, the log grows in such a crazy pace, so I haven't yet been able to see the topmost log message where it kicks off. I can create a JIRA issue for it. My code is written in C#, using the NMS library, but it's very basic. It sends the same message over and over again and another piece of code receives and ditches them.
In all cases I've seen it, there's been plenty of free disk space.
Actions
4. Re: Could not locate data file

fgynnild Aug 20, 2011 4:33 PM (in response to fgynnild)

I'm unable to create an issue it seems, but I was able to catch the log
when it happened by creating a custom script. Does anyone have a clue what can
be wrong? Is it possible to avoid it through configuration? This is a blocker issue for me, as it leaves the storage in a inconsistent state. Even restarting the message bus leaves it in this state. The only remedy I've found was to delete the entire data store which is unacceptable in a production environment since all messages are gone.

Frank

--
2011-08-19 08:43:28,459 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@5488e2dd: startup date ; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2011-08-19 08:43:33,153 | WARN | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main
2011-08-19 08:43:33,174 | INFO | PListStore:[C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\IES_CARMINE\tmp_storage ] started | org.apache.activemq.store.kahadb.plist.PListStore | main
2011-08-19 08:43:33,441 | INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb] | org.apache.activemq.broker.BrokerService | main
2011-08-19 08:43:33,756 | INFO | ActiveMQ 5.5.0-fuse-00-43 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main
2011-08-19 08:43:33,757 | INFO | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main
2011-08-19 08:43:34,183 | INFO | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main
2011-08-19 08:43:34,185 | INFO | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main
2011-08-19 08:43:34,198 | INFO | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-58246-1313736213833-0:1) started | org.apache.activemq.broker.BrokerService | main
2011-08-19 08:43:34,399 | INFO | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main
2011-08-19 08:43:35,194 | INFO | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main
2011-08-19 08:43:35,482 | INFO | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main
2011-08-19 08:43:35,929 | INFO | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main
2011-08-19 08:43:36,101 | INFO | FUSE Web Console at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main
2011-08-19 08:43:36,131 | INFO | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main
2011-08-20 20:41:43,634 | ERROR | Looking for key 2918 but not found in fileMap: {2919=db-2919.log number = 2919 , length = 1620} | org.apache.kahadb.journal.Journal | Queue:TestQueue
2011-08-20 20:41:43,647 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@634cd7b3:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=1,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2011-08-20 20:41:43,651 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@634cd7b3:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=1,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     ... 7 more
2011-08-20 20:41:43,653 | ERROR | Failed to page in more queue messages | org.apache.activemq.broker.region.Queue | Queue:TestQueue
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     ... 6 more
Caused by: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     ... 7 more
2011-08-20 20:41:43,655 | ERROR | Looking for key 2918 but not found in fileMap: {2919=db-2919.log number = 2919 , length = 19211} | org.apache.kahadb.journal.Journal | Queue:TestQueue
2011-08-20 20:41:43,656 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@634cd7b3:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=2,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log
     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
     at org.apache.kahadb.journal.Journal.read(Journal.java:597)
     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
2011-08-20 20:41:43,657 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@634cd7b3:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=2,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
--
Actions
5. Re: Could not locate data file

garytully Aug 23, 2011 1:00 PM (in response to fgynnild)

Can you post your broker xml configuration?

You can use the KahaDB options[1] ignoreMissingJournalfiles and checkForCorruptJournalFiles to allow you to recover and restart, but the missing journal file is a problem. Does the broker have exclusive access to that file system?

Can you also post your client code. I would love to be able to reproduce this to get to the bottom of it. I wonder if a Unix system behaves differently for you?

http://fusesource.com/docs/esb/4.4/amq_persistence/KahaDB-Recovery.html
Actions
6. Re: Could not locate data file

garytully Aug 23, 2011 1:08 PM (in response to garytully)

PS: If you cannot open a fusesource jira issue, please try your test case with the latest apache 5.6 (trunk) snapshot and open an issue in the apache jira. It is important that we capture as much information as possible so that we can get quickly get to the bottom of it.
Actions
7. Re: Could not locate data file

fgynnild Aug 24, 2011 5:18 AM (in response to garytully)

Thanks Gary, I'll created a AMQ issue for it as I was able to reproduce it with the stable 5.5 release. I will now re-test with latest 5.6 trunk also.
---
https://issues.apache.org/jira/browse/AMQ-3470
---
Actions

Go to original post