7 Replies Latest reply on Aug 24, 2011 5:18 AM by Frank Gynnild

    Could not locate data file

    Frank Gynnild Newbie

      Hi everyone,

      with the latest 5.5.0-00-43 of FuseMB I get an error

      similar to this MB-873

      when running stability tests (constantly producing and consuming messages).

      Has anyone run into this issue?

       

      2011-08-17 20:15:14,774 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@444c742:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=105596,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
      java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
           at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
           at org.apache.kahadb.journal.Journal.read(Journal.java:597)
           at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
           at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
           at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
           at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
           at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
           at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
           at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
           at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
           at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
           at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
           at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
           at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
           at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
      2011-08-17 20:15:14,775 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@444c742:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=105596,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
      java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
           at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
           at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
           at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
           at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
           at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
           at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
      Caused by: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
           at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
           at org.apache.kahadb.journal.Journal.read(Journal.java:597)
           at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
           at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
           at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
           at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
           at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
           at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
           at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
           ... 7 more
      2011-08-17 20:15:14,775 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue | Queue:TestQueue
      java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116)
           at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
           at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
           at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
           at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
           at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
           at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
      Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
           ... 6 more
      Caused by: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-682.log
           at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
           at org.apache.kahadb.journal.Journal.read(Journal.java:597)
           at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
           at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
           at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
           at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
           at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
           at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
           at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
           at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
           ... 7 more
      2011-08-17 20:15:14,780 | ERROR | Looking for key 682 but not found in fileMap: {687=db-687.log number = 687 , length = 33663821, 686=db-686.log number = 686 , length = 33672993, 685=db-685.log number = 685 , length = 33788068, 684=db-684.log number = 684 , length = 33726449, 683=db-683.log number = 683 , length = 33568210, 702=db-702.log number = 702 , length = 33647665, 703=db-703.log number = 703 , length = 33808723, 700=db-700.log number = 700 , length = 33644222, 701=db-701.log number = 701 , length = 33806891, 698=db-698.log number = 698 , length = 33601212, 699=db-699.log number = 699 , length = 33703938, 696=db-696.log number = 696 , length = 33810835, 697=db-697.log number = 697 , length = 33646169, 694=db-694.log number = 694 , length = 33817575, 695=db-695.log number = 695 , length = 33555466, 692=db-692.log number = 692 , length = 33585277, 693=db-693.log number = 693 , length = 33621574, 690=db-690.log number = 690 , length = 33671671, 691=db-691.log number = 691 , length = 33790940, 688=db-688.l
      

       

      When trying to restart the message bus, I get this error:

      2011-08-17 20:30:01,462 | INFO  | KahaDB is version 3 | org.apache.activemq.store.kahadb.MessageDatabase | main
      2011-08-17 20:30:01,534 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
      2011-08-17 20:30:01,535 | INFO  | Recovery replayed 1 operations from the journal in 0.049 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
      2011-08-17 20:30:01,876 | INFO  | Some journal files are missing: [682] | org.apache.activemq.store.kahadb.MessageDatabase | main
      2011-08-17 20:30:01,886 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Detected missing/corrupt journal files. 1 messages affected. | org.apache.activemq.broker.BrokerService | main
      java.io.IOException: Detected missing/corrupt journal files. 1 messages affected.
           at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:660)
           at org.apache.activemq.store.kahadb.MessageDatabase$6.execute(MessageDatabase.java:479)
           at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
           at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:477)
      

       

      The only way to deal with this seem to be to delete the data store (and loose all messages).

       

      Best regards,

      Frank

       

      Edited by: fgynnild on Aug 17, 2011 6:31 PM

        • 1. Re: Could not locate data file
          Frank Gynnild Newbie

          Happened again today after sending approximately 9 000 000 messages:

          2011-08-18 03:48:37,141 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@69bcfc26:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=1446040,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
          java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
               at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
               at org.apache.kahadb.journal.Journal.read(Journal.java:597)
               at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
               at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
               at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
               at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
               at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
               at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
               at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
               at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
               at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
               at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
               at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
               at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
               at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
               at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
               at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
          2011-08-18 03:48:37,141 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@69bcfc26:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=1446040,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue
          java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
               at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
               at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
               at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
               at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
               at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
               at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
               at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
               at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
          Caused by: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
               at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)
               at org.apache.kahadb.journal.Journal.read(Journal.java:597)
               at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)
               at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
               at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
               at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)
               at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)
               at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
               at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
               at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)
               ... 7 more
          2011-08-18 03:48:37,141 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue | Queue:TestQueue
          java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
               at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116)
               at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
               at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)
               at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)
               at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)
               at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
               at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
          Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\MessagePumpTest\FuseMB\data\kahadb\db-666.log
               at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)
               at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
               ... 6 more
          

           

          • 2. Re: Could not locate data file
            Gary Tully Novice

            Are there any other errors in the log? Can you raise a jira issue and attach your test code? This is not the same issue as MB-873.

            Are you by any chance running out of disk space?

            • 3. Re: Could not locate data file
              Frank Gynnild Newbie

              When it happens, the log grows in such a crazy pace, so I haven't yet been able to see the topmost log message where it kicks off. I can create a JIRA issue for it. My code is written in C#, using the NMS library, but it's very basic. It sends the same message over and over again and another piece of code receives and ditches them.

              In all cases I've seen it, there's been plenty of free disk space.

              • 4. Re: Could not locate data file
                Frank Gynnild Newbie

                I'm unable to create an issue it seems, but I was able to catch the log

                when it happened by creating a custom script. Does anyone have a clue what can

                be wrong? Is it possible to avoid it through configuration? This is a blocker issue for me, as it leaves the storage in a inconsistent state. Even restarting the message bus leaves it in this state. The only remedy I've found was to delete the entire data store which is unacceptable in a production environment since all messages are gone.

                 

                Frank

                 

                --

                2011-08-19 08:43:28,459 | INFO  | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@5488e2dd: startup date ; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main

                2011-08-19 08:43:33,153 | WARN  | destroyApplicationContextOnStop parameter is deprecated, please use shutdown hooks instead | org.apache.activemq.xbean.XBeanBrokerService | main

                2011-08-19 08:43:33,174 | INFO  | PListStore:[C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\IES_CARMINE\tmp_storage ] started | org.apache.activemq.store.kahadb.plist.PListStore | main

                2011-08-19 08:43:33,441 | INFO  | Using Persistence Adapter: KahaDBPersistenceAdapter[C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb] | org.apache.activemq.broker.BrokerService | main

                2011-08-19 08:43:33,756 | INFO  | ActiveMQ 5.5.0-fuse-00-43 JMS Message Broker (IES_CARMINE) is starting | org.apache.activemq.broker.BrokerService | main

                2011-08-19 08:43:33,757 | INFO  | For help or more information please see: http://activemq.apache.org/ | org.apache.activemq.broker.BrokerService | main

                2011-08-19 08:43:34,183 | INFO  | Listening for connections at: tcp://Carmine:61616?keepAlive=true&wireFormat.tightEncodingEnabled=false&wireFormat.cacheEnabled=false | org.apache.activemq.transport.TransportServerThreadSupport | main

                2011-08-19 08:43:34,185 | INFO  | Connector openwire Started | org.apache.activemq.broker.TransportConnector | main

                2011-08-19 08:43:34,198 | INFO  | ActiveMQ JMS Message Broker (IES_CARMINE, ID:Carmine-58246-1313736213833-0:1) started | org.apache.activemq.broker.BrokerService | main

                2011-08-19 08:43:34,399 | INFO  | jetty-7.1.6.v20100715 | org.eclipse.jetty.util.log | main

                2011-08-19 08:43:35,194 | INFO  | ActiveMQ WebConsole initialized. | org.apache.activemq.web.WebConsoleStarter | main

                2011-08-19 08:43:35,482 | INFO  | Initializing Spring FrameworkServlet 'dispatcher' | /admin | main

                2011-08-19 08:43:35,929 | INFO  | ActiveMQ Console at http://0.0.0.0:8161/admin | org.eclipse.jetty.util.log | main

                2011-08-19 08:43:36,101 | INFO  | FUSE Web Console at http://0.0.0.0:8161/console | org.eclipse.jetty.util.log | main

                2011-08-19 08:43:36,131 | INFO  | Started SelectChannelConnector@0.0.0.0:8161 | org.eclipse.jetty.util.log | main

                2011-08-20 20:41:43,634 | ERROR | Looking for key 2918 but not found in fileMap: {2919=db-2919.log number = 2919 , length = 1620} | org.apache.kahadb.journal.Journal | Queue:TestQueue

                2011-08-20 20:41:43,647 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@634cd7b3:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=1,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue

                java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log

                     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)

                     at org.apache.kahadb.journal.Journal.read(Journal.java:597)

                     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)

                     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)

                     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)

                     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)

                     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)

                     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)

                     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)

                     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)

                     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)

                     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)

                     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)

                     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)

                     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

                2011-08-20 20:41:43,651 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@634cd7b3:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=1,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue

                java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)

                     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)

                     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)

                     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)

                     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)

                     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)

                     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

                Caused by: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log

                     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)

                     at org.apache.kahadb.journal.Journal.read(Journal.java:597)

                     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)

                     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)

                     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)

                     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)

                     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)

                     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)

                     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)

                     ... 7 more

                2011-08-20 20:41:43,653 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue | Queue:TestQueue

                java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:116)

                     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)

                     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)

                     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)

                     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)

                     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)

                     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

                Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:269)

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)

                     ... 6 more

                Caused by: java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log

                     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)

                     at org.apache.kahadb.journal.Journal.read(Journal.java:597)

                     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)

                     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)

                     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)

                     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)

                     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)

                     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)

                     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)

                     ... 7 more

                2011-08-20 20:41:43,655 | ERROR | Looking for key 2918 but not found in fileMap: {2919=db-2919.log number = 2919 , length = 19211} | org.apache.kahadb.journal.Journal | Queue:TestQueue

                2011-08-20 20:41:43,656 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@634cd7b3:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=2,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue

                java.io.IOException: Could not locate data file C:\Program Files\VirtualWorks\Cirrus Search\FuseMB\data\kahadb\db-2918.log

                     at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:350)

                     at org.apache.kahadb.journal.Journal.read(Journal.java:597)

                     at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:820)

                     at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)

                     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)

                     at org.apache.kahadb.page.Transaction.execute(Transaction.java:765)

                     at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:482)

                     at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)

                     at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:266)

                     at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)

                     at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)

                     at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1695)

                     at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1912)

                     at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1431)

                     at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)

                     at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

                2011-08-20 20:41:43,657 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@634cd7b3:TestQueue,batchResetNeeded=false,storeHasMessages=false,size=2,cacheEnabled=false - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | Queue:TestQueue

                --

                • 5. Re: Could not locate data file
                  Gary Tully Novice

                  Can you post your broker xml configuration?

                   

                  You can use the KahaDB options[1] ignoreMissingJournalfiles and checkForCorruptJournalFiles to allow you to recover and restart, but the missing journal file is a problem. Does the broker have exclusive access to that file system?

                   

                  Can you also post your client code. I would love to be able to reproduce this to get to the bottom of it. I wonder if a Unix system behaves differently for you?

                   

                  http://fusesource.com/docs/esb/4.4/amq_persistence/KahaDB-Recovery.html

                  • 6. Re: Could not locate data file
                    Gary Tully Novice

                    PS: If you cannot open a fusesource jira issue, please try your test case with the latest apache 5.6 (trunk) snapshot and open an issue in the apache jira. It is important that we capture as much information as possible so that we can get quickly get to the bottom of it.

                    • 7. Re: Could not locate data file
                      Frank Gynnild Newbie

                      Thanks Gary, I'll created a AMQ issue for it as I was able to reproduce it with the stable 5.5 release. I will now re-test with latest 5.6 trunk also.

                      ---

                      https://issues.apache.org/jira/browse/AMQ-3470

                      ---