6 Replies Latest reply on Aug 18, 2010 12:54 AM by albertwilson

    KahaDB recovery problem after out of free disk space

    ciand7

      Hi,

       

      Activemq (5.3.1-01-00) can`t start after out of free disk space crash.

      The following exception throws:

       

      ====

      2010-07-07 18:19:49,941 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.EOFException | org.apache.activemq.broker.BrokerService | main

      java.io.EOFException

              at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)

              at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)

              at org.apache.kahadb.page.PageFile.readPage(PageFile.java:792)

              at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:440)

              at org.apache.kahadb.page.Transaction$2.(Transaction.java:431)

              at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)

              at org.apache.kahadb.page.Transaction.load(Transaction.java:404)

              at org.apache.kahadb.page.Transaction.load(Transaction.java:361)

              at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)

              at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:224)

              at org.apache.kahadb.index.BTreeNode.getLast(BTreeNode.java:553)

              at org.apache.kahadb.index.BTreeIndex.getLast(BTreeIndex.java:244)

              at org.apache.activemq.store.kahadb.MessageDatabase.loadStoredDestination(MessageDatabase.java:1224)

              at org.apache.activemq.store.kahadb.MessageDatabase.access$000(MessageDatabase.java:82)

              at org.apache.activemq.store.kahadb.MessageDatabase$2.execute(MessageDatabase.java:227)

              at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)

              at org.apache.activemq.store.kahadb.MessageDatabase.loadPageFile(MessageDatabase.java:223)

              at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:277)

              at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)

              at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)

              at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)

              at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)

              at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)

      ====

       

       

      The same results with ActiveMQ 5.4-SNAPSOT.

       

      Is there way to restore storage?

       

      Thanks.

        • 1. Re: KahaDB recovery problem after out of free disk space
          garytully

          Can you try the options: checkForCorruptJournalFiles and ingoreMissingJournalFiles on the KahaDBPersistenceAdapter. Enable both of them, which will sync up the index with any missing journal entries.

          • 2. Re: KahaDB recovery problem after out of free disk space
            ciand7

            It was starting about 1 minutes with lot of disk activity and then crashed with same exception:

            ======

            2010-07-13 16:45:27,516 | INFO  | Using Persistence Adapter: org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter@4d98defc | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain

            2010-07-13 16:46:23,435 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.EOFException | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain

            java.io.EOFException

                    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)

                    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)

                    at org.apache.kahadb.page.PageFile.readPage(PageFile.java:792)

                    at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:440)

                    at org.apache.kahadb.page.Transaction$2.(Transaction.java:431)

                    at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)

                    at org.apache.kahadb.page.Transaction.load(Transaction.java:404)

                    at org.apache.kahadb.page.Transaction.load(Transaction.java:361)

                    at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)

                    at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:224)

                    at org.apache.kahadb.index.BTreeNode.getLast(BTreeNode.java:553)

                    at org.apache.kahadb.index.BTreeIndex.getLast(BTreeIndex.java:244)

                    at org.apache.activemq.store.kahadb.MessageDatabase.loadStoredDestination(MessageDatabase.java:1224)

                    at org.apache.activemq.store.kahadb.MessageDatabase.access$000(MessageDatabase.java:82)

                    at org.apache.activemq.store.kahadb.MessageDatabase$2.execute(MessageDatabase.java:227)

                    at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)

                    at org.apache.activemq.store.kahadb.MessageDatabase.loadPageFile(MessageDatabase.java:223)

                    at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:277)

                    at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)

                    at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)

                    at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)

                    at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)

                    at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)

            ======

            • 3. Re: KahaDB recovery problem after out of free disk space
              houndz17

              Does it work?

              • 4. Re: KahaDB recovery problem after out of free disk space
                ciand7

                No, it does not.

                Broker crashed on startup with exception java.io.EOFException. Suggested parameters checkForCorruptJournalFiles and ingoreMissingJournalFiles has no effect.

                • 5. Re: KahaDB recovery problem after out of free disk space
                  garytully

                  in this case, it looks like it is necessary to rebuild the index by replaying the journal. Take a copy of the data directory as a backup, then remove the index file, db.data and restart the broker.

                  The missing index will cause the journal to be replayed but any missing journal files will be ignored.

                  • 6. Term Papers
                    albertwilson

                    I really like this blog, It's always nice when you can not only be informed, but also get entertain, from these type of blog, that is such a nice award show,  really enjoying to watch this, nice entry. Thanks

                     

                    Term Papers