6 Replies Latest reply on Aug 18, 2010 12:54 AM by albert wilson

    KahaDB recovery problem after out of free disk space

    Andrey Utkin Newbie

      Hi,

       

      Activemq (5.3.1-01-00) can`t start after out of free disk space crash.

      The following exception throws:

       

      ====

      2010-07-07 18:19:49,941 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.EOFException | org.apache.activemq.broker.BrokerService | main

      java.io.EOFException

              at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)

              at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)

              at org.apache.kahadb.page.PageFile.readPage(PageFile.java:792)

              at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:440)

              at org.apache.kahadb.page.Transaction$2.(Transaction.java:431)

              at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)

              at org.apache.kahadb.page.Transaction.load(Transaction.java:404)

              at org.apache.kahadb.page.Transaction.load(Transaction.java:361)

              at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)

              at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:224)

              at org.apache.kahadb.index.BTreeNode.getLast(BTreeNode.java:553)

              at org.apache.kahadb.index.BTreeIndex.getLast(BTreeIndex.java:244)

              at org.apache.activemq.store.kahadb.MessageDatabase.loadStoredDestination(MessageDatabase.java:1224)

              at org.apache.activemq.store.kahadb.MessageDatabase.access$000(MessageDatabase.java:82)

              at org.apache.activemq.store.kahadb.MessageDatabase$2.execute(MessageDatabase.java:227)

              at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)

              at org.apache.activemq.store.kahadb.MessageDatabase.loadPageFile(MessageDatabase.java:223)

              at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:277)

              at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)

              at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)

              at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)

              at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)

              at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)

      ====

       

       

      The same results with ActiveMQ 5.4-SNAPSOT.

       

      Is there way to restore storage?

       

      Thanks.

        • 1. Re: KahaDB recovery problem after out of free disk space
          Gary Tully Novice

          Can you try the options: checkForCorruptJournalFiles and ingoreMissingJournalFiles on the KahaDBPersistenceAdapter. Enable both of them, which will sync up the index with any missing journal entries.

          • 2. Re: KahaDB recovery problem after out of free disk space
            Andrey Utkin Newbie

            It was starting about 1 minutes with lot of disk activity and then crashed with same exception:

            ======

            2010-07-13 16:45:27,516 | INFO  | Using Persistence Adapter: org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter@4d98defc | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain

            2010-07-13 16:46:23,435 | ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.EOFException | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain

            java.io.EOFException

                    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:383)

                    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:361)

                    at org.apache.kahadb.page.PageFile.readPage(PageFile.java:792)

                    at org.apache.kahadb.page.Transaction$2.readPage(Transaction.java:440)

                    at org.apache.kahadb.page.Transaction$2.(Transaction.java:431)

                    at org.apache.kahadb.page.Transaction.openInputStream(Transaction.java:428)

                    at org.apache.kahadb.page.Transaction.load(Transaction.java:404)

                    at org.apache.kahadb.page.Transaction.load(Transaction.java:361)

                    at org.apache.kahadb.index.BTreeIndex.loadNode(BTreeIndex.java:262)

                    at org.apache.kahadb.index.BTreeNode.getChild(BTreeNode.java:224)

                    at org.apache.kahadb.index.BTreeNode.getLast(BTreeNode.java:553)

                    at org.apache.kahadb.index.BTreeIndex.getLast(BTreeIndex.java:244)

                    at org.apache.activemq.store.kahadb.MessageDatabase.loadStoredDestination(MessageDatabase.java:1224)

                    at org.apache.activemq.store.kahadb.MessageDatabase.access$000(MessageDatabase.java:82)

                    at org.apache.activemq.store.kahadb.MessageDatabase$2.execute(MessageDatabase.java:227)

                    at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)

                    at org.apache.activemq.store.kahadb.MessageDatabase.loadPageFile(MessageDatabase.java:223)

                    at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:277)

                    at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)

                    at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)

                    at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)

                    at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)

                    at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)

            ======

            • 4. Re: KahaDB recovery problem after out of free disk space
              Andrey Utkin Newbie

              No, it does not.

              Broker crashed on startup with exception java.io.EOFException. Suggested parameters checkForCorruptJournalFiles and ingoreMissingJournalFiles has no effect.

              • 5. Re: KahaDB recovery problem after out of free disk space
                Gary Tully Novice

                in this case, it looks like it is necessary to rebuild the index by replaying the journal. Take a copy of the data directory as a backup, then remove the index file, db.data and restart the broker.

                The missing index will cause the journal to be replayed but any missing journal files will be ignored.

                • 6. Term Papers
                  albert wilson Newbie

                  I really like this blog, It's always nice when you can not only be informed, but also get entertain, from these type of blog, that is such a nice award show,  really enjoying to watch this, nice entry. Thanks

                   

                  Term Papers