recovery after file system full on KahaDBMessageStore with ActiveMQ 5.3.1

[dgdavis00]

We ran a performance test and filled the file system which has our KahaDBMessageStore.

I ran into an issue when trying to restart the broker after I recovered some space by moving and softlinking a couple of the log files to a tmp directory.

What I found is that the last log, #55, was lost when the disc became full, and there for the broker could not find it and could not start.

I am running 5.3.1 Fuse activeMq version? would a later release handle this better?

Please let me know if you need additional information.

THANKS for your help!

Delails below?.

stuxsh01.st9769:/home/mqmadm$ df -k |grep activemq

                        507748     52225    429309  11% /opt/isv/activemq/fmb-5.3.0.5

/dev/mapper/vg01-lv_activemq_var

                       1032088    979756         0 100% /run/ca/activemq

/dev/mapper/vg01-lv_activemq_mars

                        507748     13659    467875   3% /opt/hd/ca/mars/activemq

/dev/mapper/vg01-lv_activemq_tmp

                        507748     11317    470217   3% /opt/tmp/ca/activemq

cd  /run/ca/activemq/data/BR0/KahaDBMessageStore

ls ?l

-rw-rr  1 mqmadm mqm 34234561 Nov  3 06:12 db-27.log

-rw-rr  1 mqmadm mqm 34497490 Nov  3 06:12 db-28.log

-rw-rr  1 mqmadm mqm 34698497 Nov  3 06:13 db-29.log

-rw-rr  1 mqmadm mqm 34062008 Nov  3 06:13 db-30.log

-rw-rr  1 mqmadm mqm 34163081 Nov  3 06:13 db-31.log

-rw-rr  1 mqmadm mqm 34367441 Nov  3 06:13 db-32.log

-rw-rr  1 mqmadm mqm 34106196 Nov  3 06:14 db-33.log

-rw-rr  1 mqmadm mqm 33906318 Nov  3 06:14 db-34.log

-rw-rr  1 mqmadm mqm 34110446 Nov  3 06:14 db-35.log

-rw-rr  1 mqmadm mqm 34073214 Nov  3 06:15 db-36.log

-rw-rr  1 mqmadm mqm 34897536 Nov  3 06:15 db-37.log

-rw-rr  1 mqmadm mqm 33664863 Nov  3 06:15 db-38.log

-rw-rr  1 mqmadm mqm 33580054 Nov  3 06:15 db-39.log

-rw-rr  1 mqmadm mqm 33902398 Nov  3 06:16 db-40.log

-rw-rr  1 mqmadm mqm 34517946 Nov  3 06:16 db-41.log

-rw-rr  1 mqmadm mqm 34367661 Nov  3 06:16 db-42.log

-rw-rr  1 mqmadm mqm 34465374 Nov  3 06:17 db-43.log

-rw-rr  1 mqmadm mqm 34597832 Nov  3 06:17 db-44.log

-rw-rr  1 mqmadm mqm 33978223 Nov  3 06:17 db-45.log

-rw-rr  1 mqmadm mqm 34099281 Nov  3 06:17 db-46.log

-rw-rr  1 mqmadm mqm 33892697 Nov  3 06:18 db-47.log

-rw-rr  1 mqmadm mqm 34259593 Nov  3 06:18 db-48.log

-rw-rr  1 mqmadm mqm 34066799 Nov  3 06:18 db-49.log

-rw-rr  1 mqmadm mqm 33968924 Nov  3 06:19 db-50.log

-rw-rr  1 mqmadm mqm 34169738 Nov  3 06:19 db-51.log

-rw-rr  1 mqmadm mqm 33955516 Nov  3 06:23 db-52.log

-rw-rr  1 mqmadm mqm 33559631 Nov  3 06:26 db-53.log

-rw-rr  1 mqmadm mqm 33288192 Nov  3 06:33 db-54.log

-rw-rr  1 mqmadm mqm 10985472 Nov  3 06:33 db.data

-rw-rr  1 mqmadm mqm   689464 Nov  3 06:33 db.redo

-rw-rr  1 mqmadm mqm        0 Oct 20 15:30 lock

$ mv db-27.log /tmp/db-28.log

$ mv db-28.log /tmp/db-28.log

ln -s /tmp/db-27.log /run/ca/activemq/data/BR0/KahaDBMessageStore/db-27.log

ln -s /tmp/db-28.log /run/ca/activemq/data/BR0/KahaDBMessageStore/db-28.log

restarted broker

Loading message broker from: xbean:file:/opt/hd/ca/mars/activemq/BR0/conf/activemq0.xml

INFO | Using Persistence Adapter: org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter@956254

INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:61660/jmxrmi

ERROR | Looking for key 55 but not found in fileMap: {27=db-27.log number = 27 , length = 34234561, 29=db-29.log number = 29 , length = 34698497, 28=db-28.log number = 28 , length = 34497490, 31=db-31.log number = 31 , length = 34163081, 30=db-30.log number = 30 , length = 34062008, 34=db-34.log number = 34 , length = 33906318, 35=db-35.log number = 35 , length = 34110446, 32=db-32.log number = 32 , length = 34367441, 33=db-33.log number = 33 , length = 34106196, 38=db-38.log number = 38 , length = 33664863, 39=db-39.log number = 39 , length = 33580054, 36=db-36.log number = 36 , length = 34073214, 37=db-37.log number = 37 , length = 34897536, 42=db-42.log number = 42 , length = 34367661, 43=db-43.log number = 43 , length = 34465374, 40=db-40.log number = 40 , length = 33902398, 41=db-41.log number = 41 , length = 34517946, 46=db-46.log number = 46 , length = 34099281, 47=db-47.log number = 47 , length = 33892697, 44=db-44.log number = 44 , length = 34597832, 45=db-45.log number = 45 , length = 33978223, 51=db-51.log number = 51 , length = 34169738, 50=db-50.log number = 50 , length = 33968924, 49=db-49.log number = 49 , length = 34066799, 48=db-48.log number = 48 , length = 34259593, 54=db-54.log number = 54 , length = 34113708, 53=db-53.log number = 53 , length = 33559631, 52=db-52.log number = 52 , length = 33955516}

ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Could not locate data file /run/ca/activemq/data/BR0/KahaDBMessageStore/db-55.log

java.io.IOException: Could not locate data file /run/ca/activemq/data/BR0/KahaDBMessageStore/db-55.log

        at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:325)

        at org.apache.kahadb.journal.Journal.getNextLocation(Journal.java:482)

        at org.apache.activemq.store.kahadb.MessageDatabase.getRecoveryPosition(MessageDatabase.java:584)

        at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:387)

        at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:280)

        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)

        at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)

        at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)

        at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)

        at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1414)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1375)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)

        at java.security.AccessController.doPrivileged(Native Method)

        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)

        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)

        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)

        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)

        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)

        at org.apache.xbean.spring.context.ResourceXmlApplicationContext.(ResourceXmlApplicationContext.java:52)

        at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:96)

        at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:52)

        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)

        at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)

        at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)

        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:91)

        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)

        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:136)

        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)

        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:82)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.apache.activemq.console.Main.runTaskClass(Main.java:251)

        at org.apache.activemq.console.Main.main(Main.java:107)

INFO | ActiveMQ Message Broker (BR0.QA.US.STUXSH01.ST9769, null) is shutting down

INFO | Network Connector NC00.QA.9 Stopped

INFO | Connector TC00.QA.ISP.ST9769 Stopped

INFO | Connector TC01.QA.ISP.ST9769 Stopped

INFO | ActiveMQ JMS Message Broker (BR0.QA.US.STUXSH01.ST9769, null) stopped

[stlewis]

Delete the db.data which should let the broker recover all of the other messages contained in the other files.  This is mentioned in the docs here under "Forcing recovery by deleting the metadata store"

[dgdavis00]

I had deleted the db.data   and index .

I later found these and in some cases,

If added these 2 paramaters checkForCorruptJournalFiles and ingoreMissingJournal to the activemq.xml it was able to recover .

In one case, someone had deleted one of the log files, and then the parameters above 2 parameters did not help, I still had to delete all the logs, data & index and lt it re-create them on restart.