recovery after file system full on KahaDBMessageStore with ActiveMQ 5.3.1
dgdavis00 Nov 3, 2010 1:56 PMWe ran a performance test and filled the file system which has our KahaDBMessageStore.
I ran into an issue when trying to restart the broker after I recovered some space by moving and softlinking a couple of the log files to a tmp directory.
What I found is that the last log, #55, was lost when the disc became full, and there for the broker could not find it and could not start.
I am running 5.3.1 Fuse activeMq version? would a later release handle this better?
Please let me know if you need additional information.
THANKS for your help!
Delails below?.
stuxsh01.st9769:/home/mqmadm$ df -k |grep activemq
507748 52225 429309 11% /opt/isv/activemq/fmb-5.3.0.5
/dev/mapper/vg01-lv_activemq_var
1032088 979756 0 100% /run/ca/activemq
/dev/mapper/vg01-lv_activemq_mars
507748 13659 467875 3% /opt/hd/ca/mars/activemq
/dev/mapper/vg01-lv_activemq_tmp
507748 11317 470217 3% /opt/tmp/ca/activemq
cd /run/ca/activemq/data/BR0/KahaDBMessageStore
ls ?l
-rw-rr 1 mqmadm mqm 34234561 Nov 3 06:12 db-27.log
-rw-rr 1 mqmadm mqm 34497490 Nov 3 06:12 db-28.log
-rw-rr 1 mqmadm mqm 34698497 Nov 3 06:13 db-29.log
-rw-rr 1 mqmadm mqm 34062008 Nov 3 06:13 db-30.log
-rw-rr 1 mqmadm mqm 34163081 Nov 3 06:13 db-31.log
-rw-rr 1 mqmadm mqm 34367441 Nov 3 06:13 db-32.log
-rw-rr 1 mqmadm mqm 34106196 Nov 3 06:14 db-33.log
-rw-rr 1 mqmadm mqm 33906318 Nov 3 06:14 db-34.log
-rw-rr 1 mqmadm mqm 34110446 Nov 3 06:14 db-35.log
-rw-rr 1 mqmadm mqm 34073214 Nov 3 06:15 db-36.log
-rw-rr 1 mqmadm mqm 34897536 Nov 3 06:15 db-37.log
-rw-rr 1 mqmadm mqm 33664863 Nov 3 06:15 db-38.log
-rw-rr 1 mqmadm mqm 33580054 Nov 3 06:15 db-39.log
-rw-rr 1 mqmadm mqm 33902398 Nov 3 06:16 db-40.log
-rw-rr 1 mqmadm mqm 34517946 Nov 3 06:16 db-41.log
-rw-rr 1 mqmadm mqm 34367661 Nov 3 06:16 db-42.log
-rw-rr 1 mqmadm mqm 34465374 Nov 3 06:17 db-43.log
-rw-rr 1 mqmadm mqm 34597832 Nov 3 06:17 db-44.log
-rw-rr 1 mqmadm mqm 33978223 Nov 3 06:17 db-45.log
-rw-rr 1 mqmadm mqm 34099281 Nov 3 06:17 db-46.log
-rw-rr 1 mqmadm mqm 33892697 Nov 3 06:18 db-47.log
-rw-rr 1 mqmadm mqm 34259593 Nov 3 06:18 db-48.log
-rw-rr 1 mqmadm mqm 34066799 Nov 3 06:18 db-49.log
-rw-rr 1 mqmadm mqm 33968924 Nov 3 06:19 db-50.log
-rw-rr 1 mqmadm mqm 34169738 Nov 3 06:19 db-51.log
-rw-rr 1 mqmadm mqm 33955516 Nov 3 06:23 db-52.log
-rw-rr 1 mqmadm mqm 33559631 Nov 3 06:26 db-53.log
-rw-rr 1 mqmadm mqm 33288192 Nov 3 06:33 db-54.log
-rw-rr 1 mqmadm mqm 10985472 Nov 3 06:33 db.data
-rw-rr 1 mqmadm mqm 689464 Nov 3 06:33 db.redo
-rw-rr 1 mqmadm mqm 0 Oct 20 15:30 lock
$ mv db-27.log /tmp/db-28.log
$ mv db-28.log /tmp/db-28.log
ln -s /tmp/db-27.log /run/ca/activemq/data/BR0/KahaDBMessageStore/db-27.log
ln -s /tmp/db-28.log /run/ca/activemq/data/BR0/KahaDBMessageStore/db-28.log
restarted broker
Loading message broker from: xbean:file:/opt/hd/ca/mars/activemq/BR0/conf/activemq0.xml
INFO | Using Persistence Adapter: org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter@956254
INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:61660/jmxrmi
ERROR | Looking for key 55 but not found in fileMap: {27=db-27.log number = 27 , length = 34234561, 29=db-29.log number = 29 , length = 34698497, 28=db-28.log number = 28 , length = 34497490, 31=db-31.log number = 31 , length = 34163081, 30=db-30.log number = 30 , length = 34062008, 34=db-34.log number = 34 , length = 33906318, 35=db-35.log number = 35 , length = 34110446, 32=db-32.log number = 32 , length = 34367441, 33=db-33.log number = 33 , length = 34106196, 38=db-38.log number = 38 , length = 33664863, 39=db-39.log number = 39 , length = 33580054, 36=db-36.log number = 36 , length = 34073214, 37=db-37.log number = 37 , length = 34897536, 42=db-42.log number = 42 , length = 34367661, 43=db-43.log number = 43 , length = 34465374, 40=db-40.log number = 40 , length = 33902398, 41=db-41.log number = 41 , length = 34517946, 46=db-46.log number = 46 , length = 34099281, 47=db-47.log number = 47 , length = 33892697, 44=db-44.log number = 44 , length = 34597832, 45=db-45.log number = 45 , length = 33978223, 51=db-51.log number = 51 , length = 34169738, 50=db-50.log number = 50 , length = 33968924, 49=db-49.log number = 49 , length = 34066799, 48=db-48.log number = 48 , length = 34259593, 54=db-54.log number = 54 , length = 34113708, 53=db-53.log number = 53 , length = 33559631, 52=db-52.log number = 52 , length = 33955516}
ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Could not locate data file /run/ca/activemq/data/BR0/KahaDBMessageStore/db-55.log
java.io.IOException: Could not locate data file /run/ca/activemq/data/BR0/KahaDBMessageStore/db-55.log
at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:325)
at org.apache.kahadb.journal.Journal.getNextLocation(Journal.java:482)
at org.apache.activemq.store.kahadb.MessageDatabase.getRecoveryPosition(MessageDatabase.java:584)
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:387)
at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:280)
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)
at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1375)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)
at java.security.AccessController.doPrivileged(Native Method)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)
at org.apache.xbean.spring.context.ResourceXmlApplicationContext.(ResourceXmlApplicationContext.java:52)
at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:96)
at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:52)
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:91)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:136)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
at org.apache.activemq.console.Main.main(Main.java:107)
INFO | ActiveMQ Message Broker (BR0.QA.US.STUXSH01.ST9769, null) is shutting down
INFO | Network Connector NC00.QA.9 Stopped
INFO | Connector TC00.QA.ISP.ST9769 Stopped
INFO | Connector TC01.QA.ISP.ST9769 Stopped
INFO | ActiveMQ JMS Message Broker (BR0.QA.US.STUXSH01.ST9769, null) stopped