2 Replies Latest reply on Dec 2, 2010 6:04 PM by dorothea davis

    recovery after file system full on KahaDBMessageStore with ActiveMQ 5.3.1

    dorothea davis Newbie

      We ran a performance test and filled the file system which has our KahaDBMessageStore.

       

      I ran into an issue when trying to restart the broker after I recovered some space by moving and softlinking a couple of the log files to a tmp directory.

       

      What I found is that the last log, #55, was lost when the disc became full, and there for the broker could not find it and could not start.

       

      I am running 5.3.1 Fuse activeMq version? would a later release handle this better?

      Please let me know if you need additional information.

       

      THANKS for your help!

       

       

      Delails below?.

       

       

      stuxsh01.st9769:/home/mqmadm$ df -k |grep activemq

                              507748     52225    429309  11% /opt/isv/activemq/fmb-5.3.0.5

      /dev/mapper/vg01-lv_activemq_var

                             1032088    979756         0 100% /run/ca/activemq

      /dev/mapper/vg01-lv_activemq_mars

                              507748     13659    467875   3% /opt/hd/ca/mars/activemq

      /dev/mapper/vg01-lv_activemq_tmp

                              507748     11317    470217   3% /opt/tmp/ca/activemq

       

       

       

      cd  /run/ca/activemq/data/BR0/KahaDBMessageStore

       

      ls ?l

       

      -rw-rr  1 mqmadm mqm 34234561 Nov  3 06:12 db-27.log

      -rw-rr  1 mqmadm mqm 34497490 Nov  3 06:12 db-28.log

      -rw-rr  1 mqmadm mqm 34698497 Nov  3 06:13 db-29.log

      -rw-rr  1 mqmadm mqm 34062008 Nov  3 06:13 db-30.log

      -rw-rr  1 mqmadm mqm 34163081 Nov  3 06:13 db-31.log

      -rw-rr  1 mqmadm mqm 34367441 Nov  3 06:13 db-32.log

      -rw-rr  1 mqmadm mqm 34106196 Nov  3 06:14 db-33.log

      -rw-rr  1 mqmadm mqm 33906318 Nov  3 06:14 db-34.log

      -rw-rr  1 mqmadm mqm 34110446 Nov  3 06:14 db-35.log

      -rw-rr  1 mqmadm mqm 34073214 Nov  3 06:15 db-36.log

      -rw-rr  1 mqmadm mqm 34897536 Nov  3 06:15 db-37.log

      -rw-rr  1 mqmadm mqm 33664863 Nov  3 06:15 db-38.log

      -rw-rr  1 mqmadm mqm 33580054 Nov  3 06:15 db-39.log

      -rw-rr  1 mqmadm mqm 33902398 Nov  3 06:16 db-40.log

      -rw-rr  1 mqmadm mqm 34517946 Nov  3 06:16 db-41.log

      -rw-rr  1 mqmadm mqm 34367661 Nov  3 06:16 db-42.log

      -rw-rr  1 mqmadm mqm 34465374 Nov  3 06:17 db-43.log

      -rw-rr  1 mqmadm mqm 34597832 Nov  3 06:17 db-44.log

      -rw-rr  1 mqmadm mqm 33978223 Nov  3 06:17 db-45.log

      -rw-rr  1 mqmadm mqm 34099281 Nov  3 06:17 db-46.log

      -rw-rr  1 mqmadm mqm 33892697 Nov  3 06:18 db-47.log

      -rw-rr  1 mqmadm mqm 34259593 Nov  3 06:18 db-48.log

      -rw-rr  1 mqmadm mqm 34066799 Nov  3 06:18 db-49.log

      -rw-rr  1 mqmadm mqm 33968924 Nov  3 06:19 db-50.log

      -rw-rr  1 mqmadm mqm 34169738 Nov  3 06:19 db-51.log

      -rw-rr  1 mqmadm mqm 33955516 Nov  3 06:23 db-52.log

      -rw-rr  1 mqmadm mqm 33559631 Nov  3 06:26 db-53.log

      -rw-rr  1 mqmadm mqm 33288192 Nov  3 06:33 db-54.log

      -rw-rr  1 mqmadm mqm 10985472 Nov  3 06:33 db.data

      -rw-rr  1 mqmadm mqm   689464 Nov  3 06:33 db.redo

      -rw-rr  1 mqmadm mqm        0 Oct 20 15:30 lock

       

      $ mv db-27.log /tmp/db-28.log

       

      $ mv db-28.log /tmp/db-28.log

       

       

      ln -s /tmp/db-27.log /run/ca/activemq/data/BR0/KahaDBMessageStore/db-27.log

       

      ln -s /tmp/db-28.log /run/ca/activemq/data/BR0/KahaDBMessageStore/db-28.log

       

       

       

      restarted broker

       

      Loading message broker from: xbean:file:/opt/hd/ca/mars/activemq/BR0/conf/activemq0.xml

      INFO | Using Persistence Adapter: org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter@956254

      INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:61660/jmxrmi

      ERROR | Looking for key 55 but not found in fileMap: {27=db-27.log number = 27 , length = 34234561, 29=db-29.log number = 29 , length = 34698497, 28=db-28.log number = 28 , length = 34497490, 31=db-31.log number = 31 , length = 34163081, 30=db-30.log number = 30 , length = 34062008, 34=db-34.log number = 34 , length = 33906318, 35=db-35.log number = 35 , length = 34110446, 32=db-32.log number = 32 , length = 34367441, 33=db-33.log number = 33 , length = 34106196, 38=db-38.log number = 38 , length = 33664863, 39=db-39.log number = 39 , length = 33580054, 36=db-36.log number = 36 , length = 34073214, 37=db-37.log number = 37 , length = 34897536, 42=db-42.log number = 42 , length = 34367661, 43=db-43.log number = 43 , length = 34465374, 40=db-40.log number = 40 , length = 33902398, 41=db-41.log number = 41 , length = 34517946, 46=db-46.log number = 46 , length = 34099281, 47=db-47.log number = 47 , length = 33892697, 44=db-44.log number = 44 , length = 34597832, 45=db-45.log number = 45 , length = 33978223, 51=db-51.log number = 51 , length = 34169738, 50=db-50.log number = 50 , length = 33968924, 49=db-49.log number = 49 , length = 34066799, 48=db-48.log number = 48 , length = 34259593, 54=db-54.log number = 54 , length = 34113708, 53=db-53.log number = 53 , length = 33559631, 52=db-52.log number = 52 , length = 33955516}

      ERROR | Failed to start ActiveMQ JMS Message Broker. Reason: java.io.IOException: Could not locate data file /run/ca/activemq/data/BR0/KahaDBMessageStore/db-55.log

      java.io.IOException: Could not locate data file /run/ca/activemq/data/BR0/KahaDBMessageStore/db-55.log

              at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:325)

              at org.apache.kahadb.journal.Journal.getNextLocation(Journal.java:482)

              at org.apache.activemq.store.kahadb.MessageDatabase.getRecoveryPosition(MessageDatabase.java:584)

              at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:387)

              at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:280)

              at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:321)

              at org.apache.activemq.store.kahadb.MessageDatabase.start(MessageDatabase.java:182)

              at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:183)

              at org.apache.activemq.broker.BrokerService.start(BrokerService.java:469)

              at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:85)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1414)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1375)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1335)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:473)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:409)

              at java.security.AccessController.doPrivileged(Native Method)

              at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:380)

              at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:264)

              at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

              at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:261)

              at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:185)

              at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)

              at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:429)

              at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:728)

              at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:380)

              at org.apache.xbean.spring.context.ResourceXmlApplicationContext.(ResourceXmlApplicationContext.java:52)

              at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:96)

              at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:52)

              at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)

              at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)

              at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)

              at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:91)

              at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)

              at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:136)

              at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)

              at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:82)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

              at java.lang.reflect.Method.invoke(Method.java:597)

              at org.apache.activemq.console.Main.runTaskClass(Main.java:251)

              at org.apache.activemq.console.Main.main(Main.java:107)

      INFO | ActiveMQ Message Broker (BR0.QA.US.STUXSH01.ST9769, null) is shutting down

      INFO | Network Connector NC00.QA.9 Stopped

      INFO | Connector TC00.QA.ISP.ST9769 Stopped

      INFO | Connector TC01.QA.ISP.ST9769 Stopped

      INFO | ActiveMQ JMS Message Broker (BR0.QA.US.STUXSH01.ST9769, null) stopped