4 Replies Latest reply on Jul 1, 2016 12:49 PM by amischler

    Restoring an inconsistent database

    vdepauw
    vdepauw Jun 29, 2016 4:46 AM

      Hi everyone,

       

      I have a corrupted database from which I'd like to restore data. It's impossible for me to open it because modeshape is unable to read namespaces and nodeTypes at loading.

      So what is the proper way to read the rest of the file ? Is it possible to rebuild a clean system/namespace node ?

       

      I'm working with modeshape 4.6.0.Final.

       

      Here is the complete stack trace:

       

      10:35:47.099 [JTP Slot 2] DEBUG org.infinispan.cache.impl.CacheImpl - Started cache dsdk-repository/system on null

      10:35:47.101 [JTP Slot 2] DEBUG o.m.jcr.cache.RepositoryCache - The 'dsdk-repository/system' workspace cache is using the cache configuration: '[I REMOVED THIS LINE TOO LONG...]'

      10:35:47.135 [JTP Slot 2] DEBUG o.m.jcr.cache.RepositoryCache - System root: Node e03bade317f1e7/: { "properties" : { "http://www.jcp.org/jcr/1.0" : { "primaryType" : { "$name" : "mode:root" } , "uuid" : "e03bade317f1e7/" } } , "children" : [ { "key" : "e03bade317f1e7jcr:system" , "name" : "jcr:system" } ] , "childrenInfo" : { "count" : 1 } }

      10:35:47.138 [JTP Slot 2] DEBUG o.m.jcr.cache.RepositoryCache - jcr:system child reference: jcr:system (key=e03bade317f1e7jcr:system)

      10:35:47.139 [JTP Slot 2] DEBUG o.m.jcr.cache.RepositoryCache - System node: Node e03bade317f1e7jcr:system: [I REMOVED THIS LINE TOO LONG...]

      10:35:47.139 [JTP Slot 2] DEBUG o.m.jcr.cache.RepositoryCache - Found existing 'system' workspace in repository 'dsdk-repository'

      10:35:47.193 [JTP Slot 2] ERROR o.m.jcr.SystemNamespaceRegistry - Node types were read from the system content, and appear to be inconsistent or invalid: dsdk-repository

      java.lang.NullPointerException: null

        at org.modeshape.jcr.SystemContent.readAllNamespaces(SystemContent.java:941) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.SystemNamespaceRegistry.refreshFromSystem(SystemNamespaceRegistry.java:75) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1153) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:980) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:388) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:653) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:624) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:145) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at com.dooapp.jcr.viewer.model.RepositoryConnection.connect(RepositoryConnection.java:34) [classes/:na]

        at com.dooapp.jcr.viewer.service.PersistenceService.doLoadRepository(PersistenceService.java:49) [classes/:na]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]

        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]

        at org.jrebirth.af.core.service.ServiceTaskBase.call(ServiceTaskBase.java:156) [core-8.0.4.jar:na]

        at javafx.concurrent.Task$TaskCallable.call(Task.java:1423) [jfxrt.jar:na]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]

        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

      10:35:47.195 [JTP Slot 2] ERROR o.m.jcr.RepositoryNodeTypeManager - Node types were read from the system content, and appear to be inconsistent or invalid: dsdk-repository

      java.lang.NullPointerException: null

        at org.modeshape.jcr.SystemContent.readAllNodeTypes(SystemContent.java:786) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.RepositoryNodeTypeManager.refreshFromSystem(RepositoryNodeTypeManager.java:743) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1155) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:980) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:388) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:653) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:624) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:145) [modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at com.dooapp.jcr.viewer.model.RepositoryConnection.connect(RepositoryConnection.java:34) [classes/:na]

        at com.dooapp.jcr.viewer.service.PersistenceService.doLoadRepository(PersistenceService.java:49) [classes/:na]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]

        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]

        at org.jrebirth.af.core.service.ServiceTaskBase.call(ServiceTaskBase.java:156) [core-8.0.4.jar:na]

        at javafx.concurrent.Task$TaskCallable.call(Task.java:1423) [jfxrt.jar:na]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]

        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

      10:35:47.265 [JTP Slot 2] DEBUG o.m.j.c.d.WritableSessionCache - Attempting to lock keys in Infinispan: [I REMOVED THIS LINE TOO LONG...]

      10:35:47.274 [JTP Slot 2] DEBUG o.m.j.c.d.WritableSessionCache - Locked the nodes: [I REMOVED THIS LINE TOO LONG...]

      10:35:47.344 [JTP Slot 2] DEBUG o.i.x.s.XSiteStateConsumerImpl - Ending state transfer from null

      10:35:47.357 [JTP Slot 2] DEBUG org.infinispan.cache.impl.CacheImpl - Stopping cache dsdk-repository/system on null

      10:35:47.359 [JTP Slot 2] DEBUG o.i.t.impl.TransactionTable - Wait for on-going transactions to finish for 30 seconds.

      10:35:47.359 [JTP Slot 2] DEBUG o.i.t.impl.TransactionTable - All transactions terminated

      10:35:47.361 [JTP Slot 2] DEBUG o.i.manager.DefaultCacheManager - Stopping cache manager ISPN on null

      10:35:47.361 [JTP Slot 2] DEBUG o.i.manager.DefaultCacheManager - Stopping cache manager ISPN on null

      10:35:47.362 [JTP Slot 2] DEBUG org.infinispan.cache.impl.CacheImpl - Stopping cache persisted-repository on null

      10:35:47.363 [JTP Slot 2] DEBUG o.i.t.impl.TransactionTable - Wait for on-going transactions to finish for 30 seconds.

      10:35:47.363 [JTP Slot 2] DEBUG o.i.t.impl.TransactionTable - All transactions terminated

      10:35:47.364 [JTP Slot 2] ERROR c.d.j.v.service.PersistenceService - Error while starting 'dsdk-repository' repository: e03bade317f1e7/jcr:system/jcr:nodeTypes/nt:base

      javax.jcr.RepositoryException: Error while starting 'dsdk-repository' repository: e03bade317f1e7/jcr:system/jcr:nodeTypes/nt:base

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:655) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:624) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:145) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at com.dooapp.jcr.viewer.model.RepositoryConnection.connect(RepositoryConnection.java:34) ~[classes/:na]

        at com.dooapp.jcr.viewer.service.PersistenceService.doLoadRepository(PersistenceService.java:49) ~[classes/:na]

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_51]

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_51]

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_51]

        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_51]

        at org.jrebirth.af.core.service.ServiceTaskBase.call(ServiceTaskBase.java:156) [core-8.0.4.jar:na]

        at javafx.concurrent.Task$TaskCallable.call(Task.java:1423) [jfxrt.jar:na]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_51]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_51]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_51]

        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

      Caused by: org.modeshape.jcr.cache.DocumentAlreadyExistsException: e03bade317f1e7/jcr:system/jcr:nodeTypes/nt:base

        at org.modeshape.jcr.cache.document.WritableSessionCache.persistChanges(WritableSessionCache.java:1343) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:488) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:443) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.SystemContent.save(SystemContent.java:120) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.RepositoryNodeTypeManager.registerNodeTypes(RepositoryNodeTypeManager.java:532) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1160) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:980) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:388) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:653) ~[modeshape-jcr-4.6.0.Final.jar:4.6.0.Final]

        ... 14 common frames omitted

      • 497 Views
      • Tags:
        • 1. Re: Restoring an inconsistent database
          hchiorean
          hchiorean Jun 29, 2016 5:07 AM (in response to vdepauw)

          If you have a backup file (i.e. a ModeShape created backup file) you can restore it after you've dropped the old database, started a new empty repository and called restore() (in other words using ModeShape's backup & restore)

          If you don't have a backup file and your database is corrupted, there's not much you can do unfortunately except drop it entirely. Data in ModeShape 4.x is stored in the DB in an Infinispan binary format which is not editable by hand.

            • Actions
          • 2. Re: Restoring an inconsistent database
            amischler
            amischler Jul 1, 2016 9:46 AM (in response to hchiorean)

            Thanks for your answer. Actually, the Infinispan file itself is not corrupted. It looks like only the system node is in an inconsistent state. By ignoring a few errors at startup I managed to run a backup and to export all repository documents. The data we want to restore seems to be present in this file.

             

            However, when I restore this backup, I still get errors regarding the system node :

             

            [ERROR]2016-07-01 15:31:43,617 o.m.j.RepositoryNodeTypeManager {JTP Slot 4} - Node types were read from the system content, and appear to be inconsistent or invalid: dsdk-repository

            org.modeshape.common.SystemFailureException: javax.jcr.nodetype.ConstraintViolationException: org.modeshape.jcr.value.ValueFormatException: Error converting "nt:base" from String to a Name

            at org.modeshape.jcr.SystemContent.readNodeTypeDefinition(SystemContent.java:832) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.SystemContent.readAllNodeTypes(SystemContent.java:788) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.RepositoryNodeTypeManager.refreshFromSystem(RepositoryNodeTypeManager.java:743) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:1157) [modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.JcrRepository$RunningState.<init>(JcrRepository.java:980) [modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:388) [modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:653) [modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:614) [modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:145) [modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at com.dooapp.dsdk.persistence.impl.ModeshapePersistenceService.init(ModeshapePersistenceService.java:77) [classes/:na]
            at com.dooapp.dsdk.core.persistence.JcromService.init(JcromService.java:235) [classes/:na]
            at com.dooapp.dsdk.core.persistence.JcromService.doInitJcrom(JcromService.java:151) [classes/:na]
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_92]
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_92]
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_92]
            at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_92]
            at org.jrebirth.af.core.service.ServiceTaskBase.call(ServiceTaskBase.java:156) [core-8.0.5.jar:na]
            at javafx.concurrent.Task$TaskCallable.call(Task.java:1423) [jfxrt.jar:na]
            at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_92]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_92]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_92]
            at java.lang.Thread.run(Thread.java:745) [na:1.8.0_92]

            Caused by: javax.jcr.nodetype.ConstraintViolationException: org.modeshape.jcr.value.ValueFormatException: Error converting "nt:base" from String to a Name

            at org.modeshape.jcr.JcrNodeTypeTemplate.setName(JcrNodeTypeTemplate.java:145) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.SystemContent.readNodeTypeDefinition(SystemContent.java:801) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            ... 21 common frames omitted

            Caused by: org.modeshape.jcr.value.ValueFormatException: Error converting "nt:base" from String to a Name

            at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:157) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:85) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:46) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            at org.modeshape.jcr.JcrNodeTypeTemplate.setName(JcrNodeTypeTemplate.java:143) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            ... 22 common frames omitted

            Caused by: org.modeshape.jcr.value.NamespaceException: There is no namespace registered for the prefix "nt"

            at org.modeshape.jcr.value.basic.NameValueFactory.create(NameValueFactory.java:148) ~[modeshape-jcr-4.7-SNAPSHOT.jar:4.7-SNAPSHOT]
            ... 25 common frames omitted

             

            I tried to clear the system node during the RepositoryCache initialization to force Modeshape to rebuild the default system node and to re-import the namespaces and nodetypes from our cnd file later. But saving the session fails because some nodes are strongly referencing nodes in the system node (primary types or mixins) which we have just removed.

             

            Any idea how we could fix the system node and re-configure it cleanly using our cnd file ?

              • Actions
            • 3. Re: Restoring an inconsistent database
              hchiorean
              hchiorean Jul 1, 2016 11:02 AM (in response to amischler)

              Any idea how we could fix the system node and re-configure it cleanly using our cnd file ?

              unfortunately no. If the system area is corrupted, there is no way to fix that, since information from it is being referenced and used everywhere internally.

                • Actions
              • 4. Re: Restoring an inconsistent database
                amischler
                amischler Jul 1, 2016 12:49 PM (in response to hchiorean)

                We fixed this issue by comparing the current backup with an older backup and copied all system namepace nodes from the older backup into the broken one. Then, we restored the fixed backup and everything loads fine now !

                 

                The only mysterious thing about it is how the system node went corrupted although our customer did not experience any crash or power failure. To prevent future cases like this one, does anybody have an idea about what can corrupt the system node like this ?

                  • Actions