Error indexing root node in portal-cms
Currently we are using Jboss portal 2.6.4 with Jboss AS 4.2.2-GA. Both of which were downloaded as binary archives from the main download site. We use Mysql 5 for our database and running on Linux.
We have not had any problems with the portal-cms or portal server until recently. Upon startup of the portal system, the CMS fails to initialize with the following error:
20:53:47,172 ERROR [RepositoryImpl] Failed to initialize workspace 'default' javax.jcr.RepositoryException: Error indexing root node: a9866f0e-1480-4556-b851-16a57747818c: Error indexing root node: a9866f0e-1480-4556-b851-16a57747818c: Error indexing root node: a9866f0e-1480-4556-b851-16a57747818c at org.apache.jackrabbit.core.SearchManager.initializeQueryHandler(SearchManager.java:525) at org.apache.jackrabbit.core.SearchManager.<init>(SearchManager.java:267) at org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.getSearchManager(RepositoryImpl.java:1509) at org.apache.jackrabbit.core.RepositoryImpl.initWorkspace(RepositoryImpl.java:546) at org.apache.jackrabbit.core.RepositoryImpl.<init>(RepositoryImpl.java:254) at org.apache.jackrabbit.core.RepositoryImpl.create(RepositoryImpl.java:497) at org.jboss.portal.cms.impl.jcr.jackrabbit.JackrabbitJCRService.start(JackrabbitJCRService.java:104) at org.jboss.portal.cms.impl.jcr.JCRCMS.startJCR(JCRCMS.java:302) at org.jboss.portal.cms.impl.jcr.JCRCMS.startService(JCRCMS.java:267) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289) at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196) at org.jboss.portal.jems.as.system.AbstractJBossService.start(AbstractJBossService.java:73) at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.portal.jems.as.system.JBossServiceModelMBean$ServiceMixin.execute(JBossServiceModelMBean.java:486) at org.jboss.portal.jems.as.system.JBossServiceModelMBean$ServiceMixin.startService(JBossServiceModelMBean.java:452) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289) at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196) at org.jboss.portal.jems.as.system.JBossServiceModelMBean$6.invoke(JBossServiceModelMBean.java:374) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995) at $Proxy0.start(Unknown Source) at org.jboss.system.ServiceController.start(ServiceController.java:417) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy189.start(Unknown Source) at org.jboss.deployment.SimpleSubDeployerSupport.startService(SimpleSubDeployerSupport.java:356) at org.jboss.deployment.SimpleSubDeployerSupport.start(SimpleSubDeployerSupport.java:127) at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy9.deploy(Unknown Source) at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421) at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263) at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289) at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978) at $Proxy0.start(Unknown Source) at org.jboss.system.ServiceController.start(ServiceController.java:417) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy4.start(Unknown Source) at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302) at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy5.deploy(Unknown Source) at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482) at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362) at org.jboss.Main.boot(Main.java:200) at org.jboss.Main$1.run(Main.java:508) at java.lang.Thread.run(Thread.java:595) Caused by: java.io.IOException: Error indexing root node: a9866f0e-1480-4556-b851-16a57747818c at org.apache.jackrabbit.core.query.lucene.MultiIndex.<init>(MultiIndex.java:284) at org.apache.jackrabbit.core.query.lucene.SearchIndex.doInit(SearchIndex.java:229) at org.apache.jackrabbit.core.query.AbstractQueryHandler.init(AbstractQueryHandler.java:44) at org.apache.jackrabbit.core.SearchManager.initializeQueryHandler(SearchManager.java:523) ... 119 more java.io.IOException: Error indexing root node: a9866f0e-1480-4556-b851-16a57747818c at org.apache.jackrabbit.core.query.lucene.MultiIndex.<init>(MultiIndex.java:284) at org.apache.jackrabbit.core.query.lucene.SearchIndex.doInit(SearchIndex.java:229) at org.apache.jackrabbit.core.query.AbstractQueryHandler.init(AbstractQueryHandler.java:44) at org.apache.jackrabbit.core.SearchManager.initializeQueryHandler(SearchManager.java:523) at org.apache.jackrabbit.core.SearchManager.<init>(SearchManager.java:267) at org.apache.jackrabbit.core.RepositoryImpl$WorkspaceInfo.getSearchManager(RepositoryImpl.java:1509) at org.apache.jackrabbit.core.RepositoryImpl.initWorkspace(RepositoryImpl.java:546) at org.apache.jackrabbit.core.RepositoryImpl.<init>(RepositoryImpl.java:254) at org.apache.jackrabbit.core.RepositoryImpl.create(RepositoryImpl.java:497) at org.jboss.portal.cms.impl.jcr.jackrabbit.JackrabbitJCRService.start(JackrabbitJCRService.java:104) at org.jboss.portal.cms.impl.jcr.JCRCMS.startJCR(JCRCMS.java:302) at org.jboss.portal.cms.impl.jcr.JCRCMS.startService(JCRCMS.java:267) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289) at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196) at org.jboss.portal.jems.as.system.AbstractJBossService.start(AbstractJBossService.java:73) at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.portal.jems.as.system.JBossServiceModelMBean$ServiceMixin.execute(JBossServiceModelMBean.java:486) at org.jboss.portal.jems.as.system.JBossServiceModelMBean$ServiceMixin.startService(JBossServiceModelMBean.java:452) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289) at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:196) at org.jboss.portal.jems.as.system.JBossServiceModelMBean$6.invoke(JBossServiceModelMBean.java:374) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:995) at $Proxy0.start(Unknown Source) at org.jboss.system.ServiceController.start(ServiceController.java:417) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at org.jboss.system.ServiceController.start(ServiceController.java:435) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy189.start(Unknown Source) at org.jboss.deployment.SimpleSubDeployerSupport.startService(SimpleSubDeployerSupport.java:356) at org.jboss.deployment.SimpleSubDeployerSupport.start(SimpleSubDeployerSupport.java:127) at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782) at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy9.deploy(Unknown Source) at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:421) at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:634) at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263) at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:336) at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:289) at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:245) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978) at $Proxy0.start(Unknown Source) at org.jboss.system.ServiceController.start(ServiceController.java:417) at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy4.start(Unknown Source) at org.jboss.deployment.SARDeployer.start(SARDeployer.java:302) at org.jboss.deployment.MainDeployer.start(MainDeployer.java:1025) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:819) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:782) at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:766) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142) at org.jboss.mx.server.Invocation.invoke(Invocation.java:88) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210) at $Proxy5.deploy(Unknown Source) at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:482) at org.jboss.system.server.ServerImpl.start(ServerImpl.java:362) at org.jboss.Main.boot(Main.java:200) at org.jboss.Main$1.run(Main.java:508) at java.lang.Thread.run(Thread.java:595) 20:53:47,176 ERROR [RepositoryImpl] Unable to start repository, forcing shutdown... 20:53:47,177 INFO [RepositoryImpl] Shutting down repository... 20:53:47,180 INFO [RepositoryImpl] shutting down workspace 'default'... 20:53:47,181 INFO [ObservationDispatcher] Notification of EventListeners stopped. 20:53:47,184 INFO [RepositoryImpl] workspace 'default' has been shutdown
I commented out the "SearchIndex" option in the workspace.xml located in ./server/default/data/portal/cms/conf/workspaces/default/workspace.xml
This allows the CMS to start up correctly and it "appears" to function as normal.
I have tried the following to resolve the issue:
- removed the index directory to force a rebuild
- set the consistency check in workspace.xml to true
- copying the database to a different system (produced identical errors)
I believe the problem is sort of similar to this issue:
https://issues.apache.org/jira/browse/JCR-964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12519548
Is there any way to identify what is referring to this missing/corrupted object? Right now I feel that the only solution to clear up this problem will be an entire export of the CMS with a reimport.
I am trying to determine:
How can I prevent this from happening again?
What was the root cause of the problem?
Is there any way to clean up the problem outside of an export/import?
Any help would be much appreciated.
thanks,