Cannot start the repository when configuring Modeshape as a cluster

[folch]

Hi,

I'm having some strange issues when I try to setup a Modeshape cluster using Oracle 11 as a backend. and using Modeshape 4.0.4

It works in most of the enviroments but we have a special setup where it fails systematically when starting the repository for the first time and trying to login.

[4/20/16 9:42:25:514 CEST] 00000003 ThreadMonitor W   WSVR0605W: Thread "server.startup : 0" (0000000a) has been active for 714598 milliseconds and may be hung.  There is/are 1 thread(s) in total in the server that may be hung.

  at sun.misc.Unsafe.park(Native Method)

  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:182)

  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:822)

  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:980)

  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1292)

  at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:229)

  at java.util.concurrent.FutureTask.get(FutureTask.java:94)

  at org.hibernate.search.backend.impl.lucene.LuceneBackendQueueProcessor.applyWork(LuceneBackendQueueProcessor.java:97)

  at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.performOperations(DirectoryBasedIndexManager.java:138)

  at org.hibernate.search.backend.impl.WorkQueuePerIndexSplitter.commitOperations(WorkQueuePerIndexSplitter.java:63)

  at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:96)

  at org.hibernate.search.backend.impl.TransactionalWorker.performWork(TransactionalWorker.java:104)

  at org.modeshape.jcr.query.lucene.basic.BasicLuceneSchema.updateIndex(BasicLuceneSchema.java:436)

  at org.modeshape.jcr.RepositoryQueryManager.reindexContent(RepositoryQueryManager.java:385)

  at org.modeshape.jcr.RepositoryQueryManager.reindexContent(RepositoryQueryManager.java:262)

  at org.modeshape.jcr.RepositoryQueryManager.reindexContent(RepositoryQueryManager.java:228)

  at org.modeshape.jcr.JcrRepository$RunningState.postInitialize(JcrRepository.java:1324)

  at org.modeshape.jcr.JcrRepository.doStart(JcrRepository.java:408)

  at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:648)

  at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:609)

  at org.modeshape.jcr.JcrRepository.login(JcrRepository.java:155)

I can send you config files if needed, but I don't understand what the issue could be as I can run the same configuration in my laptop without any issue.

Thanks in advance

[hchiorean]

There is no official 4.0.4 release that I can remember. Moreover, ModeShape 4 stopped using Hibernate Search and Lucene out-of-the-box and implemented the new indexing mechanism (your stack trace suggests otherwise)

Since it's happening only in a certain environment, you'll have to investigate locally. The current supported community version is 5.0.0.Final.

[folch]

My fault.

Modeshape version is 3.8.1.

It's an old instance. Do you have any clue about what the issue could be? Have you seen it?

If we configure Modeshape not as a cluster it works.

Thanks

[hchiorean]

It's not something which I've seen before, no. But there are many variables: the stack trace indicates a Hibernate Search/Lucene indexing call, so nothing related to data/binary persistence. Are you sure this is the case ?

The stack trace indicates that Hibernate Search is waiting for something to complete, so the problem may lie somewhere else.

What does your indexing/querying configuration look like ? Have you tried configuring it to be asynchronous to see if the problem goes away ? (i.e. a reindexing background thread may "eventually" complete). Are you using local (per cluster node) indexes or are your indexes distributed as well ?

3.8.1 community is an extremely old version and we've fixed numerous bugs since then, so the only thing I can recommend in these cases is to debug/investigate the issue locally and track where it's coming from.

[folch]

Thanks for you answer Horia,

I've attached the infinispan_config and modehsape config, just in case you find something evident.

We are using local indexes per cluster node.

I know that 3.8.1 is old but we sitll have some instances in production that are not still migrated to the new Modeshape version. Sometimes bussiness process take too long .

I'll try with asynch index, just to have more information.

[hchiorean]

I don't see any attachments on this thread.

My main suggestion would be to look at all the other threads (other than the posted stack) since the problem lies elsewhere. That particular piece of code submits either a Synchronous or Asynchronous job to perform the actual Lucene indexing (i.e. write data to the indexes). So there must be at least one other thread stuck on doing "something meaningful".

If the problem is indeed with Lucene, after you have an idea what it can be (for example file locking), you can try changing/setting some advanced configuration options (see Getting Started - ModeShape 3 - Project Documentation Editor , namely the JSON example).

[folch]

Files are attached now to my previous comment.

I'll try what you mention about synchronous/asynchronous indexing and also I'll investigate what are doing other threads.

Thanks again.