Modeshape 5 event buffer locking up?
Hello all,
I am from the University of Maryland Libraries, where we are using Fedora 4 [1] as our digital repositories tool. Recently, we have been doing some mass ingests of objects into our repository, and are encountering problems with stuck threads in Tomcat. The stack traces for these stuck threads seem to indicate that there is something going on in the Modeshape event buffer locking:
Aug 22, 2017 10:44:02 AM org.apache.catalina.valves.StuckThreadDetectionValve notifyStuckThreadDetected
WARNING: Thread "http-bio-9601-exec-212" (id=28634) has been active for 39,738 milliseconds (since 8/22/17 10:43 AM) to serve the same request for https://fcrepo.lib.umd.edu/fcrepo/rest/tx:766af166-6106-40f0-bd3e-583b73f18f58/fcr:tx/fcr:commit and may be stuck (configured threshold for this StuckThreadDetectionValve is 30 seconds). There is/are 4 thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at org.modeshape.common.collection.ring.RingBuffer.add(RingBuffer.java:153)
at org.modeshape.jcr.bus.RepositoryChangeBus.notify(RepositoryChangeBus.java:180)
at org.modeshape.jcr.cache.document.WorkspaceCache.changed(WorkspaceCache.java:320)
at org.modeshape.jcr.txn.Transactions.updateCache(Transactions.java:296)
at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:748)
at org.modeshape.jcr.JcrSession.save(JcrSession.java:1179)
We have tried both increasing the JVM max heap for the Fedora 4 app (currently at 8GB, running on a VM with a total of 16GB), and increasing the Modeshape event buffer size from the default 1024 to 2048. However, both of these solutions have not eliminated the stuck thread problems. So far, the only thing that has is to turn of the Solr indexing that was running concurrently with the ingest process.
Could this be a case where too many concurrent read and write requests to Modeshape are somehow clogging up the event buffer in a way that it cannot clear itself and resume normal activity? Does the above stack trace provide any other indication of what might be going on?
System stats:
* RHEL 6.9
* Java 1.8u65
* Tomcat 7.0.77 with a JVM max heap 8GB
* Fedora 4.7.4 built with Modeshape 5.3.0.Final
Thanks,
-Peter Eichman
[1] GitHub - fcrepo4/fcrepo4 at fcrepo-4.7.4
--
Peter Eichman
Senior Software Developer
University of Maryland Libraries