Modeshape event bus ring buffer size

[mohideen]

Hello,

I am Mohamed from the University of Maryland Libraries. We have been trying to figure out a problem related to a stuck thread on our Fedora repository that uses Modeshape 5.1.0.Final. After reading Using a ring buffer for events in 4.0 | ModeShape and https://issues.jboss.org/browse/MODE-2195, I increased the eventBusSize to 2048 and that did delay the stuck thread problem. Initially, the problem was happening on the 4th transaction (where each transaction had about 2000 items) and after increasing the bus size, the problem happened only on the 21st transaction. 

I have a couple of question on this:

1. How to identify the consumer that is causing the ring buffer to block?
   1. How to debug/log the event bus?
   2. How to monitor the listeners on the event bus?
2. What is the impact of increasing the eventBusSize to an even larger value? (Performance/Memory overhead?)

I would really appreciate any help on this issue.

Fedora Tech Post: https://groups.google.com/forum/?fromgroups=#!topic/fedora-tech/0CletEhKO7s

Stacktrace

Jun 30, 2017 4:25:39 PM org.apache.catalina.valves.StuckThreadDetectionValve notifyStuckThreadDetected
WARNING: Thread "http-nio-9601-exec-4" (id=54) has been active for 33,205 milliseconds (since 6/30/17 4:25 PM) to serve the same request for https://fcrepolocal/fcrepo/rest/tx:0dc4a797-bb7e-4651-a3a3-bba3d9c73aef/fcr:tx/fcr:commit and may be stuck (configured threshold for this StuckThreadDetectionValve is 30 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck.
java.lang.Throwable
 at sun.misc.Unsafe.park(Native Method)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
 at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
 at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
 at org.modeshape.common.collection.ring.RingBuffer.add(RingBuffer.java:153)
 at org.modeshape.jcr.bus.RepositoryChangeBus.notify(RepositoryChangeBus.java:180)
 at org.modeshape.jcr.cache.document.WorkspaceCache.changed(WorkspaceCache.java:320)
 at org.modeshape.jcr.txn.Transactions.updateCache(Transactions.java:296)
 at org.modeshape.jcr.cache.document.WritableSessionCache.save(WritableSessionCache.java:800)
 at org.modeshape.jcr.JcrSession.save(JcrSession.java:1162)
...

See attachment for the full stack trace.

Thanks

Mohamed

University of Maryland Libraries

[mohideen]

Hello,

As It has been a couple of weeks since my initial post, I wanted to check with the community again on any inputs regarding the debugging/logging of the Modeshape event bus.

rhauch and hchiorean, I see both your names on the PR #1461 for MODE-2195. Could you please share your thought on this? It would be very helpful.

Thanks

Mohamed