Reading threads are blocked
We are currently testing the 2.0 CR2 in a load intensive environment. I started to notice a lot of blocked threads that were blocked by the cache.
In order to investigate further I constructed a simple load test on a cache. The cache uses the standard replAsync-service.xml
This is what I do:
1. Create a cache
2. Initialize the cache with some nodes and data
3. Start a large number of concurrent readers
4. Observe the threads and monitors in the system using jprofiler
2. Initialize the cache with some nodes and data
3. Start a large number of concurrent readers
4. Observe the threads and monitors in the system using jprofiler
Contents of the created and initialized cache:
DEBUG CacheLockGetTest - null {}
/d {}
/g {1=Data}
/h {1=Data}
/f {1=Data}
/e {1=Data}
/b {}
/g {1=Data}
/h {1=Data}
/f {1=Data}
/e {1=Data}
/a {}
/g {1=Data}
/h {1=Data}
/f {1=Data}
/e {1=Data}
/c {}
/g {1=Data}
/h {1=Data}
/f {1=Data}
/e {1=Data}
The task used to read from the cache:
private class CacheExecutor implements Runnable {
private final Fqn fqn;
public CacheExecutor(Fqn fqn) { this.fqn = fqn; }
public void run() {
cache.get(fqn, 1);
}
}
The cache has been initialized before I start the readers, so that when the readers are running no write occurs on the cache. Now, I thought that read operations on the cache would be concurrent but when inspecting the threads in jprofiler I got this:
http://www.robotsociety.com/cache/threads.GIF
If we look at the contending threads' stack traces we see that they all block against UnversionedNode (See: http://www.robotsociety.com/cache/monitors.GIF)
I have looked at unversioned node and there is much synchronization going on in there. I am not sure why the blocking is so frequent and I also saw threads working in different regions blocking each other.
One question is, if I access a/b/c, will a read lock be acquired for all those nodes, if so then we will ultimately have synchronization on the common root node for all read operations in the cache, correct?
I understand that the test above is a bit artificial since we do not run any other type of logic that would permit pauses for access, but I also run into this blocking issues in our server where we do execute other functionality.
Now, our cache is pretty large so I will try and divide the regions into separate caches to minimize the synchronization issues or try to find some other workaround, but shouldn't a cache solution permit more concurrent access rather then having what looks like a global sync lock?
On a side note, after running a while the UnversionedNode sync lock is replaced by the readOwnerList_ in LockMap (see: http://www.robotsociety.com/cache/monitors_copyarray.GIF).
