Concurrency issue in OrderedSynchronizationHandler (2.0.0GA

[fungrim]

Hi,

Two issues/questions regarding OrderedSynchronizationHandler.

1) The "instances" member is not synchronized
OrderedSynchronizationHandler seems to have an unguarded static HashMap member called "instances". This is a potentional crash issue. On one of our instances (in a four member cluster) with medium high load (aprox 1500 transaction per second), the CPU flat-lined on 200% with 21 threads caught and spinning in HashMap.get(Object) as called from OrderedSynchronizationHandler.getInstance(Transaction). I'll copy a typical stack trace below.

2) The "instances" member is static
This is more of a question. If several caches are run within the same VM/ClassLoader they will all share the "instances" member. This seems fairly inefficient, especially if the map in question needs to be synchronized as per #1.

Below is a typical, slightly trunctated, stack trace for issue #1.

"Incoming Thread,TableSpace,172.16.0.2:8786" daemon prio=1 tid=0x097e9208 nid=0x14de runnable [0x83228000..0x83229030]
 at java.util.HashMap.get(HashMap.java:346)
 at org.jboss.cache.interceptors.OrderedSynchronizationHandler.getInstance(OrderedSynchronizationHandler.java:50)
 at org.jboss.cache.interceptors.TxInterceptor.registerHandler(TxInterceptor.java:901)
 at org.jboss.cache.interceptors.TxInterceptor.handleRemotePrepare(TxInterceptor.java:246)
 at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:100)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76)
 at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:123)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76)
 at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:62)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76)
 at com.cubeia.space.cache.HaltableInterceptor.invoke(HaltableInterceptor.java:32)
 at org.jboss.cache.CacheImpl.invokeMethod(CacheImpl.java:3939)
 at org.jboss.cache.CacheImpl._replicate(CacheImpl.java:2853)

What say you?
/Lars J. Nilsson
Cubeia Ltd, UK Fillial

[fungrim]

The title was truncated, but it went on to say that this is an issue in 2.1.0.CR3 as well.

[fredrikj]

I just experienced what might be the same problem.

I started up 6 nodes and one node was using up all CPU without any real load on it. Dumping all stacks shows that all threads are parked except one which is stuck on map.get in OrderedSynchronizationHandler.

"ReceivingGameEventDaemon-1" prio=1 tid=0x092d5eb0 nid=0x5660 runnable [0x85421000..0x85421f30]
 at java.util.HashMap.get(HashMap.java:346)
 at org.jboss.cache.interceptors.OrderedSynchronizationHandler.getInstance(OrderedSynchronizationHandler.java:50)
 at org.jboss.cache.interceptors.TxInterceptor.registerHandler(TxInterceptor.java:901)
 at org.jboss.cache.interceptors.TxInterceptor.registerTransaction(TxInterceptor.java:877)
 at org.jboss.cache.interceptors.TxInterceptor.attachGlobalTransaction(TxInterceptor.java:342)
 at org.jboss.cache.interceptors.TxInterceptor.handleNonTxMethod(TxInterceptor.java:292)
 at org.jboss.cache.interceptors.TxInterceptor.invoke(TxInterceptor.java:131)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76)
 at org.jboss.cache.interceptors.CacheMgmtInterceptor.invoke(CacheMgmtInterceptor.java:81)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76)
 at org.jboss.cache.interceptors.InvocationContextInterceptor.invoke(InvocationContextInterceptor.java:62)
 at org.jboss.cache.interceptors.Interceptor.invoke(Interceptor.java:76)

[manik]

As with all problems of this nature, this is pretty hard to reproduce and I haven't been successful at it, but I can see how it is a problem, both in being not thread safe as well as a static context variable potentially shared across cache instances in the same VM.

In general, the whole use of statics here is pretty ugly - I have changed this in HEAD so if you guys have the time, I'd appreciate your giving it a try in HEAD.

The change is simple and should be very low risk - OSH has no static variables anymore; instead it is instantiated when a TransactionEntry is created and the ref is stored in and obtained from the TransactionEntry, rather than the factory method on OSH.

[manik]

http://jira.jboss.org/jira/browse/JBCACHE-1304

[fungrim]

Brilliant. Thanks for the attention. I'll see if I'll have time later this week to give head a swirl.

[manik]

Thank you for spotting and reporting this - helps us build a better product. :-)

[metastaz]

Hi,

We observe same issue in 1.4.1 SP5. Are there any plans on getting it fixed in 1.x code base?
If not how safe is it to use latest stable 2.x release with 4.2.x JBoss AS?

Thank you,
Stas.

[metastaz]

Hi,

Does anybody know anything with regards to the questions above?

Thank you,
Stas.

[manik]

There are no plans to fix this in 1.x. The latest stable in 2.x is pretty stable, although I'd look at http://wiki.jboss.org/wiki/JBossCacheAsCompatibility about using it with AS 4.x.