We're loading the following system:
Hibernate 3.3.1.GA
JBC 3.0.3.GA w/ invalidation_async as Transactional 2nd level cache (MVCC)
JBossTM JTA 4.6.0.GA
Tomcat 6 NIO
sun JVM 1.6

Cache's read/write ratio is around 100:1

We're noticing the following problems:
1. heap consumption much higher than expected
2. average latency increases at the same rate as # of concurrent users (not much scale)

I wonder whether #2 is affected by concurrencyLevel, currently set at 1000. If I understand correctly, increasing concurrencyLevel should decrease chances of lock contention and lock waiting times at the expense of memory. Correct? (our lockAcquisitionTimeout is set to 500ms, we retry tx up to 3 times on TimeoutException)

If so,
- is there a way to track average time spent waiting to acquire a lock?
- is there a way to track percentage of lock acquisition attempts that have to wait at all?
- are locks released at the end of Invocation or Transaction?
- any guesstimate of memory/other implication is concurrencyLevel were to be changed from 1000 to 10000?

Any other hints on tuning JBC3/Hibernate perf @ concurrency?

thanks
-nikita