Synchronization errors in pooling implementation?
Hi,
We're running JBoss 3.2.3 with an Oracle XA datasource pooled witht the standard JBoss implementation. From time to time, the database goes down which results in several timeout exceptions from org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106) which of course is to be expected. In most cases things get back to normal when the database gets back up. Good.
However, after this has happened a couple of times (we think it depends on the number of times connections are requested and unable to be served), we get this stacktrace:
java.lang.ArrayIndexOutOfBoundsException: 10
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:163)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:496)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:425)
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:318)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:477)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:814)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
at our.implementation.class.method(x)
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:683)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:185)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:72)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)
at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:267)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:128)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:118)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)
at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)
at org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java:331)
at org.jboss.ejb.Container.invoke(Container.java:700)
at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:101)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:90)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:46)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:45)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:100)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:85)
at $Proxy69.getDispatchRouteUpdates(Unknown Source)
at com.glt.paperdist.backend.scheduletasks.BTDispatchDeliveryPollerTask.perform(BTDispatchDeliveryPollerTask.java:30)
at org.jboss.varia.scheduler.Scheduler$Listener.handleNotification(Scheduler.java:1263)
at org.jboss.mx.server.NotificationListenerProxy.handleNotification(NotificationListenerProxy.java:69)
at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:95)
at javax.management.timer.Timer.sendNotifications(Timer.java:441)
at javax.management.timer.Timer.access$000(Timer.java:31)
at javax.management.timer.Timer$RegisteredNotification.doRun(Timer.java:612)
at org.jboss.mx.util.SchedulableRunnable.run(SchedulableRunnable.java:164)
at org.jboss.mx.util.ThreadPool$Worker.run(ThreadPool.java:225)
The pool has a max size of 10, so index 10 is of course invalid on InternalManagedConnectionPool on line 163:
cl = cls[currentPoolIndex];
I see no code in InternalManagedConnectionPool that can cause the index to exceed maxSize since there are only two assignments to currentPoolIndex and they are 1. syncrhonized and 2. cls[++currentPoolIndex] = cl; so that the out-of-bounds exception should have occured here. Very strange.
In any case, there is something wrong - the MBean (belongs to the wrapping JBossManagedConnectionPool) currently shows that there are -18 connections in use... I would guess that there is some sort synchronization error in there somewhere. The current HEAD shows that the implementation of InternalManagedConnectionPool has changed quite a bit since the 3.2.3 branch. For instance, it no longer uses an array for holding the ConnectionListeners. We do not have the opportunity to test HEAD, but if these changes fixes our problems, I would suggest that it should be backported to 3.2.3. :)
Cheers,
Dag Liodden
