Remote Jndi Datasource: ResultSet gets randomly closed
Hello experts,
A short description of our deployment:
We have a central reporting application that reads database divided (for scalability) among a number of machines. Each of these machines have one database and one JBoss (4.2.3.GA) that houses the datasource object. The reporting app fetches data by looking up one of these remote JNDI trees and obtaining datasource object. We do not run any distributed transactions therefore we're ok with this antipattern note. This model has worked out beautifully so far and integrates seamlessly with our existing model.
The problem:
In the event of multiple requests, the resultsets tend to close halfway and the underlying statement throws a SQLExcpetion. Here's a trace:
java.sql.SQLException: The result set is closed. at org.jboss.resource.adapter.jdbc.WrappedResultSet.checkState(WrappedResultSet.java:1959) at org.jboss.resource.adapter.jdbc.WrappedResultSet.getString(WrappedResultSet.java:885) at sun.reflect.GeneratedMethodAccessor334.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.doResultSetMethod(WrapperDataSourceService.java:431) at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.invoke(WrapperDataSourceService.java:233) at sun.reflect.GeneratedMethodAccessor326.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155) at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94) at org.jboss.mx.server.Invocation.invoke(Invocation.java:86) at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264) at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659) at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:818) at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:419) at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160) at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source) at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133) at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365) at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197) at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74) at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100) at $Proxy78.getString(Unknown Source) at com.aryaka.nos.stats.feeder.ChartMapperDb.getChart(ChartMapperDb.java:73) at com.aryaka.nos.stats.StatsAPIImpl.getStats(StatsAPIImpl.java:171) at com.aryaka.nos.stats.StatsAPIImpl.getLinkTimeseriesStats(StatsAPIImpl.java:724) at com.aryaka.nos.stats.StatsAPIImpl.getAppTimeseriesStats(StatsAPIImpl.java:633) at sun.reflect.GeneratedMethodAccessor276.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322) at sun.rmi.transport.Transport$1.run(Transport.java:177) at sun.rmi.transport.Transport$1.run(Transport.java:174) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:173) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)
This particular exception got thrown in this code:
if(rs.next()) {
ArrayList<Formula> arrFormula = new ArrayList<Formula>();
ArrayList<ChartConstants> arrConst = new ArrayList<ChartConstants>();
ChartConstants graphConstant = null;
do{
//do stuff
}while(rs.next()); //<<OFFENDING LINE
}
The code is the same production code that runs just fine on local datasources. Statements/Connection closures are handled in finally blocks. There are no shared connections/statements. Each request pulls a new connection from pool. All standard stuff.
I'm clueless as to what might be causing resultsets to close randomly. Any help is appreciated!
-Thanks
