6 Replies Latest reply on Oct 30, 2012 3:18 AM by troubleshooter

    Remote Jndi Datasource: ResultSet gets randomly closed

    troubleshooter

      Hello experts,

       

      A short description of our deployment:

      We have a central reporting application that reads database divided (for scalability) among a number of machines. Each of these machines have one database and one JBoss (4.2.3.GA) that houses the datasource object. The reporting app fetches data by looking up one of these remote JNDI trees and obtaining datasource object. We do not run any distributed transactions therefore we're ok with this antipattern note. This model has worked out beautifully so far and integrates seamlessly with our existing model.

       

      The problem:

      In the event of multiple requests, the resultsets tend to close halfway and the underlying statement throws a SQLExcpetion. Here's a trace:

       

      java.sql.SQLException: The result set is closed.
              at org.jboss.resource.adapter.jdbc.WrappedResultSet.checkState(WrappedResultSet.java:1959)
              at org.jboss.resource.adapter.jdbc.WrappedResultSet.getString(WrappedResultSet.java:885)
              at sun.reflect.GeneratedMethodAccessor334.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.doResultSetMethod(WrapperDataSourceService.java:431)
              at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.invoke(WrapperDataSourceService.java:233)
              at sun.reflect.GeneratedMethodAccessor326.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
              at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
              at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
              at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
              at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
              at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:818)
              at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:419)
              at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
              at sun.rmi.transport.Transport$1.run(Transport.java:177)
              at sun.rmi.transport.Transport$1.run(Transport.java:174)
              at java.security.AccessController.doPrivileged(Native Method)
              at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
              at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
              at java.lang.Thread.run(Thread.java:722)
              at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:273)
              at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:251)
              at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:160)
              at org.jboss.invocation.jrmp.server.JRMPInvoker_Stub.invoke(Unknown Source)
              at org.jboss.invocation.jrmp.interfaces.JRMPInvokerProxy.invoke(JRMPInvokerProxy.java:133)
              at org.jboss.invocation.InvokerInterceptor.invokeInvoker(InvokerInterceptor.java:365)
              at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:197)
              at org.jboss.proxy.ClientMethodInterceptor.invoke(ClientMethodInterceptor.java:74)
              at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:100)
              at $Proxy78.getString(Unknown Source)
              at com.aryaka.nos.stats.feeder.ChartMapperDb.getChart(ChartMapperDb.java:73)
              at com.aryaka.nos.stats.StatsAPIImpl.getStats(StatsAPIImpl.java:171)
              at com.aryaka.nos.stats.StatsAPIImpl.getLinkTimeseriesStats(StatsAPIImpl.java:724)
              at com.aryaka.nos.stats.StatsAPIImpl.getAppTimeseriesStats(StatsAPIImpl.java:633)
              at sun.reflect.GeneratedMethodAccessor276.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
              at sun.rmi.transport.Transport$1.run(Transport.java:177)
              at sun.rmi.transport.Transport$1.run(Transport.java:174)
              at java.security.AccessController.doPrivileged(Native Method)
              at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
              at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
              at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
              at java.lang.Thread.run(Thread.java:722)
      

       

      This particular exception got thrown in this code:

       

      if(rs.next()) {
                ArrayList<Formula> arrFormula = new ArrayList<Formula>();
                ArrayList<ChartConstants> arrConst = new ArrayList<ChartConstants>();
                ChartConstants graphConstant = null;
                do{
                     //do stuff
                  }while(rs.next()); //<<OFFENDING LINE
      }
      

       

      The code is the same production code that runs just fine on local datasources. Statements/Connection closures are handled in finally blocks. There are no shared connections/statements. Each request pulls a new connection from pool. All standard stuff.

       

      I'm clueless as to what might be causing resultsets to close randomly. Any help is appreciated!

       

      -Thanks

        • 1. Re: Remote Jndi Datasource: ResultSet gets randomly closed
          wdfink

          Do you mean you lookup a DS deployed at a remote JBoss (and here the DS is a local DB)?

           

          I don't understand the reason for that, as if you spread the requests across multiple JBoss instances why you spend another (remote) access to a DS?

          • 2. Re: Remote Jndi Datasource: ResultSet gets randomly closed
            troubleshooter

            Thanks for replying @Wolf-Dieter Fink.

             

            Yes, the database is local to JBoss instance. But these JBosses serve the central app and not web requests. For web requests there's a different JBoss. This way we can issue identical deployment configurations to all these machines (eg: same database privileges across instances). Moreover, the application that feeds the database sits locally on that JBoss. So in a way JBoss becomes an interface to that portion of the database.

             

            I doubt this could be the cause though.

             

            And yes, I'd be grateful if you could suggest a better cleaner way ...

            • 3. Re: Remote Jndi Datasource: ResultSet gets randomly closed
              wdfink

              I'm a bit puzzled. This let me think you have lookup the DS remote

              The code is the same production code that runs just fine on local datasources.

               

              So what is your scenario? A JBoss as web frontend and a couple of JBoss instances as EJB worker each with one local db?

              • 4. Re: Remote Jndi Datasource: ResultSet gets randomly closed
                troubleshooter

                Our current model has a monoithic database today and there's a single JBoss that houses the frontend as well as the datasource to connect to that database.

                 

                The new model is to divide them up over several machines and let them expose datasources so the central appication can remotely connect to those datasources. There are no EJBs. There is no hibernate.

                • 5. Re: Remote Jndi Datasource: ResultSet gets randomly closed
                  wdfink

                  Manu,

                  sorry but it is still not clear to me.

                   

                  Is there still one central JBoss which ask other JBoss instances to provide a DS remote?

                  Or do you split the 'central JBoss' into many and for each a separate DS?

                  • 6. Re: Remote Jndi Datasource: ResultSet gets randomly closed
                    troubleshooter

                    Sorry about that @Wolf-Dieter. Let me try it again.

                     

                    Our current model:

                    1. JBoss 1: Runs application (A1) that fetches data from database (DB) using local datasource (LDS).

                    2. JBoss 2: Runs application (A2) that feeds data into DB using local datasource (LDS2).

                     

                    This works perfectly ok. No problems what so ever (that code snippet also works fine here).

                     

                    Now (for scalability reasons) we're migrating to a new model. In this model DB is divided into several DBs (say DB1, DB2, DB3 ... DBn):

                    1. JBoss 1: Runs application A1 that fetches data from these DBs using remote datasources (RDS1, RDS2, RDS3 ... RDSn).

                    2. JBoss N: Runs application A2 that feeds data to DBn through LDS2, and hosts/exposes RDSn.

                     

                    Therefore, it follows that for RDSn to be available to A1, JBossN has to be running, else the lookup fails. Note also that each JBoss sits in its own machine. That summarizes the scheme we're after. Hope I'm clearer this time.

                     

                    The new scheme works fine under a normal load. But under stress, the resultsets abruptly close in the middle of a fetch operation (eg: in the code snippet above). There's another more serious issue I ran into yesterday. Connections spawned using remote datasource don't properly close and after a while they choke the machine with 100% CPU. Here is a thread dump:

                     

                    "RMI TCP Connection(2334)-121.10.1.14" daemon prio=10 tid=0x00002aaad0009800 nid=0x570d runnable [0x000000004c394000]
                       java.lang.Thread.State: RUNNABLE
                            at java.util.HashMap.transfer(HashMap.java:499)
                            at java.util.HashMap.resize(HashMap.java:478)
                            at java.util.HashMap.addEntry(HashMap.java:768)
                            at java.util.HashMap.put(HashMap.java:402)
                            at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.createStatementProxy(WrapperDataSourceService.java:522)
                            at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.doConnectionMethod(WrapperDataSourceService.java:366)
                            at org.jboss.resource.adapter.jdbc.remote.WrapperDataSourceService.invoke(WrapperDataSourceService.java:211)
                            at sun.reflect.GeneratedMethodAccessor76.invoke(Unknown Source)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                            at java.lang.reflect.Method.invoke(Method.java:601)
                            at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
                            at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
                            at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
                            at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
                            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
                            at org.jboss.invocation.jrmp.server.JRMPInvoker$MBeanServerAction.invoke(JRMPInvoker.java:818)
                            at org.jboss.invocation.jrmp.server.JRMPInvoker.invoke(JRMPInvoker.java:419)
                            at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
                            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                            at java.lang.reflect.Method.invoke(Method.java:601)
                            at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
                            at sun.rmi.transport.Transport$1.run(Transport.java:177)
                            at sun.rmi.transport.Transport$1.run(Transport.java:174)
                            at java.security.AccessController.doPrivileged(Native Method)
                            at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
                            at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
                            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
                            at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
                            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
                            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
                            at java.lang.Thread.run(Thread.java:722)
                    

                     

                    This continued even after manually killing the connections from MySQL. Goes only after stopping JBoss. Please note that this is JBoss alone and none of our applications have been deployed on it.

                     

                    Thanks for taking time to read. Any help is greatly appreciated!