7 Replies Latest reply on May 26, 2004 4:35 PM by treespace

    Misleading out of memory error while loadtesting

    memoryerror Newbie

      While loadtesting with a mix of calls to our various APIs, the clients locked up and the log said out of memory. This surprised us, because we had never seen out of memory in the logs before, only in a command window--and not at all when running as a service. It turned out that what had actually happened was that the pool of database connections had been exhausted. Enlarging the database pool fixed the problem, but we were very lucky to have been able to diagnose the problem, particularly because the out of memory error was so misleading (and because we couldn't turn on logging, because that serialized requests, making the problem go away). We wasted several days trying to chase down a non-existent memory leak. We request that a more useful exception be thrown (something like Database Connection Pool Size Exhausted would have been a huge help) and/or that the pool size automatically grow in response to load. This is our first bug report, so we're not sure if this is how bugs are supposed to be reported. If not, please provide guidance for how to properly submit bug reports, and we will follow that procedure from now one (we tried to find instructions on the site, but didn't see anything about bug reporting). We searched the site for any mention of database connection pool and out of memory error, but didn't find anything. Are we the first to report this problem? We are using version 3.0.7. Is this problem already fixed in a later version?

        • 1. Re: Misleading out of memory error while loadtesting
          Scott Stark Master

          JBoss never throws OutOfMemoryErrors, the vm does to either low memory conditions or on some platforms too many threads. Since you have not shown the actual exception I can't say what your seeing. Describe the os, vm and show the original exception.

          • 2. Re: Misleading out of memory error while loadtesting
            memoryerror Newbie

            Windows 2000 Server with latest service packs.

            Here is the top of jboss-server.log:

            2004-04-12 15:05:45,656 WARN [org.jboss.jbossweb] WARNING: Exception for /dps.h/images/shared_shim.gif: java.net.SocketException: Connection reset by peer: socket write error
            2004-04-12 15:39:32,890 ERROR [STDERR] java.lang.OutOfMemoryError
            2004-04-12 15:46:11,218 ERROR [org.jboss.ejb.plugins.LogInterceptor] Unexpected Error:
            java.lang.OutOfMemoryError
            2004-04-12 15:46:11,296 ERROR [org.jboss.ejb.plugins.LogInterceptor] Unexpected Error:
            java.lang.OutOfMemoryError
            2004-04-12 15:46:11,312 WARN [org.jboss.jbossweb] WARNING: Stopping Acceptor [SSL: ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=7402]]
            2004-04-12 15:46:11,625 ERROR [STDERR] java.lang.OutOfMemoryError
            2004-04-12 15:46:11,921 WARN [org.jboss.jbossweb] WARNING:
            java.lang.OutOfMemoryError

            2004-04-12 15:46:13,109 WARN [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] Destroying connection that could not be successfully matched: org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@53d61c
            2004-04-12 15:46:50,578 WARN [org.jboss.jbossweb] WARNING: OUT OF THREADS: SocketListener@0.0.0.0:7401
            2004-04-12 15:47:57,875 ERROR [org.jboss.ejb.plugins.LogInterceptor] Unexpected Error:
            java.lang.OutOfMemoryError
            2004-04-12 15:48:17,250 ERROR [org.jboss.ejb.plugins.LogInterceptor] EJBException, causedBy:
            org.jboss.util.NestedSQLException: No ManagedConnections Available!; - nested throwable: (javax.resource.ResourceException: No ManagedConnections Available!)
            at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
            at com.xporta.util.db.DBHelper.getConnection(DBHelper.java:486)
            at com.xporta.common.ejb.xporta.CountryGroupsDao.getCountryGroups(CountryGroupsDao.java:315)
            at com.xporta.common.ejb.xporta.CountryHome.getCountry(CountryHome.java:174)
            at com.xporta.common.ejb.xporta.XportaBean.getCountry(XportaBean.java:131)
            at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:660)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:107)
            at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:144)
            at org.jboss.ejb.plugins.TxInterceptorBMT.invoke(TxInterceptorBMT.java:62)
            at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:77)
            at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:130)
            at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:208)
            at org.jboss.ejb.StatelessSessionContainer.invoke(StatelessSessionContainer.java:313)
            at org.jboss.ejb.Container.invoke(Container.java:738)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:99)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:102)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:77)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:80)
            at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:111)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:76)
            at $Proxy70.getCountry(Unknown Source)
            at com.xporta.landedcost.ejb.ShipmentImp.codeToCountry(ShipmentImp.java:999)
            at com.xporta.landedcost.ejb.ShipmentImp.getLocation(ShipmentImp.java:1975)
            at com.xporta.landedcost.ejb.ShipmentImp.setLocations(ShipmentImp.java:1173)
            at com.xporta.landedcost.ejb.ShipmentImp.(ShipmentImp.java:855)
            at com.xporta.landedcost.ejb.OrderImp.(OrderImp.java:143)
            at com.xporta.landedcost.ejb.LandedCostCalc.getQuote(LandedCostCalc.java:696)
            at com.xporta.landedcost.ejb.LandedCostBean.getQuote(LandedCostBean.java:222)
            at com.xporta.landedcost.ejb.LandedCostWrapper.getQuote(LandedCostWrapper.java:50)
            at sun.reflect.GeneratedMethodAccessor111.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:660)
            at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
            at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:107)
            at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:144)
            at org.jboss.ejb.plugins.TxInterceptorBMT.invoke(TxInterceptorBMT.java:62)
            at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:77)
            at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:130)
            at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:208)
            at org.jboss.ejb.StatelessSessionContainer.invoke(StatelessSessionContainer.java:313)
            at org.jboss.ejb.Container.invoke(Container.java:738)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
            at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:99)
            at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:102)
            at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:77)
            at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:80)
            at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:111)
            at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:76)
            at $Proxy52.getQuote(Unknown Source)
            at com.xporta.landedcost.ejb.LandedCostAdapter.getQuote(LandedCostAdapter.java:104)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.apache.soap.server.RPCRouter.invoke(RPCRouter.java:146)
            at org.apache.soap.providers.RPCJavaProvider.invoke(RPCJavaProvider.java:144)
            at org.apache.soap.server.http.RPCRouterServlet.doPost(RPCRouterServlet.java:431)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
            at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360)
            at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
            at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
            at org.mortbay.http.HttpContext.handle(HttpContext.java:1714)
            at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507)
            at org.mortbay.http.HttpContext.handle(HttpContext.java:1664)
            at org.mortbay.http.HttpServer.service(HttpServer.java:863)
            at org.jboss.jetty.Jetty.service(Jetty.java:460)
            at org.mortbay.http.HttpConnection.service(HttpConnection.java:775)
            at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939)
            at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792)
            at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201)
            at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
            at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455)
            Caused by: javax.resource.ResourceException: No ManagedConnections Available!
            at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:182)
            at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:335)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:467)
            at org.jboss.resource.connectionmanager.LocalTxConnectionManager.getManagedConnection(LocalTxConnectionManager.java:221)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:532)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:828)
            at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
            ... 78 more
            2004-04-12 15:48:29,046 WARN [org.jboss.jbossweb] WARNING: No thread for Socket[addr=/192.168.1.239,port=3653,localport=7401]

            • 3. Re: Misleading out of memory error while loadtesting
              memoryerror Newbie

              Sorry, forgot to include the VM. It's JDK 1.4.2.

              • 4. Re: Misleading out of memory error while loadtesting
                Scott Stark Master

                So you are running out of threads. Take thread dumps over time to see what threads are being leaked. There have been some fixes in the jca connection pooling including some leaks that could be affecting you here. jboss-3.2.3 has the most stable code in this regard.

                • 5. Re: Misleading out of memory error while loadtesting
                  arifshaikh Newbie

                  Check your server.log file, it must be huge, please empty log folder
                  turn of FILE logging from log4j.xml, because log4j appends to server.log,
                  file IO on huge file not fun for JVM.

                  I had same problem, but now its fine

                  • 6. Re: Misleading out of memory error while loadtesting
                    memoryerror Newbie

                    We were overwhelming the garbage collector with objects being created from rows returned from the database. The database connections were a red herring. All sorts of things started failing once the garbage piled up. We changed the algorithm to read from an in-memory cache instead of from the database, and it works okay. We do have one remaining problem, but I'll post that separately.

                    • 7. Re: Misleading out of memory error while loadtesting
                      treespace Novice

                      This looks like a classic case of exhausted perm space.

                      Try setting -XX:MaxPermSpace=128m if you are loading a lot of classes. The default with -server is 64m. This is a known problem when there are large numbers of classes being loaded. WebLogic 8.1 SP2, for example, uses almost 40 megabytes of perm just to load. They increased their setting to 128 by default! JBoss is much lighter but you can still fill up perm space (class information for each loaded class).

                      Rod