Misleading out of memory error while loadtesting

[memoryerror]

While loadtesting with a mix of calls to our various APIs, the clients locked up and the log said out of memory. This surprised us, because we had never seen out of memory in the logs before, only in a command window--and not at all when running as a service. It turned out that what had actually happened was that the pool of database connections had been exhausted. Enlarging the database pool fixed the problem, but we were very lucky to have been able to diagnose the problem, particularly because the out of memory error was so misleading (and because we couldn't turn on logging, because that serialized requests, making the problem go away). We wasted several days trying to chase down a non-existent memory leak. We request that a more useful exception be thrown (something like Database Connection Pool Size Exhausted would have been a huge help) and/or that the pool size automatically grow in response to load. This is our first bug report, so we're not sure if this is how bugs are supposed to be reported. If not, please provide guidance for how to properly submit bug reports, and we will follow that procedure from now one (we tried to find instructions on the site, but didn't see anything about bug reporting). We searched the site for any mention of database connection pool and out of memory error, but didn't find anything. Are we the first to report this problem? We are using version 3.0.7. Is this problem already fixed in a later version?

[starksm64]

JBoss never throws OutOfMemoryErrors, the vm does to either low memory conditions or on some platforms too many threads. Since you have not shown the actual exception I can't say what your seeing. Describe the os, vm and show the original exception.

[memoryerror]

Windows 2000 Server with latest service packs.

Here is the top of jboss-server.log:

2004-04-12 15:05:45,656 WARN [org.jboss.jbossweb] WARNING: Exception for /dps.h/images/shared_shim.gif: java.net.SocketException: Connection reset by peer: socket write error
2004-04-12 15:39:32,890 ERROR [STDERR] java.lang.OutOfMemoryError
2004-04-12 15:46:11,218 ERROR [org.jboss.ejb.plugins.LogInterceptor] Unexpected Error:
java.lang.OutOfMemoryError
2004-04-12 15:46:11,296 ERROR [org.jboss.ejb.plugins.LogInterceptor] Unexpected Error:
java.lang.OutOfMemoryError
2004-04-12 15:46:11,312 WARN [org.jboss.jbossweb] WARNING: Stopping Acceptor [SSL: ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=7402]]
2004-04-12 15:46:11,625 ERROR [STDERR] java.lang.OutOfMemoryError
2004-04-12 15:46:11,921 WARN [org.jboss.jbossweb] WARNING:
java.lang.OutOfMemoryError

2004-04-12 15:46:13,109 WARN [org.jboss.resource.connectionmanager.JBossManagedConnectionPool] Destroying connection that could not be successfully matched: org.jboss.resource.adapter.jdbc.local.LocalManagedConnection@53d61c
2004-04-12 15:46:50,578 WARN [org.jboss.jbossweb] WARNING: OUT OF THREADS: SocketListener@0.0.0.0:7401
2004-04-12 15:47:57,875 ERROR [org.jboss.ejb.plugins.LogInterceptor] Unexpected Error:
java.lang.OutOfMemoryError
2004-04-12 15:48:17,250 ERROR [org.jboss.ejb.plugins.LogInterceptor] EJBException, causedBy:
org.jboss.util.NestedSQLException: No ManagedConnections Available!; - nested throwable: (javax.resource.ResourceException: No ManagedConnections Available!)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:106)
at com.xporta.util.db.DBHelper.getConnection(DBHelper.java:486)
at com.xporta.common.ejb.xporta.CountryGroupsDao.getCountryGroups(CountryGroupsDao.java:315)
at com.xporta.common.ejb.xporta.CountryHome.getCountry(CountryHome.java:174)
at com.xporta.common.ejb.xporta.XportaBean.getCountry(XportaBean.java:131)
at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:660)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:107)
at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:144)
at org.jboss.ejb.plugins.TxInterceptorBMT.invoke(TxInterceptorBMT.java:62)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:77)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:130)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:208)
at org.jboss.ejb.StatelessSessionContainer.invoke(StatelessSessionContainer.java:313)
at org.jboss.ejb.Container.invoke(Container.java:738)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:99)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:102)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:77)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:80)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:111)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:76)
at $Proxy70.getCountry(Unknown Source)
at com.xporta.landedcost.ejb.ShipmentImp.codeToCountry(ShipmentImp.java:999)
at com.xporta.landedcost.ejb.ShipmentImp.getLocation(ShipmentImp.java:1975)
at com.xporta.landedcost.ejb.ShipmentImp.setLocations(ShipmentImp.java:1173)
at com.xporta.landedcost.ejb.ShipmentImp.(ShipmentImp.java:855)
at com.xporta.landedcost.ejb.OrderImp.(OrderImp.java:143)
at com.xporta.landedcost.ejb.LandedCostCalc.getQuote(LandedCostCalc.java:696)
at com.xporta.landedcost.ejb.LandedCostBean.getQuote(LandedCostBean.java:222)
at com.xporta.landedcost.ejb.LandedCostWrapper.getQuote(LandedCostWrapper.java:50)
at sun.reflect.GeneratedMethodAccessor111.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java:660)
at org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:186)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:107)
at org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:144)
at org.jboss.ejb.plugins.TxInterceptorBMT.invoke(TxInterceptorBMT.java:62)
at org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java:77)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java:130)
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:208)
at org.jboss.ejb.StatelessSessionContainer.invoke(StatelessSessionContainer.java:313)
at org.jboss.ejb.Container.invoke(Container.java:738)
at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:517)
at org.jboss.invocation.local.LocalInvoker.invoke(LocalInvoker.java:99)
at org.jboss.invocation.InvokerInterceptor.invoke(InvokerInterceptor.java:102)
at org.jboss.proxy.TransactionInterceptor.invoke(TransactionInterceptor.java:77)
at org.jboss.proxy.SecurityInterceptor.invoke(SecurityInterceptor.java:80)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke(StatelessSessionInterceptor.java:111)
at org.jboss.proxy.ClientContainer.invoke(ClientContainer.java:76)
at $Proxy52.getQuote(Unknown Source)
at com.xporta.landedcost.ejb.LandedCostAdapter.getQuote(LandedCostAdapter.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:324)
at org.apache.soap.server.RPCRouter.invoke(RPCRouter.java:146)
at org.apache.soap.providers.RPCJavaProvider.invoke(RPCJavaProvider.java:144)
at org.apache.soap.server.http.RPCRouterServlet.doPost(RPCRouterServlet.java:431)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:760)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:360)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:294)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:558)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1714)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:507)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1664)
at org.mortbay.http.HttpServer.service(HttpServer.java:863)
at org.jboss.jetty.Jetty.service(Jetty.java:460)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:775)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:939)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:792)
at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:201)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:289)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:455)
Caused by: javax.resource.ResourceException: No ManagedConnections Available!
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:182)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:335)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:467)
at org.jboss.resource.connectionmanager.LocalTxConnectionManager.getManagedConnection(LocalTxConnectionManager.java:221)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:532)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:828)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
... 78 more
2004-04-12 15:48:29,046 WARN [org.jboss.jbossweb] WARNING: No thread for Socket[addr=/192.168.1.239,port=3653,localport=7401]

[memoryerror]

Sorry, forgot to include the VM. It's JDK 1.4.2.

[starksm64]

So you are running out of threads. Take thread dumps over time to see what threads are being leaked. There have been some fixes in the jca connection pooling including some leaks that could be affecting you here. jboss-3.2.3 has the most stable code in this regard.

[arifshaikh]

Check your server.log file, it must be huge, please empty log folder
turn of FILE logging from log4j.xml, because log4j appends to server.log,
file IO on huge file not fun for JVM.

I had same problem, but now its fine

[memoryerror]

We were overwhelming the garbage collector with objects being created from rows returned from the database. The database connections were a red herring. All sorts of things started failing once the garbage piled up. We changed the algorithm to read from an in-memory cache instead of from the database, and it works okay. We do have one remaining problem, but I'll post that separately.

[treespace]

This looks like a classic case of exhausted perm space.

Try setting -XX:MaxPermSpace=128m if you are loading a lot of classes. The default with -server is 64m. This is a known problem when there are large numbers of classes being loaded. WebLogic 8.1 SP2, for example, uses almost 40 megabytes of perm just to load. They increased their setting to 128 by default! JBoss is much lighter but you can still fill up perm space (class information for each loaded class).

Rod