OutOfMemoryErrors under load

[sgregg67]

I have a fairly large application running under JBOSS 3.0.6. The app may run for a few hours, or just a few minutes, but almost always will throw an OutOfMemoryError.

I have also tried JBOSS 3.2.1 and get exactly the same results.

The application exhibits the same behaviour under Compaq's Tru64 JVM as under linux with Sun's JDK 1.4.1 & 1.4.2, Blackdown's JDK 1.4.1 and IBM's 1.4 JVM.

The Linux box is a dual 2.8GHz machine with Red Hat Advanced server 2.1 with 2GB RAM.

I have tried file persisted JMS, Hypersonic persisted JMS and oracle persisted JMS - this does not appear to be the problem.

I have also increased the -Xmx to 1800m, but still no success, and I have played with -Xss up to 288k - again no success.

The application makes many calls to EJBs - if I put one call in a loop (using JMS to drive the calls), the call will succeed hundreds of thousands of times - the same goes for each of the EJB methods called, so I no longer suspect the EJB code for a memory leak.

On a separate JBOSS instance, the application also has a web front end (which also gets OutOfMemoryErrors) which when connections increase causes the problem to appear more frequently - decreasing the number of threads available for this does help a little, but this is only a bandaid for some real problem.

I have monitored the current thread count, open file handles, freeMemory, etc, and see none of these limits even getting close to the max (or 0 in thge case of free memory), yet I still see the OutOfMemory error.

Does anyone have any advise on how to further tackle this problem?

Thanks

Stephen.

[jonlee]

Employ jmemprof or Jprobe to profile your memory use. You might be able to use optimizeit to a more limited degree. See the performance thread on memory study. You can also do the simple thing like verbose:gc to get an idea of a general area of memory use. Is it possible that you are getting a JDBC related memory leak or breakage?

[akirdat]

I have seen this problem before. This is what is going on: The number of threads a JVM can create goes down as you increase the stack size (-mx1800m). The way I solved my problem was to balance the number of threads I need for my app and the memory needed to run my application.
Also IBM's JVM handles these issues very nicely on Linux.

[jonlee]

Agreed. Some of the issues that people raise here I have never experienced on a production Linux boxen but we exclusively use the IBM JDK.

[sgregg67]

Thanks for the input - I have changed to using the IBM JDK1.4, and have put our code through the jmemprof profiler - this has pointed out some weaknesses in our code, and potentially in the oracle OCI drivers - that's tomorrows task.

Thanks Again for the input - I think I'll solve this using IBM and fixing our code,

Stephen.

[colinws]

Oracle OCI drivers.

Memory leaks in native code, not visible in the normal information available from the JVM.

First try the thin drivers. This cured similar problems for us.

[akirdat]

I haven't noticed any problems with the OCI driver. We have an app running in production for over a year now with no problems related to Oracle's OCI driver.

[ofels]

It might be related to caching issues.
Entity beans are created and then cached for a specific amount of time. The standardjboss.xml holds a threshold entry which specifies how long entities are cached by the container, if you lower this value it will definitely decrease memory usage as beans are deactivated earlier.
We had such an issue as we are importing and distributing lots of measurement data where the beans are created once but not needed any more afterwards.
Increasing the threshold also increases the overall memory level while decreasing it lowers memory consumption. This also increases performance btw.

Oliver

[rabbuhl]

We had the same problem and it was caused by memory leaks in the OCI driver. Switching to the thin driver solved the problem.

[borus]

Never use OCI drivers, use only thin driver or 3rd party drivers. This solved problems in many projects I was involved in.