-
1. Re: Too many open files on Linux- error, system hangs
juha Nov 14, 2002 4:05 AM (in response to madalin)You really don't provide enough detail to try to estimate where such resource leak would occur. Try to narrow down your application to a minimal case which shows the resource leakage.
-
2. Re: Too many open files on Linux- error, system hangs
jimmyjimjams Nov 19, 2002 7:05 PM (in response to madalin)I am having a similar issue.
Running :
Redhat 6.2 kernel 2.2.14
JBoss 3.0.4
Tomcat 4.1.12
Java SDK 1.3.1
We are getting the "Too many open files" error on our system with a very light load.
Using "lsof", I found that $JBOSS_HOME/lib/concurrent.jar was loaded 49 times (each with a separate PID).
In a similar manner, I found that an EJB was also loaded 49 times (each with a separate PID).
Just for grins, I randomly picked some of the PIDs that had concurrent.jar loaded and
found that each of the PIDs had 226 files open.
49*226=11074
With JBoss/Tomcat running, I found 12637 open files.
With JBoss/Tomcat stoped, I found 1563 open files
12637 - 1563 = 11074 (matches the estimated # of open files for JBoss/Tomcat).
We have made minimal changes to the standard JBoss/Tomcat config files, so this seems
like default behavior.
It does not take much activity to cause the system to fail with "Too many open files".
I am rather new to JBoss and Tomcat, so any suggestions for diagnosing the issue are appreciated.
Also, this seems like it should be a rather common issue.
Thanks for any help/info. -
3. Re: Too many open files on Linux- error, system hangs
mikefinn Nov 22, 2002 12:32 AM (in response to madalin)We too have seen this, not under heavy load, but under light load over a ~2 week period of time. I suspect it's time related - more specifically, redeploy-related. It seems as though (at least in our setup), undeployed apps do not release file descriptors. See below, where I started a JBoss server up (on which there are 5 deployed EARs), then force a redeploy by touching the EAR of one of the apps. Using lsof, I get an approximation of open FDs owned by the 'java' application (JBoss is the only Java stuff on this box).
<<< JBoss started >>>
$ bin/lsof_4.64 -c java | wc -l
6879
$ bin/lsof_4.64 -c java | wc -l
6879
$ bin/lsof_4.64 -c java | wc -l
6879
$ touch ~/jboss/server/cap/deploy/cap_sys.ear
$ bin/lsof_4.64 -c java | wc -l
7411
$ touch ~/jboss/server/cap/deploy/cap_sys.ear
$ bin/lsof_4.64 -c java | wc -l
7943
$ touch ~/jboss/server/cap/deploy/cap_sys.ear
$ bin/lsof_4.64 -c java | wc -l
8475
So 632 new file descriptors get opened everytime I redeploy this app (which is not much more than a re-written/tweaked jmx-console). Yikes.
JBoss 3.0.2 (Jetty)
RH 7.2
Sun 1.3.1 JDK
Hmmmmm..... -
4. Re: Too many open files on Linux- error, system hangs
davidm Nov 22, 2002 9:32 AM (in response to madalin)Count as in on this as well.
Running 3.0.4/Tomcat 4.1.12/Red Hat 7.2 with a very light load, a couple of EARs, no re deployment activity required to set it off.
Light usage for only minutes and system goes wild, lightweight process count goes through the roof, files count way up (assumed to be pipes).
Desperately hoping this becomes a priority.
David -
5. Re: Too many open files on Linux- error, system hangs
crazynala Dec 4, 2002 9:42 AM (in response to madalin)I am having the same problem -- running MySQL with JBoss 3.0.4 we are hitting hundreds of thousands of files open after a couple of hours. I turned off hot-deploy, suspecting that was causing issues. That didn't help. I understand that every Java thread needs its own file descriptors for all the deployment files, but what is strange is that the output from lsof seems to grow at a more or less constant rate. I have the line count plugged into MRTG and whenever we start JBoss the graph starts hiking up a mountain. When it hits 600k - 650k files open, it starts behaving very badly.
Has anyone come up with anything? Seems like a common experience for JBoss users.
Cheers,
Dan Norcross
KES
Beijing -
6. Re: Too many open files on Linux- error, system hangs
mikefinn Dec 8, 2002 8:37 AM (in response to madalin)Dan-
- You mean MRTG the network traffic tool?
- Holy cow: 600,000 files open? Have you analyzed with lsof or the like to see what the files are?
Seems like a few issues here:
1) In some implementations- in hot-redeploys, 'undeployed' resources' files are not getting released. A big issue in development or test. I see this on both Linux and Windows. I am going to put in a bug in SF for this one as soon as I confirm which versions in cvs have the problem.
2) High initial file handle count, due to each deployable unit having their own copies of JARs, other resources open. Probably (I suspect) since each deployable has its own classloader. This probably isn't really a 'problem', but an effect of the classloader design.
3) Growing file handle count regardless of hotdeploy.
Agreed? -
7. Re: Too many open files on Linux- error, system hangs
rpmorris Dec 10, 2002 10:16 AM (in response to madalin)Is there a bug open on this problem already? We're seeing this problem as well.
We're seeing a similar problem. I've noticed in our log that handles to deleted files are hanging around along time. -
8. Re: Too many open files on Linux- error, system hangs
dcartier Dec 14, 2002 5:46 AM (in response to madalin)Hello,
One of the offenders in this, are the /tmp/ddtbXXXXX.tmp files. Close to 100% of the java threads (processes) have every ddtb file open at all times. The files are all 0 bytes long, but the quantity of files seems to grow continually and quickly reaches the 5000+ open file range when multiplied by the number of threads.
I don't think this accounts for all the open files in JBoss under Linux, but certainly quite a few of them.
Dennis -
9. Re: Too many open files on Linux- error, system hangs
dcartier Dec 21, 2002 12:29 PM (in response to madalin)Can any of the developers please respond to this thread?
If we are barking up the wrong tree than please say that as well.
The silence is a bit disconcerting.
Dennis -
10. Re: Too many open files on Linux- error, system hangs
juhalindfors Dec 21, 2002 2:14 PM (in response to madalin)Please file it to SF bug tracker.
-
11. Re: Too many open files on Linux- error, system hangs
rpmorris Jan 8, 2003 3:34 PM (in response to madalin)I opened a bug on SF for this. #664635
-
12. Re: Too many open files on Linux- error, system hangs
starksm64 Jan 8, 2003 4:01 PM (in response to madalin)[ 664635 ] Too many open files causes JBoss 3.0 to crash
How is the fact that opening and closing files files still leaves the descriptor open a JBoss problem? We don't do anything to intercept filesystem calls so the issue seems VM related. If you have a sample program that runs in a standalone VM fine, but fails when deployed to JBoss using the same VM then submit that. -
13. Re: Too many open files on Linux- error, system hangs
fboss Jan 17, 2003 5:49 PM (in response to madalin)We had that problem and it turns out you have to make sure you close all the jdbc objects in your database access objects. That means you need to close preparedstatement, resultset, and ofcourse, the connection itself in the finally clause.
-
14. Re: Too many open files on Linux- error, system hangs
dcartier Jan 19, 2003 11:34 AM (in response to madalin)Yes, yes, yes! Thank you. I have been trying to figure out what was creating the /tmp/ddtbXXXXX files for ever now. I was closing my connections and thinking that the ResultSet and Statements would be closed automatically. Apparently not.
Now if I can just figure out why JBoss hates the IBM JVM so much, life would be good.
Dennis