14 Replies Latest reply on Jan 19, 2003 11:34 AM by dcartier

    Too many open files on Linux- error, system hangs

    madalin

      Hi,
      I am using Jboss 3.0.4 with Tomcat 4.1.12 on a Red Hat Linux 6.2 (kernel 2.4.7). The max # of files /proc/sys/fs/file-max is set to 17000. In several days os playing with the server (not heavy load), this limit is reached and the system is going crazy. If I stop the jboss server, almost all the file handlers are dealocatted. It looks like the server is opening some files and forget to close them...It also seems that the number of processes are increased.... Do you have an idea what may cause the problem? I am using a MySQl database.

      Best Regards.
      Madalin

        • 1. Re: Too many open files on Linux- error, system hangs

          You really don't provide enough detail to try to estimate where such resource leak would occur. Try to narrow down your application to a minimal case which shows the resource leakage.

          • 2. Re: Too many open files on Linux- error, system hangs
            jimmyjimjams

            I am having a similar issue.

            Running :
            Redhat 6.2 kernel 2.2.14
            JBoss 3.0.4
            Tomcat 4.1.12
            Java SDK 1.3.1

            We are getting the "Too many open files" error on our system with a very light load.

            Using "lsof", I found that $JBOSS_HOME/lib/concurrent.jar was loaded 49 times (each with a separate PID).
            In a similar manner, I found that an EJB was also loaded 49 times (each with a separate PID).

            Just for grins, I randomly picked some of the PIDs that had concurrent.jar loaded and
            found that each of the PIDs had 226 files open.

            49*226=11074

            With JBoss/Tomcat running, I found 12637 open files.

            With JBoss/Tomcat stoped, I found 1563 open files

            12637 - 1563 = 11074 (matches the estimated # of open files for JBoss/Tomcat).

            We have made minimal changes to the standard JBoss/Tomcat config files, so this seems
            like default behavior.

            It does not take much activity to cause the system to fail with "Too many open files".

            I am rather new to JBoss and Tomcat, so any suggestions for diagnosing the issue are appreciated.

            Also, this seems like it should be a rather common issue.



            Thanks for any help/info.

            • 3. Re: Too many open files on Linux- error, system hangs
              mikefinn

              We too have seen this, not under heavy load, but under light load over a ~2 week period of time. I suspect it's time related - more specifically, redeploy-related. It seems as though (at least in our setup), undeployed apps do not release file descriptors. See below, where I started a JBoss server up (on which there are 5 deployed EARs), then force a redeploy by touching the EAR of one of the apps. Using lsof, I get an approximation of open FDs owned by the 'java' application (JBoss is the only Java stuff on this box).

              <<< JBoss started >>>
              $ bin/lsof_4.64 -c java | wc -l
              6879
              $ bin/lsof_4.64 -c java | wc -l
              6879
              $ bin/lsof_4.64 -c java | wc -l
              6879
              $ touch ~/jboss/server/cap/deploy/cap_sys.ear
              $ bin/lsof_4.64 -c java | wc -l
              7411
              $ touch ~/jboss/server/cap/deploy/cap_sys.ear
              $ bin/lsof_4.64 -c java | wc -l
              7943
              $ touch ~/jboss/server/cap/deploy/cap_sys.ear
              $ bin/lsof_4.64 -c java | wc -l
              8475

              So 632 new file descriptors get opened everytime I redeploy this app (which is not much more than a re-written/tweaked jmx-console). Yikes.

              JBoss 3.0.2 (Jetty)
              RH 7.2
              Sun 1.3.1 JDK

              Hmmmmm.....

              • 4. Re: Too many open files on Linux- error, system hangs
                davidm

                Count as in on this as well.
                Running 3.0.4/Tomcat 4.1.12/Red Hat 7.2 with a very light load, a couple of EARs, no re deployment activity required to set it off.

                Light usage for only minutes and system goes wild, lightweight process count goes through the roof, files count way up (assumed to be pipes).

                Desperately hoping this becomes a priority.

                David

                • 5. Re: Too many open files on Linux- error, system hangs
                  crazynala

                  I am having the same problem -- running MySQL with JBoss 3.0.4 we are hitting hundreds of thousands of files open after a couple of hours. I turned off hot-deploy, suspecting that was causing issues. That didn't help. I understand that every Java thread needs its own file descriptors for all the deployment files, but what is strange is that the output from lsof seems to grow at a more or less constant rate. I have the line count plugged into MRTG and whenever we start JBoss the graph starts hiking up a mountain. When it hits 600k - 650k files open, it starts behaving very badly.

                  Has anyone come up with anything? Seems like a common experience for JBoss users.

                  Cheers,
                  Dan Norcross
                  KES
                  Beijing

                  • 6. Re: Too many open files on Linux- error, system hangs
                    mikefinn

                    Dan-
                    - You mean MRTG the network traffic tool?
                    - Holy cow: 600,000 files open? Have you analyzed with lsof or the like to see what the files are?


                    Seems like a few issues here:

                    1) In some implementations- in hot-redeploys, 'undeployed' resources' files are not getting released. A big issue in development or test. I see this on both Linux and Windows. I am going to put in a bug in SF for this one as soon as I confirm which versions in cvs have the problem.

                    2) High initial file handle count, due to each deployable unit having their own copies of JARs, other resources open. Probably (I suspect) since each deployable has its own classloader. This probably isn't really a 'problem', but an effect of the classloader design.

                    3) Growing file handle count regardless of hotdeploy.

                    Agreed?

                    • 7. Re: Too many open files on Linux- error, system hangs
                      rpmorris

                      Is there a bug open on this problem already? We're seeing this problem as well.

                      We're seeing a similar problem. I've noticed in our log that handles to deleted files are hanging around along time.

                      • 8. Re: Too many open files on Linux- error, system hangs
                        dcartier

                        Hello,

                        One of the offenders in this, are the /tmp/ddtbXXXXX.tmp files. Close to 100% of the java threads (processes) have every ddtb file open at all times. The files are all 0 bytes long, but the quantity of files seems to grow continually and quickly reaches the 5000+ open file range when multiplied by the number of threads.

                        I don't think this accounts for all the open files in JBoss under Linux, but certainly quite a few of them.

                        Dennis

                        • 9. Re: Too many open files on Linux- error, system hangs
                          dcartier

                          Can any of the developers please respond to this thread?

                          If we are barking up the wrong tree than please say that as well.

                          The silence is a bit disconcerting.

                          Dennis

                          • 10. Re: Too many open files on Linux- error, system hangs

                            Please file it to SF bug tracker.

                            • 11. Re: Too many open files on Linux- error, system hangs
                              rpmorris

                              I opened a bug on SF for this. #664635

                              • 12. Re: Too many open files on Linux- error, system hangs
                                starksm64

                                [ 664635 ] Too many open files causes JBoss 3.0 to crash

                                How is the fact that opening and closing files files still leaves the descriptor open a JBoss problem? We don't do anything to intercept filesystem calls so the issue seems VM related. If you have a sample program that runs in a standalone VM fine, but fails when deployed to JBoss using the same VM then submit that.

                                • 13. Re: Too many open files on Linux- error, system hangs
                                  fboss

                                  We had that problem and it turns out you have to make sure you close all the jdbc objects in your database access objects. That means you need to close preparedstatement, resultset, and ofcourse, the connection itself in the finally clause.

                                  • 14. Re: Too many open files on Linux- error, system hangs
                                    dcartier

                                    Yes, yes, yes! Thank you. I have been trying to figure out what was creating the /tmp/ddtbXXXXX files for ever now. I was closing my connections and thinking that the ResultSet and Statements would be closed automatically. Apparently not.

                                    Now if I can just figure out why JBoss hates the IBM JVM so much, life would be good.

                                    Dennis