4 Replies Latest reply on Sep 17, 2007 7:09 AM by timfox

    Too many open files


      I just upgraded my JBM installation from 1.0.1 SP4 to 1.2.0 SP1 this week, and ever since, the AS won't stay up for more than a 2 hour period, almost to the minute.

      The stack trace I'm seeing in server.log is:

      2007-04-19 17:31:36,764 ERROR [org.jboss.remoting.transport.bisocket.BisocketServerInvoker] Failed to accept socket connection
      80030 java.net.SocketException: Too many open files
      80031 at java.net.PlainSocketImpl.socketAccept(Native Method)
      80032 at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
      80033 at java.net.ServerSocket.implAccept(ServerSocket.java:450)
      80034 at java.net.ServerSocket.accept(ServerSocket.java:421)
      80035 at org.jboss.remoting.transport.bisocket.BisocketServerInvoker$SecondaryServerSocketThread.run(BisocketServerInvoker.java:745)

      Obviously rebooting the AS fixes it, but only for a 2 hour period.

      This is running on SUSE SLES 9.0 x86_64 Jboss 4.0.5_GA, I have tried it with the Sun JVM 1.6.0_01 and JRockit R27.2.0-jdk1.6.0, with the same result.

      Anywhere I can start looking?

        • 1. Re: Too many open files

          What does "netstat" and "lsof" display?

          You should be able to identify if there are leaking sockets.

          • 2. Re: Too many open files

            I have seen this in the past as well. The problem for us was not related to Jboss at that time. It occurred on a Linux Box, running the 2.4 Kernel.

            Our work around was:

            1) Become root

            2) Execute ulimit -n 8192
            This may also be any other large number. The total number of open file handles is very large for even normal machines.

            3) su username
            (Where this is the user that runs your jboss instance, and start the server).

            While I don't think this really fixed the problem; it allowed us to continue.

            • 3. Re: Too many open files

              Hi, thanks for the suggestions, I've had to roll back to 1.0.1 SP4 as this is a productions server, so I can't diagnose further at this point. Before I rolled back I tried lsof whilst I was getting the error and it was reporting between 5000-7000 handles, the rolled back system is currently reporting around 2000. I didn't get chance to try netstat.
              I'm not happy about increasing ulimit, I think it's asking for trouble isn't it?

              • 4. Re: Too many open files

                Are you sure you are running Remoting 2.2.2.GA as specified in the release announcement?

                If so, maybe you are not closing your connections?