3 Replies Latest reply on Mar 7, 2012 5:56 PM by tc7

    Failed to load users/passwords/role files - Too many open files

    tc7
    tc7 Mar 6, 2012 7:23 PM

      We've had JBoss-7.1.0.Final (Thunder) running in our testing environment for a few nights now.

      Each night we get the following error:

       


      		20120307 07:11:34,447 ERROR [org.jboss.as.security.RealmUsersRolesLoginModule] (Old I/O server worker (parentId: 976087715, [id: 0x3a2deaa3, /160.222.152.133:5445])) Failed to load users/passwords/role files: java.io.IOException

       

       

      Followed by:

       


      		20120307 07:11:34,556 WARNING [org.jboss.netty.channel.socket.oio.OioServerSocketPipelineSink] (Old I/O server boss ([id: 0x3a2deaa3, /160.222.152.133:5445])) Failed to accept a connection.: java.net.SocketException: Too many open files

      		at java.net.PlainSocketImpl.socketAccept(Native Method) [rt.jar:1.7.0_01]

      		at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:396) [rt.jar:1.7.0_01]

      		at java.net.ServerSocket.implAccept(ServerSocket.java:522) [rt.jar:1.7.0_01]

      		at java.net.ServerSocket.accept(ServerSocket.java:490) [rt.jar:1.7.0_01]

      		at org.jboss.netty.channel.socket.oio.OioServerSocketPipelineSink$Boss.run(OioServerSocketPipelineSink.java:202) [netty-3.2.6.Final.jar:]

      		at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181) [netty-3.2.6.Final.jar:]

      		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_01]

      		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_01]

      		at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_01]

       

      Has anyone else experienced this?

      We've been running current environment for a couple of years - recently moving to JBoss 7.0.2, then 7.1.0.CRb1, and recently 7.1.0.Final. The issue has only occurred with 7.1.0.Final. I'm unsure yet whether this (file limit) is the cause, or simply a symptom.

       

      increasing per user file limits on RHEL

      This can be increased on a per user bais by editing /etc/security/limits.conf

      We're running on Redhat RHEL 5.4 (2.6.18-164.el5) x64, JDK 1.7.0_01

      I checked the max open files using:

       

      cat /proc/sys/fs/file-max

      363094

       

      but then there are user limits too which can be queried (from the current user):

      su - <username>

      $ ulimit -Hn <-- hard limit

      $ ulimit -Sn <-- soft limit

       

      Which both return 1024 (the user default I imagine).

      This can be increased on a per user bais by editing /etc/security/limits.conf

       

      I've set the following limits for the user running JBoss process:

      <username> soft nofile 4096
      <username> hard nofile 10240

       

      The following article explains in detail:

      http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

       

       

      Action

      Having made the above OS configuration change (and crossed my fingers) I'll report back soon!