Failed to load users/passwords/role files - Too many open files
tc7 Mar 6, 2012 7:23 PMWe've had JBoss-7.1.0.Final (Thunder) running in our testing environment for a few nights now.
Each night we get the following error:
20120307 07:11:34,447 ERROR [org.jboss.as.security.RealmUsersRolesLoginModule] (Old I/O server worker (parentId: 976087715, [id: 0x3a2deaa3, /160.222.152.133:5445])) Failed to load users/passwords/role files: java.io.IOException |
Followed by:
20120307 07:11:34,556 WARNING [org.jboss.netty.channel.socket.oio.OioServerSocketPipelineSink] (Old I/O server boss ([id: 0x3a2deaa3, /160.222.152.133:5445])) Failed to accept a connection.: java.net.SocketException: Too many open files | |
at java.net.PlainSocketImpl.socketAccept(Native Method) [rt.jar:1.7.0_01] | |
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:396) [rt.jar:1.7.0_01] | |
at java.net.ServerSocket.implAccept(ServerSocket.java:522) [rt.jar:1.7.0_01] | |
at java.net.ServerSocket.accept(ServerSocket.java:490) [rt.jar:1.7.0_01] | |
at org.jboss.netty.channel.socket.oio.OioServerSocketPipelineSink$Boss.run(OioServerSocketPipelineSink.java:202) [netty-3.2.6.Final.jar:] | |
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:181) [netty-3.2.6.Final.jar:] | |
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_01] | |
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_01] | |
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_01] |
Has anyone else experienced this?
We've been running current environment for a couple of years - recently moving to JBoss 7.0.2, then 7.1.0.CRb1, and recently 7.1.0.Final. The issue has only occurred with 7.1.0.Final. I'm unsure yet whether this (file limit) is the cause, or simply a symptom.
increasing per user file limits on RHEL
This can be increased on a per user bais by editing /etc/security/limits.conf
We're running on Redhat RHEL 5.4 (2.6.18-164.el5) x64, JDK 1.7.0_01
I checked the max open files using:
cat /proc/sys/fs/file-max
363094
but then there are user limits too which can be queried (from the current user):
su - <username>
$ ulimit -Hn <-- hard limit
$ ulimit -Sn <-- soft limit
Which both return 1024 (the user default I imagine).
This can be increased on a per user bais by editing /etc/security/limits.conf
I've set the following limits for the user running JBoss process:
<username> soft nofile 4096
<username>
hard nofile 10240
The following article explains in detail:
http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Action
Having made the above OS configuration change (and crossed my fingers) I'll report back soon!