Our production retail application in last few days has started crashing with the error as 'ERROR [org.apache.tomcat.util.net.JIoEndpoint] (ajp-0.0.0.0-8009-Acceptor-0) Socket accept failed: java.net.SocketException: Too many open files'. We need dire help to resolve this tricky production issue. To give more information, below is the highlight of the application-
Application is Web based and was hosted in three separate servers hvaing structure as below-
Web Server having Apache service for Internet Access (Connected to App Server using AJB port 8009)
App Server having Apache & JBoss server for Intranet Access (Connection from Apache to JBOss using HTTP port 8080)
DB Server having Oacle 10g for database
Below is the environment software of all the three Servers
09:35:24,985 INFO [AbstractServer] Starting: JBossAS [6.0.0.Final "Neo"]
09:35:26,393 INFO [ServerInfo] Java version: 1.6.0_24,Sun Microsystems Inc.
09:35:26,394 INFO [ServerInfo] Java Runtime: Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
09:35:26,394 INFO [ServerInfo] Java VM: Java HotSpot(TM) 64-Bit Server VM 19.1-b02,Sun Microsystems Inc.
09:35:26,394 INFO [ServerInfo] OS-System: Linux 2.6.18-194.el5,amd64
Application was hosted and availalbe for access from 10-Oct-2011 and everything was working fine with a load of 50users cosistently till 20-Aug-2012. On 21-Aug-2012, application was no more accessible through Internet/Intranet and when checked in the server log files of JBoss in the path /app/jboss/jboss-6.0.0.Final/server/default/log, in the 5 server log files, below exception is thrown
2012-08-26 02:54:16,871 ERROR [org.apache.tomcat.util.net.JIoEndpoint] (ajp-0.0.0.0-8009-Acceptor-0) Socket accept failed: java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method) [:1.6.0_24]
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) [:1.6.0_24]
at java.net.ServerSocket.implAccept(ServerSocket.java:462) [:1.6.0_24]
at java.net.ServerSocket.accept(ServerSocket.java:430) [:1.6.0_24]
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) [:6.0.0.Final]
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:343) [:6.0.0.Final]
at java.lang.Thread.run(Thread.java:662) [:1.6.0_24]
Attached is one of server log file for your reference. Also checked the file count through Putty and below is output
[tfouser@abc ~]$ /usr/sbin/lsof -p 26844 -l | wc -l
[tfouser@abc ~]$ /usr/sbin/lsof -l | wc -l
[tfouser@abc ~]$ cat /proc/sys/fs/file-max
[tfouser@abc ~]$ cat /proc/sys/fs/file-nr
2550 0 3238931
[tfouser@abc~]$ ulimit -n
We restarted the JBOss server multiple times, restarted the OS, but still the problem exists.
Please help what and where could be issue. Highly appreciate any insight given on this high critical production issue.
server.log.1.zip 1.2 KB