JBoss crash under high load
Hi,
I am using an application with SB-BMP only and 500 clients. I use Linux 2.4.12, Sun JDK 1.3.1, Apache 1.3.22, Tomcat 3.2.4, JBoss 2.4.4, MySQL 3.23.43. Apache, Tomcat, JBoss and MySQL run on separate machines (4 different servers) and clients run also on seperate machines.
After 3 minutes I get this error message from JBoss:
[01:20:50,770,Default] java.lang.NullPointerException
[01:20:50,771,Default] at org.jboss.deployment.AutoDeployer.run(AutoDeployer.java:164)
[01:20:50,771,Default]
[01:20:50,771,Default] at java.lang.Thread.run(Thread.java:484)
[01:20:50,772,Default]
Then 1 minute later all connections are stacked on the EJB server machine that runs out of file descriptors and then says:
[01:21:41,483,NamingService] Naming stopped
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:421)
at java.net.ServerSocket.implAccept(ServerSocket.java:243)
at java.net.ServerSocket.accept(ServerSocket.java:222)
at org.jnp.server.Main.run(Main.java:243)
at java.lang.Thread.run(Thread.java:484)
[01:21:41,484,NamingService] Restarting naming
[01:21:41,485,NamingService] Starting jnp server
[01:21:41,486,NamingService] Could not start on port 1099
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketCreate(Native Method)
at java.net.PlainSocketImpl.create(PlainSocketImpl.java:78)
at java.net.ServerSocket.(ServerSocket.java:166)
at org.jnp.server.Main.start(Main.java:200)
at org.jnp.server.Main.run(Main.java:251)
at java.lang.Thread.run(Thread.java:484)
[01:21:41,487,Default] java.lang.NullPointerException
[01:21:41,487,Default] at org.jnp.server.Main.run(Main.java:276)
[01:21:41,487,Default]
[01:21:41,488,Default] at java.lang.Thread.run(Thread.java:484)
[01:21:41,488,Default]
[01:21:41,489,Default] java.lang.NullPointerException
[01:21:41,489,Default] at org.jnp.server.Main.run(Main.java:243)
[01:21:41,489,Default]
[01:21:41,489,Default] at java.lang.Thread.run(Thread.java:484)
[01:21:41,489,Default]
Note that the program seems to work fine with lower loads. The application seems correct since it runs with the same load on other application servers.
Any help on this is welcome. Especially if someone can explain why the naming service stops by itself ?
If I can tune some traces to get some more information without slowing down too much the server, let me know how to do it.
Thanks,
Emmanuel