java.net.SocketException: Too many open files on Red Hat lin

[Kiran Krishnamurthy]

Dear All,
We are using JBoss 4.2.2GA on a RHEL with Apache as Webserver.
We keep running out of open files whenever after just hrs since the JBoss server is started.

We also increased the ulimit max open files number ffrom 1024 to 8192 but still we seem to be maxing out this number after a few hrs or when doing load testing. The application that is running on the JBoss server makes HTTP connections to other systems using Apache HTTPClient and the code is releasing the connections using the HTTPClient.

can somebody pls help with this issue.

JBoss 4.2.2 GA, RHELinux -- Quad Code 8GBRAM,

There has been some internal JBoss errors which started occurring at a rapid pace and the server.log went up from 900kb to 350mb in a span of 30-60mins. The following are the errors in the server.log. this error is the only one that is logged in this whole 350mb server.log. there has been no other exception/error logged



2009-06-22 16:20:39,512 ERROR [org.apache.tomcat.util.net.JIoEndpoint] Socket accept failed

java.net.SocketException: Too many open files

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

at java.net.ServerSocket.implAccept(ServerSocket.java:453)

at java.net.ServerSocket.accept(ServerSocket.java:421)

at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)

at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:309)

at java.lang.Thread.run(Thread.java:619)

2009-06-22 16:20:39,512 ERROR [org.apache.tomcat.util.net.JIoEndpoint] Socket accept failed

java.net.SocketException: Too many open files

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

at java.net.ServerSocket.implAccept(ServerSocket.java:453)

at java.net.ServerSocket.accept(ServerSocket.java:421)

at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61)

at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:309)

at java.lang.Thread.run(Thread.java:619)



thanks in advance

Cheers
Kiran

[vony jon]

try to see what file handles are open using following command

lsof -p PID

I did face same problem while using Jboss 4.3 on SunOS.
We were using jboss webservices module and while load testing it was throwing same exception for WSDL

java.net.SocketException: Too many open files

I believe there is a patch available for this. We used that patch and that resolved the problem.

[Kiran Krishnamurthy]

Yes. we are also hitting a Webservices system and we post xml over http instead of WSDL.

There are 2 things that i have done to overcome this problem right now.

1) turned off the deployment scanner in the jboss-service.xml

2) added a executor tag for the server.xml so that it will close the TCP connections after there is idle time.

jboss-4.2.2.GA/server/ottserver/deploy/jboss-web.deployer/server.xml

<Executor name="appThreadpool" namePrefix="activeThread-" maxThreads="1000" minSpareThreads="30" maxIdleTime="30



for now we seem to have jumped over this max open files problem.

But there is a new problem which has cropped up now.

After running load tests for 3/4 hours using JMeter at the end of the 4th hour we saw a sudden spike in the jboss process where its hitting 100% CPU and this didn’t come back down after we stopped the requests from jmeter to the server. we had to kill the server to revive it.

any ideas as to what seems to be the problem.


also can you pls give me more info abt the patch.

thanks in advance.

Cheers
Kiran

[vony jon]

Do you really need maxThreads="1000" in your tomcat connector??

You should not set it that high, that might be reason your CPU is spiking when you do the load test, it will keep creating new threads for your incoming request.

Even if you are expecting 500 concurrent request, i think 50 threads should be able to handle that.
Also you can use acceptCount which is a queue for all your incoming connection requests when all possible request processing threads are in use.

Refer to following URL for the bug which i was talking about
https://jira.jboss.org/jira/browse/JBWS-2325

[Leonid Batizhevsky]

You are resolved your problem?

[Kiran Krishnamurthy]

Yes.. I was able to resolve the problem. the executor was not mentioned which will close the connections/files that are open

Navigate to "jboss-5.1.0.GA/server/ottserver/deploy/jbossweb.sar" and edit server.xml as mentioned below.

Look for following xml snippet.

<Connector port="9090" address="${jboss.bind.address}" maxThreads="250" maxHttpHeaderSize="8192" emptySessionPath="true" protocol="HTTP/1.1" enableLookups="false" redirectPort="8443" acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" />

Replace this snippet with below one.

<Executor name="appThreadpool" namePrefix="activeThread-" maxThreads="150" minSpareThreads="3" maxIdleTime="30000" />

    <Connector executor="appThreadpool" port="9090" address="${jboss.bind.address}"          maxThreads="150" maxHttpHeaderSize="8192"

         minSpareThreads="3" maxSpareThreads="20"          emptySessionPath="true" protocol="HTTP/1.1"

         enableLookups="false" redirectPort="8443" acceptCount="100"      connectionTimeout="120000" disableUploadTimeout="true" />

Look for following xml snippet.

<Connector port="8009" address="${jboss.bind.address}" protocol="AJP/1.3"  emptySessionPath="true" enableLookups="false" redirectPort="8443" />

Replace this snippet with below one.

<Connector executor="appThreadpool" port="8009" address="${jboss.bind.address}" protocol="AJP/1.3" emptySessionPath="true" enableLookups="false" redirectPort="8443" />

hope that helps.

thanks

Kiran

[Leonid Batizhevsky]

Yes. On jboss 4.2.x I resolve this problem  too.

I just compile new jbossweb.jar from svn (2.0.x). It have bugfix.

At now I have CLOSE_WAIT = 0 and happy))

[silambarasan m]

Hi Kiran,

i have tried the same way which you explained. but seems that i am getting again same exception.

is there any new patch is available in Jboss for this issue?

Please let me know how to resolve this issue...

Thanks in Advance.

Regards,

Silambarasan

[Leonid Batizhevsky]

compile this: http://anonsvn.jboss.org/repos/jbossweb/branches/2.0.x/

or use this patch https://jira.jboss.org/secure/attachment/12316577/patch.txt with JBOSSWEB_2_0_0.

IF  patch work you seen close_wait arount zero:

netstat -a | grep CLOSE_WAIT .

[silambarasan m]

Thanks for your reply....

Sorry for asking this...do we have any binary available for the latest with fix for jbossweb.2.0.x?

it would be great if i get jar instead of source code.

here is the exception we are getting in our jboss. but finally all the transcations getting broken and the server got hang-up...

04:55:34,751 ERROR [org.jboss.naming.Naming] Naming accept handler stopping
java.net.SocketException: Too many open files
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

java.net.SocketException: Broken pipe
    at java.net.SocketOutputStream.socketWrite0(Native Method)

I believe the main issue is with the "too many open files", probably this could be solved with the latest jbossweb...

Cheers,

Silambarasan

[Leonid Batizhevsky]

I make myself binary and test it for jboss 4.2.1 and jboss 4.2.3.

just replace jbossweb.jar in deploy/jboss-web.deployer/ folder

[Shane Weaver]

We're having the same problem on JBoss 4.2.1 GA - we compiled jbossweb 2.0.0.x and replaced the jbossweb.jar file under jboss-web.deployer as suggested, but getting the following error on startup.  Are you sure no other change is needed?

org.jboss.deployment.DeploymentException: - nested throwable: (java.lang.reflect.UndeclaredThrowableException)

Caused by: java.lang.NoClassDefFoundError: org/apache/catalina/Lifecycle

It's odd, because Lifecycle is definitely found within the jbossweb.jar file.

[Leonid Batizhevsky]

Shane Weaver

No, I do not have any problems like this