JBoss 7.1.1 undeploying the application after too many files open exception

[pallavijain20]

Hi There,

I am seeing strange behaviour od JBoss AS7, on load testing when we run out of file descriptor and start getting following exception

2012-10-31 15:08:04,232 | INFO | http--0_0_0_0_0_0_0_0-8080-194 | LogUtils | Application {http://cardmanagement.payment.services.com/}CardManagementServiceImplService#{http://www.com/external/services/payment/cardmanagement/}AddCard has thrown exception, unwinding now: com.external.services.payment.cardmanagement.CardManagementFaultMessage: Too many open files in system

2012-10-31 15:08:04,318 | ERROR | http--0_0_0_0_0_0_0_0-8080-327 | AbstractLoggingWriter | java.net.SocketException: Too many open files in system

2012-10-31 15:08:04,327 | INFO | http--0_0_0_0_0_0_0_0-8080-197 | LogUtils | Application {http://withdraw.payment.services.sportingbet.com/}WithdrawServiceImplService#{http://www.com/external/services/payment/withdraw/}Withdraw has thrown exception, unwinding now: com.external.services.payment.withdraw.WithdrawFaultMessage: Too many open files in system

for previous releases of JBoss server use to become non responsive, but strangely JBoss AS7 undeploy all the services and keep running.

Is anyone faced same issue or some possible explanation for behaviour.

Pallavi

[smarlow]

No (not since JBoss 4.0.4 anyway), but am curious if you have tried monitoring the open files in use enough to know the cause of the "out of file descriptors"?

Sometimes we forget to close the (java.io) input/output stream and rely on the Java garbage collector to close the stream.  When I say we, I mean to include your application but not limited to your app.

[pallavijain20]

Hi Scott,

Yeah I looked for the open files and look like  third party application called from one of our services may be cause of FD leaks.

But I just wanted confirm that JBoss 7 is not behaving strange since I cannot see any bug report either.

Regards,

Pallavi

[cguzzman]

Hello Pallavi,

I have exactlly the same issue.

Did you figure out what the cause is?

Regards!

[ctomc]

Hi and welcome to forums.

issue is either

1) bug in your code

2) too low ulimit settings which defaults to 1024 on linux

i would recomend you to read http://stackoverflow.com/questions/4289447/java-too-many-open-files

and http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/

[cguzzman]

Hey Tomaz,

Thank you for your super fast answer.

This happen when we stress the server. The deployed applications are Web Services.

It seem like is a low ulimit problem, because the lot of sockets opened to dispatch the request. And because, in our code, we never open files.

I will give a try increasing the limit and I will let you know.

Regards!

[cguzzman]

Hello to everyone.

I just want to let you know that the problem was solved increasing the ulimit. JBoss stays up like a rock.

My last question is:

is this behavior normal? I mean the undeployment proccess if the server throws the exception "too many open files".

Have a nice day.

[henk53]

Carlos Guzman wrote:

My last question is:

is this behavior normal? I mean the undeployment proccess if the server throws the exception "too many open files".

It seems to be normal indeed. I've seen this happening with many JBoss versions from the earlier AS 4 to the very latest. It's easily reproducible via a JMeter based load test. I've never seen any explicit documentation that says that JBoss undeploys when it's out of file descriptors. It's kinda like the Linux OOM killer, with the main difference that the Linux OOM killer is documented and as far as I know the JBoss undeployer is not.