jBoss hang up with " Too many open files" error

[headiron]

Hi All.

We have used jboss4.0.2 as application server for our service.

We also use DataSource to connect database.

Two days ago, Our service was stopped because DataSource wasn't work correctly.

( "Too many open files" message was displayed on sysout.)

We restarted jBoss and service worked correctly.

I reviewed sys log to find main reason and I found some strange message.

1. Destroying connection message is displayed regurally.

    We set "valid-connection-checker-class-name" element on datasource config and it was printed during execute connection checking routine.

    I expect it displayed few time, but it printed every 10 mins.

    Is it correct behavior?

2. ERROR [AbstractDeploymentScanner$ScannerThread] Scanning failed; continuing was founded.

   Above error message also displayed on sysout with "Too many open files" message.

   What is above message and which service is concerned ? 

3. "Too many open files"

    Above message was written with stack trace about DB connection.

Above two messages are concerned with "Too many open files" error.

Could you help me to find main reason of it ?

[nateboyce]

Sounds like its a ulimit problem or possibly a mysql connection limit.

If on Linux you can run $ulimit -n as the user which jboss is running as to see the max open files in which that user can have per process and $sudo lsof -p $PID |wc -l (where $PID is the jboss java process ID) to get an idea how many open files the process has open. I think open network sockets will be excluded in the lsof -P but they're also counted as open files.
You can run $sudo cat /proc/$PID/limits (where $PID is the jboss java process ID) to view the limits of the process. 

To fix the ulimit (if on Linux) you can add the below to /etc/security/limits.conf where jboss is the user jboss is running as.
jboss soft nofile 65535
jboss hard nofile 65535

After the change to limits.conf become the user jboss is running as and check that the ulimit -n is equal to 65535 and restart jboss.

If its a mysql connection limit see http://dev.mysql.com/doc/refman/5.1/en/too-many-connections.html for more info

[headiron]

Hi  Nathan.

Today, this issue happened again.

I check whether we spent lots of db connection , but I found that we only use 88 although max-connection size is 800.

So we couldn't imagine this issue is happened by db connection problem.

I execute "lsof -i -p 31843" command to find what files is opened and used by this process,

then I found that so many "can't identify protocol" message is displayed. ( about 787 line is displayed. )

Could you imagine what problem could make this message ?

Thanks

Ducheol