Sounds like its a ulimit problem or possibly a mysql connection limit.
If on Linux you can run $ulimit -n as the user which jboss is running as to see the max open files in which that user can have per process and $sudo lsof -p $PID |wc -l (where $PID is the jboss java process ID) to get an idea how many open files the process has open. I think open network sockets will be excluded in the lsof -P but they're also counted as open files.
You can run $sudo cat /proc/$PID/limits (where $PID is the jboss java process ID) to view the limits of the process.
To fix the ulimit (if on Linux) you can add the below to /etc/security/limits.conf where jboss is the user jboss is running as.
jboss soft nofile 65535
jboss hard nofile 65535
After the change to limits.conf become the user jboss is running as and check that the ulimit -n is equal to 65535 and restart jboss.
If its a mysql connection limit see http://dev.mysql.com/doc/refman/5.1/en/too-many-connections.html for more info
Today, this issue happened again.
I check whether we spent lots of db connection , but I found that we only use 88 although max-connection size is 800.
So we couldn't imagine this issue is happened by db connection problem.
I execute "lsof -i -p 31843" command to find what files is opened and used by this process,
then I found that so many "can't identify protocol" message is displayed. ( about 787 line is displayed. )
Could you imagine what problem could make this message ?