I have just added a Stress Test that is supposed to push client server communication to the limit. (It is failing actually - http://jira.jboss.com/jira/browse/JBMESSAGING-1012)
For that test to run properly is required to increase the maxFile limit.
On Linux, you need to do it on /etc/security/limits.conf.. Example:
#<domain> <type> <item> <value> clebert hard nofile 10240 # End of file