I have a bit more information, which should reduce the need to resort to divining chicken entrails for those attempting to answer this question.
Unfortunately, this information probably makes it only a little bit simpler to diagnose this issue. Here it is: We are using MySQL, and doing inserts during the file processing referenced above, and running the query "select LAST_INSERT_ID()" in order to obtain the id of the last inserted row.
*** If that LAST_INSERT_ID query is removed, there are no issues. But, if it runs - and it runs generally a few dozen times in the transaction that this file processing code starts - the server no longer responds to HTTP requests afterwards. ***
Pretty wacky stuff, eh?
I am certain that:
- There are no issues with transaction management. (All manuals - all BMP - and we happen to be using the Typesafe Slick library.)
- There are no issues of any kind with threading (there is actually only a single thread involved here).
I've also eliminated the user-initiated thread as a possible culprit by kicking the thing off with a JEE TimerService event instead.
Problem solved. Short version is that we were loading a properties file inside a tight loop and running out of file handles.
On Mac OS, there is NO error or warning message - even with full debug - making this quite mysterious.
Running the same code on RH Linux, JBoss dutifully reports "too many files", making it a breeze to track down.
The universe makes sense after all.