can you post some data to JVM crash? In Java 1.6.0_20 I hade once a problem, that JBoss crashed (within native code) when it ran in a specific Database-Query. The problem was, that an explicit stack size was set and JVM crashed instead of throwing StackOverflowException. Try a more direct call instead of nested EJB calls (if applicable). When exec() hangs, is your shell-script done or does it hang within the shell-script?
The script does not complete, the code hangs on the line "process = Runtime.getRuntime().exec(command, null, workingDir);"
The command is alchemy to generate low res rendtions.
We did not see any StackOverflowExceptions in the logs.
When the number of users requesting renditions increase [which tiggers the Runtime.getRuntime().exec code to be called]
then we are unable to execute Runtime.getRuntime().exec and the code hangs at that line or jboss crashes.
We tried directly executing this code from within jboss when we reached the hung state where only Runtime.getRuntime().exec would hang up but the rest of the application ran fine. however if we run the Runtime.getRuntime().exec form a java program via command line it ran fine.
I think it has to do with some memory settings with jboss which may explain why the same code works fine outside jboss when jboss or the code reaches that hung state.