You need to make sure your Sybase stored procedure is set to "Any" mode.
For example: EXEC sp_procxmode '<your_proc_name>','anymode'
This will allow the transaction for the stored procedure to be controlled by external clients (ie the JBoss container). It will also work with local clients as well, thus the "Any" mode.
Hope that helps,