It should be possible but we haven't tried it. Other people have integrated JBossTS and Tomcat in the past though.
Do you know if anyone has described the process? If not, could you possibly give some pointers on where to start investigating how to do this?
I am looking also for some JTA/JTS functionality in Tomcat+Hibernate. Did you have any luck finding some examples how to set it up?
In order to add JBossTS as a JTA provider for Tomcat you need to create a custom UserTransactionFactory which implements javax.naming.spi.ObjectFactory and returns a com.arjuna.ats.jta.UserTransaction.userTransaction().
In your app context file, in the Context tag add something like this:
<Resource name="UserTransaction" auth="Container" type="javax.transaction.UserTransaction" factory="com.yourpackage.UserTransactionFactory"/>
From you application code, ctx.lookup("java:comp/env/UserTransaction") should give you a UserTransaction instance.
As for Hibernate, see http://docs.jboss.org/jbossas/jboss4guide/r3/html/ch13.html, Using the hibernate objects.