Sorry to hear that the architecture documentation does not cover this in too much detail. The entry point for all our architectural documentation is available here: http://community.jboss.org/wiki/blacktiearchitecture
Anyway, before I answer your question I should point out the following:
1. Although we have users in the community who develop on Solaris we do not provide binaries for this platform as yet
2. Unfortunately we do not interoperate at the wire protocol level with other XATMI implementations such as Tuxedo at this point in time although that will be the focus for version 3 of the software (we are due to commence development on this at the start of Summer).
3. We do not provide Cobol bindings for BlackTie at this point
On to your questions about service deployment.
The message server and transaction manager are JBoss provided components these are respectively, JBoss Messaging (to be updated to HornetQ as part of the upgrade to AS6) and JBoss Transactions. They are deployed into the java JBoss application server. A Java BlackTie administration is also deployed here.
For C/C++ XATMI services (i.e. user code that implements tpservice), these are deployed external to the application server in freestanding processes. Although we are looking at allowing users to deploy these within the application server process itself.
For Java XATMI services, these are deployed within the application server effectively as specialised XATMI MDB services.
In a typical application therefore you would see:
Standalone C client using the XATMI API to invoke tpcall/tpconnect etc connected to JBoss AS
A Java JBoss AS process providing the infrastructure services
Standalone C process executing user defined tpservice code connected to JBoss AS
Hope this helps to answer your query! We are happy to do anything we can to help fit your use case, and always review issues raised in our Jira instance: https://jira.jboss.org/browse/BLACKTIE
I should also mention that we are always happy for community members to help us out by adding new useful features, so if there is anything you see in BlackTie that you would like to add yourself, please let me know. I am always on IRC, #blacktie on freenode.
All the best,