With ESB 3.x, the ESB itself was a transactional resource. This means you could enlist sending and receiving messages to it in a transaction. However, in this model, the exchanges needed to be sent synchronously.
For ESB 4.0, the ESB no longer does anything specific with regards to transactions. It does convey transaction information over the NMR though, even in an asynchronous exchange, something that wasn't possible before. This means you can now read a message of a JMS queue in a transaction and send that to another endpoint asynchronously (e.g. a JDBC endpoint which can now insert into the db in the same transaction).
One last thing: in the initial release for ESB 4, we probably won't have all the flows (ST, SEDA, JMS, JCA) that are in ESB 3.x now. You can make up for most of those by explicitly declaring JMS endpoints, from which messages can be read with or without transaction support and sent along in the NMR. The extra overhead in configuration comes with a lot more fine-grained control over transactions than with an all-or-nothing flow approach in ESB 3.x. More advanced flows will be added to ESB 4 again in later releases though.