This is about any typical two EJB methods calls.
1. CMT and the two methods is within a Transaction.
If the 1st method has called EJBContext.setRollbackOnly() and return, the next method call will cause an javax.transaction.TransactionRolledbackException with a message of "Transaction marked for rollback, possibly a timeout"
2. CMT and the two methods is not within a Transaction.
If the 1st method has called EJBContext.setRollbackOnly() and return, the next method will not cause such an TransactionRolledbackException. The transaction of the 1st method just silently rollback.
This kind of behavior implies that a Bean provider has to tell the Bean user that a setRollbackOnly() has been called within a method or the user cannot correctly predict a possible TransactionRolledbackException and catch it.
By the way, the "..., possibly a timeout" is not very descriptive about the real cause of the rollback. This exception comes from QueuedPessimisticEJBLock.doSchedule() which seems only checks the TransactionManager's rollback flag (isTxExpired(miTx)) to decide wether a "Timeout" has happened. Shouldn't a timeout flag be provided to make it more specific?