The EJB3 forum is the correct place for this question, not here.
I haven't looked at the EJB spec, but from a transaction service perspective (and specifically the OTS/JTS), the answer is clear: yes, it should cause the transaction to rollback. All errors and system-level exceptions must cause the transaction to abort to ensure consistency. In the case of the OTS, user-level exceptions are the "exception" to the rule and it's up to the application to decide whether or not it can recover without affecting the transaction.
This is a question for the EJB3 forum. and not here
The spec is clear on how these are handled. Any throwable which is not declared as an application exception (or marked with the appropriate annotation) causes the transaction to rollback.
Of course you can have an Error which does not rollback the transaction should you decide to do this. (not recommended)