Perhaps your transaction timeout in WebLogic is less than your lock acquisition timeout in JBoss Cache? Or perhaps your sync repl timeout? So while still waiting to acquire locks on remote nodes to replicate stuff, the tx times out?
Invalidation is a lot more efficient than replication since data is not replicated across the wire. Recommended if you have a large data set, probably backed by a DB or something, that makes it unlikely for 2 users to work on the same data. (If this happens, you lose the efficiency of the cache and see more cache misses than hits)
Still though, I'd focus on looking for the reason for timeout because this is irrelevant to cache mode used.