You should be fine as long as you
- use the same JDK (vendor, version)
- the same version of JGroups and
- the same version of JBoss
Hmm, in most instances this should be fine, but in your particular case (Mac OS + Linux) you may have issues since the Sun JDK is only avbl for Linux, Windows and Solaris. Apple has their own JDK impl for Mac OS X.
Still, do give it a go (make sure you use the same JDK version even if vendors will differ).
As a non-production test environment here i'm running a two node Linux and Windows cluster - it seems to be working okay
I am also using a WindowsXP/Linux hybrid cluster... No problems whatsoever.