As far as I can tell, JBM can run on 4.0.x series without any major issues. I am running JBM 1.2.GA on 4.0.5.GA in Production!
Although this is not advisable, due to many restrictions in the project, we have no option except running JBM on 4.0.5.GA. All seems well (except few minor quirks) so far.
Related to your *how* question, follow the manual and deploy the JBM.
Madhu is correct. When running JBM with the AS 4.0.x, Jboss Messaging must run inside of a scoped classloader. 4.0.x has versions a libraries for remoting and aop that are not compatible with Jboss Messaging. JBM requires more up to date libraries. To keep these straight the app server and JBM have to be run in two different classloaders.
I have heard of people running JBM 1.3 and 1.4 with 4.0.x, but you have to look at jbm 1.2 and use the scoped classloader defined in that version. Not sure I would recommend it, because it isn't common, which means that it's not tested.