Yeah, that's a problem. Setting the -Djboss.remoting.pre_2_0_compatible=true flag affects Remoting throughout the JVM, so you can't satisfy both EJB and JBossMessaging. I don't think there's a way around that in AS 4, which uses Remoting version 2.2.x.
On the other hand, AS 5 uses Remoting 2.4/2.5, which has an option for setting the "remotingVersion" parameter on each Remoting Connector. So, in $JBOSS_HOME/server/$CONFIG/deploy/remotng-jboss-beans.xml, you could update the "UnifiedInvokerConfiguration" bean with
<!-- Parameters visible to both client and server -->
<map keyClass="java.lang.String" valueClass="java.lang.String">
That would configure EJB2 to use the old wire format, and JBossMessaging would default to the new wire format.