Did you ever get any leads on this? I'm having a similar problem.
In my case, the backslashes are simply stripped out. If I installed JBoss 5.0.0 here:
and used the jboss.server.home.dir variable in properties-service.xml in this fashion:
then JBoss told me that it couldn't find this file:
I've seen this behavior in v5.0.0, v5.0.1, and v5.1.0, despite the fact that https://jira.jboss.org/browse/JBAS-3701 claims that this bug was fixed in v5.0.0 beta 1.
Please create a new JBAS JIRA if this is still a problem in JBoss AS 5.x and add the details. Please also point to this thread in that JIRA.