I had a problem starting up server from 'ant start-server' in the qa directory. The error displayed below was reported in the AS 5 boot.log. The worked fine with AS 5.0.1.GA but when using the Branch_5_x the server could no longer be started from ant.
This server (test-esb) could be started manually from the command line using ./run.sh -c test-esb with out problem.

Exception:
From boot.log:

11:44:20,475 DEBUG [ServerImpl] Failed to start
org.jboss.xb.binding.JBossXBException: Failed to parse source: file:/opt/jboss/as/src/Branch_5_x_anon/build/output/jboss-5.1.0.CR1/server/test-esb/conf/bootstrap/vfs.xml@13,37
 at org.jboss.xb.binding.parser.sax.SaxJBossXBParser.parse(SaxJBossXBParser.java:177)
 at org.jboss.xb.binding.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:147)
 at org.jboss.bootstrap.microcontainer.TempBasicXMLDeployer.deploy(TempBasicXMLDeployer.java:150)
 at org.jboss.bootstrap.microcontainer.ServerImpl.doStart(ServerImpl.java:138)
 at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:450)
 at org.jboss.Main.boot(Main.java:216)
 at org.jboss.Main$1.run(Main.java:546)
 at java.lang.Thread.run(Thread.java:595)
Caused by: org.jboss.xb.binding.JBossXBRuntimeException: {urn:jboss:bean-deployer:2.0}property cannot appear in this position. Expected content of {urn:jboss:bean-deployer:2.0}bean is sequence: {urn:jboss:bean-deployer:2.0}alias* {urn:jboss:bean-deployer:2.0}related-class* {urn:jboss:bean-deployer:2.0}annotation* {urn:jboss:bean-deployer:2.0}classloader? {urn:jboss:bean-deployer:2.0}constructor? {urn:jboss:bean-deployer:2.0}property* {urn:jboss:bean-deployer:2.0}create? {urn:jboss:bean-deployer:2.0}start? {urn:jboss:bean-deployer:2.0}stop? {urn:jboss:bean-deployer:2.0}destroy? {urn:jboss:bean-deployer:2.0}depends* {urn:jboss:bean-deployer:2.0}demand* {urn:jboss:bean-deployer:2.0}supply* {urn:jboss:bean-deployer:2.0}install* {urn:jboss:bean-deployer:2.0}uninstall* {urn:jboss:bean-deployer:2.0}incallback* {urn:jboss:bean-deployer:2.0}uncallback*
 at org.jboss.xb.binding.sunday.unmarshalling.SundayContentHandler.startElement(SundayContentHandler.java:449)
 at org.jboss.xb.binding.parser.sax.SaxJBossXBParser$DelegatingContentHandler.startElement(SaxJBossXBParser.java:401)
 at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source)
 at org.apache.xerces.xinclude.XIncludeHandler.startElement(Unknown Source)
 at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanStartElement(Unknown Source)
 at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
 at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
 at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
 at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
 at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
 at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
 at org.jboss.xb.binding.parser.sax.SaxJBossXBParser.parse(SaxJBossXBParser.java:173)
 ... 7 more
11:44:20,507 INFO [ServerImpl] Runtime shutdown hook called, forceHalt: true

Since the server can be started manually I wanted to look at the differences when starting AS 5.0.1.GA and Branch_5_x.

I found that an environment variable had been added to run.sh in Branch_5_x:

# This should be removed when JBoss configuration XML files can be validated JBAS-6744
JAVA_OPTS="$JAVA_OPTS -Dxb.builder.useUnorderedSequence=true"

The following jira gives the details : https://jira.jboss.org/jira/browse/JBAS-6743

At the moment I've added 'xb.builder.useUnorderedSequence=true' before starting the server from ant and this works.