The short and safe answer: Yes, it must be restarted every time you wish to make config changes.
The seems to be working for us answer: You can get a reference to the MainDeployerMBean and call redeploy() on your .esb archive.
The are you sure you want to do this? answer: If your changes are structural in nature (adding, removing services, adding file gateways), perhaps you should consider whether JBESB is the solution for you. If your changes are configuration-based (hostnames, timeout values, etc), perhaps you should consider removing these arguments from the jboss-esb.xml file, and providing them to your application through other means.
The jbossesb-properties.xml is a deployer-level file, so I don't think so, but maybe a maintainer can step-in and correct me. You could implement your own message interceptor, and add it to the chain. You could then tag messages from one .esb archive with a property that would indicate to the custom interceptor that those messages should be trace-logged. Of course, cross-deployment-unit message passing might create an over-logging problem. Trace-logging sounds like it's really, really expensive, btw. But we have millions of messages a day, so my perspective may be different.