Thanks for asking about this. It's a good question.
One of the reasons the agent self-install code is in BMUnit but not in the agent jar itself (byteman.jar) is that it can only be compiled & invoked with a dependency on bminstall.jar. Now, byteman.jar cannot depend on bminstall.jar. For a start, the former may get hoisted into the bootstrap classpath while the latter still remains in the system classpath.
The BMUnit code you are interested in is mostly a fairly thin wrapper around an API provided by class Install. However, it does include some useful stuff to propagate configuration settings to Install and, more importantly, work out the current process id. It might well make sense to migrate a little bit of that BMUnit code down into Install and/or provide an alternative API to access the Install functionality.
If you can think of a way to retain compatibility for existing BMUnit functionality while exposing a more easy to use API from the bminstall jar then I'll be happy to consider pushing the necessary changes. If you need advice or are finding it hard to get right then I can also look into it. Raise a JIRA and propose a fix!