You can actually use the Host Controller to monitor your memory usage, for example by using the REST API exposed by it. But if you want to use JMX, you have to:
1- create a socket binding
2 - create a connector
3 - enable JMX without management endpoint
4 - create an application user
done by running bin/add-user.sh (option b)
Then from your JMX client (like VisualVM) use "service:jmx:remote://<server-node-host>:4447" with the credentials of the user you've created.
All of that can be done without a restart of the domain.