Documentation has been moving around alot. But its in a new home now:
User docs: http://rhq-project.org/display/JOPR2/Home
Developer docs: http://rhq-project.org/display/RHQ/Home
As for protocols and things.
The server talks to the agent over port 16163 by default (this is configurable - you can tell the agent what to listen to, and the server will talk over that port).
The agent talks to the server over http on port 7080 or over http/s on port 7443. Again those ports are configurable.
server->agent utilizes JBoss/Remoting, over raw socket protocol - its built into the Jboss/Remoting infrastructure. This may or may not use SSL, depending on how you configure it.
agent->server also utilizes JBoss/Remoting but it goes over http/https protocol (by default)
If you want the source, all low level comm stuff (regardless of direction - server->agent or agent->server) is in the modules/enterprise/comm maven module from the source tree. That comm module utilizes Jboss/Remoting under the covers, so you can look at the JBoss/Remoting source. Everything is open source and available.
Some doc links that may or may not be helpful:
Thanks a lot. That's exactly what I was looking for.