Sure. Technically, would only have to implement org.jboss.remoting.transport.ClientInvoker interface for client side implementation and extend the org.jboss.remoting.ServerInvoker class on the server side. However, would extend RemoteClientInvoker on the client side, so then only have to implement the abstract transport() method. User guide and wiki (http://www.jboss.org/wiki/Wiki.jsp?page=Remoting_configuration) goes into detail on how to configure for your own custom implementations.
Feel free to also extend the SocketClientInvoker and SocketServerInvoker as you like.
If you do build your own, would be great if you could contribute it back to the repository.