At a high level, the purpose of JBoss Remoting is to provide a consistent API for network based invocations and related services, regardless of transport type or underlying implementation. For example, if use JBoss Remoting to make client calls on a server, can swap from rmi transport to http transport via config change (not code change). Could also create own custom transport implementation and easily plug into JBoss Remoting (again just via config change). One of the fundamental concepts with JBoss Remoting is ability to easily customize, extend, or replace implementation components within the JBoss Remoting stack (e.g. transport, marshalling, detection, etc.).
JBoss Remoting does not currently allow for point to multipoint invocations (but is a feature request for 2.2.0 - http://jira.jboss.com/jira/browse/JBREM-433), which JGroups does provide. This is one very compelling reason to use JGroups over JBoss Remoting at the moment.
JGroups, using multicast, is also a much faster transport than the default ones provided by JBoss Remoting (as transports such as http will never be the most performant). We are planning on adding JGroups as one of the JBoss Remoting transports in the future, but since could use JGroups standalone, the other JBoss Remoting transports took priority.
So to answer questions directly:
1. The general use for JBoss Remoting is client to server invocation. It can be used for peer-to-peer invocations, but will require extra coding to set this up (at least until get point-to-multipoint invocations added). Right now, JGroups is a better option for this since most server cluster environments have machines located on same network, so can use multicast (for discovery and group invocations).
2. Think above explains why JBoss Remoting is different from JGroups, but to summarize, JBoss Remoting generalizes common network based features and behavior so that can easily swap out implementations within the stack underneath the same API without side effects. JGroups is specialized for performance.
JBoss Remoting could (and hopefully will) use JGroups as one of its implementation options, but it being the only option would defeat purpose of JBoss Remoting. A quick example of where this would come into play is with auto-discovery within JBoss Remoting. Currently there is a multicast detector, which is similar to JGroups in how it discovers remote servers. There is also a jndi detector implementation. The reason a jndi detector is important is if have servers that exist on different network segments, multicast would not be able to communicate to the different servers (thus some servers would not be detected). With the jndi detector, which uses a central jndi server as registry for the different servers, each server can register with it and detector can see all of the servers registered. Both the multicast and jndi detector use the exact same JBoss Remoting detector api, so from a user standpoint, does not matter which one is deployed (only need config changes to swap between the two based on situation).
3. I think JGroups is the best option (over JBoss Remoting or anything else) when making in network invocations and performance is a priority (especially peer-to-peer). A good example of this is state replication within a cluster as will likely have all the servers on the same network. This is what JBossCache uses and works very well.
If making network invocations across different networks and/or environment is likely to change, then think JBoss Remoting is a good choice since will have very minimal impact on app that has been built (just config changes to meet the need).