We had a similar problem on some systems. This was fixed by making sure there is a DNS reference to the target system. Apparently, the RMI code seems to attempt a reverse dns lookup which can cause delays if not found. This can easily be solved by having a reference to the server in the hosts file.
Hope this helps you,
Also, make sure netbios is switched off. Especially if there is firewall between your servers. Windows send UDP broadcasts to find other machines if netbios is enables, which, if the firewall block them can lead to 2-3 sec delay.
Thank you for your help.
Having a reference in the hosts file solved the delay problem.