Silly me, found my own answer on google:
Seems that having purely an IP only difference isn't good enough due to the linux IP stack implementation (or so it would seem).
Using the same IP, but different ports seems to work (at least for linux 2.6.7)
So I'll switch my stuff to leave the port as they ship, but switch the IP to be specific to a cluster.
(in case that link above didn't work, turns out win 95 and 'some unix' have this issue, but NT+ and some 'other unix' don't.... YMMV)