Yes, that describes the problem very well.
We had the bind_port specified, thus causing the rejoined node to have the same identity as before.
I removed the bind_port from the protocol and it seems to work now.
Great ! We will definitely fix this with logical addresses, but I'm glad to hear it works for you until then.