NAKACK errors on retransmit when cluster is under heavy load

[hmesha]

Env:
Hardware: 2 x86 based machines
OS: Suse Linux Enterprise Server 9 SP3
Server: JBoss 4.0.5.GA
JGroups : 2.4.1

We're running a performance test on the cluster above. 100 concurrent virtual users with 1 second pause time between actions. After sometime (4hours) in the test I observed the following error on the server console.

2007-02-16 17:12:32,294 ERROR [org.jgroups.protocols.pbcast.NAKACK:error] (requester=-.-.-.205:33509, local_addr=-.-.-.205:33509) message -.-.-.205:33509::1172743 not found in sent msgs.
Sent messages: [1172262 - 1172861] (600)
2007-02-16 17:44:39,599 ERROR [org.jgroups.protocols.pbcast.NAKACK:error] (requester=-.-.-.205:33509, local_addr=-.-.-.205:33509) message -.-.-.205:33509::1628057 not found in sent msgs.
Sent messages: [1627729 - 1628398] (670)

Note: I omitted the servers subnet for security reasons.

The errors above occur not freqently (once or twice every 4 hours) on the servers but I'd like to understand why they happen. It looks to me like a concurrency issue with NAKACK when looking at the code base for JGroups 2.4.1.

Can someone explain the error?

Thanks,

[belaban]

It is an intermittent issue, when a retransmission request arrives for a message that has not yet been added to the retransmission table.
I assume those warnings are intermittent; they don't cause further problems,do they ?

[brian.stansberry]

Bela, I noticed in the error msg the requester and original_sender all appear to be the local address (assuming the hidden subnet is the same). Is that expected?

[belaban]

Yes, that can be expected. Especially if loopback is enabled

[hmesha]

Bela,

Thanks for the prompt response. I'm not sure if it's intermittent or not. I asked my QA person to rerun the test to confirm that and I'll report back here.

Tthe loopback adaptor are enabled on the machines as Brian noted and the fact that the requester and the sender ip address are the same in the error makes me think that's not affecting anything since there's no lost messages, techincally.

Again I asked our QA to perform a second test with loopback disabled after the first test mentioned above to confirm that the error is not showing anymore.

Will report back.

Hany

[belaban]

What I meant was that if this was not intermittent, then the process would stop delivering messages due to the gap in sequence numbers. This does not appear to be the case