I know why the test is failing.
For some reason the page is being deleted before the ACKs were replicated. Whenever the server failed over, the current page was still the old page, however it had a record that was already acked.
From Print-pages on live server:
###################################################################################################
Exploring store FailoverTestAddress folder = page//ed1ee6eb-171a-11e2-9c01-4984cbdc0191
******* Page 26
~
From Print-pages on backup server:
####################################################################################################
Exploring store FailoverTestAddress folder = page_backup//ed20467c-171a-11e2-9c01-4984cbdc0191
******* Page 25
pg=25, msg=0,pgTX=-1, msg=ServerMessage[messageID=199,priority=4, bodySize=200,expiration=0, durable=true, address=FailoverTestAddress,properties=TypedProperties[{counter=98, sendCallNumber=98}]]@1829923591,Queues = 3 (ACK)
pg=25, msg=1,pgTX=-1, msg=ServerMessage[messageID=2147483652,priority=4, bodySize=200,expiration=0, durable=true, address=FailoverTestAddress,properties=TypedProperties[{counter=100, sendCallNumber=100}]]@1412485250,Queues = 3 (ACK)
pg=25, msg=2,pgTX=-1, msg=ServerMessage[messageID=2147483654,priority=4, bodySize=400,expiration=0, durable=false, address=FailoverTestAddress,properties=TypedProperties[{counter=101, sendCallNumber=101}]]@1355362812,Queues = 3
pg=25, msg=3,pgTX=-1, msg=ServerMessage[messageID=2147483656,priority=4, bodySize=200,expiration=0, durable=true, address=FailoverTestAddress,properties=TypedProperties[{counter=102, sendCallNumber=102}]]@1223737555,Queues = 3
....
and htis was before the test could ack any more messages.
With this in hand I will be able to fix the test quickly now. Any time between tomorrow and Wed depending on what I have to do. (like if I have to do any major changes on the replciation for paging).
I will update this thread as I find more.
