Message loss in network of brokers - transactional send
Hi,
According to my tests, I seem to be losing messages in my network of brokers setup when I kill source and target brokers randomly while producer is pushing messages to one broker.
It is as if store-and-forward is removing the message from the source broker before making sure that the message is persisted in the target broker.
My setup is like this:
Producer --> Broker1 --> Broker2
Broker1 and Broker2 are configured in network of brokers.
Broker1 has IN.QUEUE and Q1. I've setup IN.QUEUE as compositeQueue which forwards messages to Q1. Q1 is configured in staticallyIncludedDestinations so that all messages are forwarded to broker2.
I have one publisher which sends x number of messages to IN.QUEUE in broker1.
I start both brokers and purge all messages.
I start my producer which sends 5000 messages to broker1 (tested separately with transactional and untransactional send cases).
While the messages are flowing between producer, broker1 and broker2, I start killing (kill the console) broker1 and broker2 randomly and I randomly start them back. In the end, I refresh the queue page in web console and observe number of messages on each queue. I see
Here is my producer code:
private static void Run()
{
var textMessage = GetMessageContent();
var connectionFactory = new ConnectionFactory("failover:(tcp://localhost:61616)");
using (var connection = connectionFactory.CreateConnection())
{
connection.Start();
//using (var session = connection.CreateSession(AcknowledgementMode.Transactional))
using (var session = connection.CreateSession())
{
var queue = session.GetQueue("Q.IN.DC1");
using (var producer = session.CreateProducer(queue))
{
producer.DeliveryMode = MsgDeliveryMode.Persistent;
var startTick = DateTime.Now.Ticks;
for (int i = 1; i < numberOfMessagesToSend1; i+)
{
var message = producer.CreateTextMessage(textMessage);
producer.Send(message);
//if (i % 100 == 0)
//{
// session.Commit();
//}
}
var endTick = DateTime.Now.Ticks;
Console.WriteLine(TimeSpan.FromTicks(endTick - startTick).TotalSeconds);
}
}
}
}