1 Reply Latest reply on Jun 16, 2011 10:34 AM by Ozan Seymen

    Message loss in network of brokers - transactional send

    Ozan Seymen Newbie

      Hi,

       

      According to my tests, I seem to be losing messages in my network of brokers setup when I kill source and target brokers randomly while producer is pushing messages to one broker.

       

      It is as if store-and-forward is removing the message from the source broker before making sure that the message is persisted in the target broker.

       

      My setup is like this:

       

      Producer --> Broker1 --> Broker2

       

      Broker1 and Broker2 are configured in network of brokers.

       

      Broker1 has IN.QUEUE and Q1. I've setup IN.QUEUE as compositeQueue which forwards messages to Q1. Q1 is configured in staticallyIncludedDestinations so that all messages are forwarded to broker2.

       

      I have one publisher which sends x number of messages to IN.QUEUE in broker1.

       

      I start both brokers and purge all messages. 

       

      I start my producer which sends 5000 messages to broker1 (tested separately with transactional and untransactional send cases).

       

      While the messages are flowing between producer, broker1 and broker2, I start killing (kill the console) broker1 and broker2 randomly and I randomly start them back. In the end, I refresh the queue page in web console and observe number of messages on each queue. I see

       

      Here is my producer code:

       

      private static void Run()

              {

                  var textMessage = GetMessageContent();

                  var connectionFactory = new ConnectionFactory("failover:(tcp://localhost:61616)");

       

                  using (var connection = connectionFactory.CreateConnection())

                  {

                      connection.Start();

       

                      //using (var session = connection.CreateSession(AcknowledgementMode.Transactional))

                      using (var session = connection.CreateSession())

                      {

                          var queue = session.GetQueue("Q.IN.DC1");

                          using (var producer = session.CreateProducer(queue))

                          {

                              producer.DeliveryMode = MsgDeliveryMode.Persistent;

       

                              var startTick = DateTime.Now.Ticks;

                              for (int i = 1; i < numberOfMessagesToSend1; i+)

                              {

                                  var message = producer.CreateTextMessage(textMessage);

                                  producer.Send(message);

       

                                  //if (i % 100 == 0)

                                  //{

                                  //    session.Commit();

                                  //}

                              }

                              var endTick = DateTime.Now.Ticks;

                              Console.WriteLine(TimeSpan.FromTicks(endTick - startTick).TotalSeconds);

                          }

                      }

                  }

              }

        • 1. Re: Message loss in network of brokers - transactional send
          Ozan Seymen Newbie

          Here is some more info:

           

          I enabled debug logging and can see where the problem is (however I still don't know what the solution is).

           

          I ran my test again and send 5000 messages to one broker which is configured to store-and-forward the message to broker2.

           

          After killing brokers randomly, I ended up 4997 messages in broker2. I wrote a consumer that will consume all messages from broker2 and create a spreadsheet with all properties. Using this spreadsheet I can pinpoint which messages have failed using sequential ids. One of the messages that has failed is #3751.

           

          Looking at the debug log I can see:

           

          (line 976) bridging (broker1 -> broker2) messageId =

          ID:HAM-NB-073-56843-634438260819794239-1:0:1:1:3748

          (line 979) bridging (broker1 -> broker2) messageId =

          ID:HAM-NB-073-56843-634438260819794239-1:0:1:1:3749

          (line 983) bridging (broker1 -> broker2) messageId =

          ID:HAM-NB-073-56843-634438260819794239-1:0:1:1:3750

           

          Then broker1 is killed.

          When it comes back, it says:

          (line 1311) last stored sequence id set: 3751

          (line 1312) suppressing duplicate message send ID:HAM-NB-073-56843-634438260819794239-1:0:1:1:3751 with producerSequenceId less than last stored: 3751

           

          There is not message sent information in the log for 3751!!!

           

          You can see the full log here: http://activemq.2283324.n4.nabble.com/file/n3602343/activemq.log.4

           

          I'd appreciate any help to solve this problem.

           

          This test was done with apache-activemq-5.4.2-fuse-02-00