4 Replies Latest reply on Feb 28, 2014 1:42 PM by jiraiya-sama

    Message grouping best practice for defining IDs

    jiraiya-sama

      Hi,

       

      I have to implement messages processing with grouping needs. These messages will come from OPC servers each one connected to a set of physical devices. Data sent by these devices can be split in multiple group of messages with a well identified ID, that is where grouping needs appear because each group has to be processed by a single consumer to manage treatment consistency (this is part of my functional needs, so trust me for this point).

       

      I can easily make this ID a group ID  to use HornetQ message grouping feature but here comes my question: How HornetQ manage relastionship between a group and a consumer?

       

      I ask this because documentation speaks about an example with a group per stock, So associations between consumers and stocks are quite durable and cardinality should be stable.

      In my case, for each group, I will have a new ID and so, if HornetQ keep trace indefinitely of all associations, I think I will have a problem after a certain execution duration.

       

      So finally, do you think I can use this ID for grouping or should I find something else more stable in term of cardinality?

       

      Thanks in advance for your feedback.

       

      Yohann

        • 1. Re: Message grouping best practices
          ataylor

          How HornetQ manage relastionship between a group and a consumer?

          One a new group id is used the broker will pick the next available consumer and use that for all the messages with this id.

          I ask this because documentation speaks about an example with a group per stock, So associations between consumers and stocks are quite durable and cardinality should be stable.

          In my case, for each group, I will have a new ID and so, if HornetQ keep trace indefinitely of all associations, I think I will have a problem after a certain execution duration.

          what problem do you think you will have?

          So finally, do you think I can use this ID for grouping or should I find something else more stable in term of cardinality?

          can you explain what you mean by more stable in terms of cardinality?

          • 2. Re: Message grouping best practice for defining IDs
            jiraiya-sama

            Hi, thanks for your fast feedback.

             

            Per day, I have more or less 10,000,000 of messages with approximately 10 messages per group so 1,000,000 groups will be used per day. Messages will be consumed by approximately 10 permanent consumers.

             

            HornetQ will be used with high availability features. So, for a complete year without any stop (but it is not my limit), I will have a huge number of id used (365,000,000). If HornetQ keeps trace indefinitely of all these associations between Ids and consumers, It is sure that there will be a memory or filesystem issue depending on storage mechanism, or performance issue on grouping handlers. In other words, it will be just like an hashtable which grows continuously. So the only question is when limit will be reached.

             

            Because  my groupids are only used once for a set of 10 messages, so for a short time, I am asking if HornetQ really keeps these associations permanently or if there is a mean to discard them (after a defined delay since last message or something like this).

             

            If it is permanent, I will have to find another way to define my Id (per physical device for example: with this, I will have 10 000 group ids reused along the days) to have a more stable and lower number of groupid and so associations in memory like it is in documentation example with one ID per stock. But it is more complicated for me to use id per device, with potential performance issues.

             

            I hope it is clear (I am french so I hope my english speaking is good enough).

             

            Thanks in advance.

             

            Yohann

            • 3. Re: Message grouping best practice for defining IDs
              ataylor

              you can set a group-timout on the handler so groups are GC'd after not being used for this amount of time.

              • 4. Re: Message grouping best practice for defining IDs
                jiraiya-sama

                Thanks a lot. It is exactly what I need. I did not see it in user manual but it is in configuration reference. Sorry for my lack of search...