3 Replies Latest reply on May 27, 2009 5:03 AM by timfox

    Behavior with selector and fullSize cache

    mclu

      Hi!

      I am using one of the latest versions of the 1_4 branch

      We are using one queue where multiple bridges/clients consume/transport messages using a selector.

      Try to visualize it: :-)
      Queue A
      Bridge X fetch messages with selector destnode=X
      Bridge Y fetch messages with selector destnode=Y
      Bridge Z fetch messages with selector destnode=Z

      The business logic always create one message per destinationnode with the header property destnode set to X,Y or Z
      The messages in the queue are added in a sequence like:
      a(x),a(y),a(z), b(x),b(y),b(z),....

      Our messages are really big. from 20k to 20 megabyte so we want to limit the fullSize from default 200000 to 100 or similar.

      What happens now if destination X is not reachable for a longer time. What implications does this fullSize setting have then.
      This means that all message with destination (x) can not be delivered for that offline time. The messages in the queue then should look like this

      a(X),b(x),c(x),....t(x),t(y),t(z)

      Lets say we set it to 100 and there are 120 (x) messages in the queue.
      If we now add 3 new messages for x,y and z, does this now mean that the y and z message is not cached in the memoryqueue. Are those messages then delivered? And if so is each message then fetched from the database?

      to summarize:
      What happens is I use a selector and set the fulSize setting to a very low value.

      THX for a short explanation!




        • 1. Re: Behavior with selector and fullSize cache
          gaohoward

          Hi,

          Using fullSize you can limit the max number of messages in memory of a queue. If the queue's messages (not delivered) reaches fullSize, any more messages that are routed to the queue will be 'paged' to DB. As messages in memory are delivered, more messages are 'fetched' from the DB into memory for delivery.

          So if you have 120 messages in queue while the fullSize is 100, 20 messages will be paged to DB.

          You also need to notice that once Bridge X is deployed, it will always create a consumer X even the target X is not reachable for the moment.

          • 2. Re: Behavior with selector and fullSize cache
            mclu

            Hi !

            Thx for explanation. But this basics I know already.

            I did the test with a limit of 50 and added a lot of messages for x y and z to my queue. I also undeployed all bridges.

            Then I deploy bridge x. All messages in the memory which contains the header property destination=x are bridged then. Some messages are reloaded but after a short time the memory queue contains only messages for y and z. The messages which are paged to db for x are NOT delivered.
            If I then deploy Bridge Y some messages of Y and (because of reload) some X are bridged. Then the Queue stopped again because only Z messages are in the memory queue.
            If I deploy Z the queue is completely bridged to all destinations.

            This behaviour of course makes sense! Paged messages are somehow invisible for the JMS system.
            But I think there should be a section in the bridge documentation (user guide) or somewhere else which points to this behaviour. Otherwise users maybe wonder why the queue "stops".

            I my case I think I have to create onle queue per destination. Otherwise I can not prevent memory problems if one or more destinations are not reachable.

            • 3. Re: Behavior with selector and fullSize cache
              timfox

              Yes selectors do not work well with paged messages.

              To implement it otherwise would be difficult - we'd have to scan the entire file store for matching messages every time a consumer is added.

              If you find yourself needing functionality like this, then it sounds like you need a database not a messaging system!

              Databases are designed to do exactly this kind of thing: Large tables of data which can't all fit in memory and "select" subsets of rows matching a criteria.

              Messaging systems support this kind of functionality to some extent, but it is not their primary purpose!