-
1. Re: Paging
Tim Fox Jan 6, 2010 1:14 PM (in response to Marwan Simaan)You are not acknowledging your messages.
This has been discussed on several threads before.
-
2. Re: Paging
Tim Fox Jan 6, 2010 1:16 PM (in response to Tim Fox)If you haven't acked messages they will stil be in memory on the server. Therefore the server cannot depage any more messages from storage so you can consume them, since this will increase server memory above the max limit.
10K is a very small figure for max size. Make sure it is greater than ack batch size.
-
3. Re: Paging
Clebert Suconic Jan 6, 2010 1:17 PM (in response to Tim Fox)10 KIB is too little also. You should probably use something like 10 * 1024 * 1024 (10 MIB). -
4. Re: Paging
Marwan Simaan Jan 7, 2010 5:25 AM (in response to Clebert Suconic)Thanks guys for your quick replies.
I read the following thread and jira issue: http://community.jboss.org/thread/19106?start=30&tstart=0 and https://jira.jboss.org/jira/browse/HORNETQ-182
So in the end, on my session is on AUTO_ACKNOWLEDGE. So the problem must was the maximum size. I changed it to 10 MB and was able to consume all the messages.
I still have one more question, I noticed that I couldn't see all the messages in Hermes though. Do you have an idea why?
The thread also convinced me of reverting from using Spring's JmsTemplate. I will replace it with my own implementation.
Thanks,
Marwan
-
5. Re: Paging
Jeff Mesnil Jan 7, 2010 5:27 AM (in response to Marwan Simaan)1 of 1 people found this helpfulHermes allows to see only messages from a queue which are in memory but not the "paged" messages (saved on the file system).
So it is expected to not be able to browse all messages if some are paged.
-
6. Re: Paging
Marwan Simaan Jan 7, 2010 5:32 AM (in response to Jeff Mesnil)But in some way this is misleading. For an end use, the paging mechanism should be transparent. My guess is that Hermes does a browse on the queues, and this can be made to read the messages paged.
Does the paging work with consume/browse + selectors?
-
7. Re: Paging
Jeff Mesnil Jan 7, 2010 5:45 AM (in response to Marwan Simaan)You're right: browsing and paging are conflicting with each other.
The whole idea behind paging messages is to not exhaust server memory by keeping all messages in memory, so we store them on file system once a given threshold size is reached.
Meanwhile, browsing a message means that the server will be left it in the queue and the client will not acknowledge it.
If we were depaging messages when the client browses the queue, this mean that eventually *all* messages could be depaged and
take server memory without being ever acknowledged by the client. This could result in an OutOfMemoryException on the server just
when browsing a huge queue and this is not something we want to allow.
Having said that, we have several enhancements planned to HornetQ 2.1 to give a bit more information on paged messages.
Paging works fine with consumers (and selectors) but is conflicting with browsers.
Hope this clarifies your issue,
jeff
-
8. Re: Paging
Tim Fox Jan 7, 2010 6:28 AM (in response to Jeff Mesnil)It's not really a conflict.
When a message is paged, this occurs *before* a message is routed to any queues, therefore the message is not in any queues yet, so browsing is correct in not viewing that message - it really *is not* in the queue.
We've also discussed this before on other threads.
-
9. Re: Paging
Marwan Simaan Jan 7, 2010 8:27 AM (in response to Tim Fox)If the messages are not really on the queue how does the consumer receive with a selector?
From a user point of view this is a conflict because you send a message to the queue and you can't see (browse) this message. This is not in the JMS specification at least.
To get this straight, is paging the recommended solution with HornetQ if you have a lot of queues with a lot of messages? I know it is not the default solution.
We used to work with Websphere MQ before and we didn't have this problem. My guess that messages are not stored in memory in MQ, but this is transparent to the user.
We are interested a lot in HornetQ as a standalone server due to portability advantages. So i am trying to understand how this works.
Thanks a lot for the help,
Marwan
-
10. Re: Paging
Andy Taylor Jan 7, 2010 8:42 AM (in response to Marwan Simaan)Messages can not be consumed by a consumer if they are paged. It works like this, a message is routed via an address which will have 1 or more queues. At this point a check is made to see if paging should occur. If yes then the message is paged, if not they are delivered to the queues. At some point in time when n messages have been acknowledged then messages are depaged and put onto the queues. So if you are using selectors it is possible that a message won't get consumed even if it should be the next message delivered. The trick here is to tune your application so this doesn't happen, adjust the paging size and make sure you have enough memory to cope with this.1 of 1 people found this helpful -
11. Re: Paging
Marwan Simaan Jan 7, 2010 8:50 AM (in response to Andy Taylor)Okay it is clear now.
But this doesn't follow the JMS standard, right?
-
12. Re: Paging
Tim Fox Jan 7, 2010 8:54 AM (in response to Marwan Simaan)I don't follow.
What is it that you think doesn't follow the JMS spec?
-
13. Re: Paging
Marwan Simaan Jan 7, 2010 9:13 AM (in response to Tim Fox)The fact that you cannot browse all the messages and the fact that you cannot consume any message using a selector.
What about the recommendation for paging? Is there another way?
-
14. Re: Paging
Andy Taylor Jan 7, 2010 9:18 AM (in response to Marwan Simaan)A browser is just a snapshot of a queue at a particular point in time, if a message is paged then it wont be browsed. remember that producers and consumers are decoupled so there no requirement by the jms spec regarding timeliness of when a message should actually reach the queue.
Regarding another solution re paging, i'm not sure what you are trying to acheive. Its a trade of a server can only handle so many messages in memory before running out of it, so you either delay messages and page them or run out of memory. I am guessing that most messaging vendors will do something similar. Like i said before, you can tune paging so this doesnt occur.