-
15. Re: Memory problems in paging mode makes server unresponsive
If the code getting the connection from this <tx-connection-factory> is running in an active JTA transaction (e.g. in an EJB method with REQUIRES or REQUIRES_NEW, etc.) then any operation done with that connection will be part of that JTA transaction.
I see. Thanks for this information. Then it's up to the user to verify that its client is running inside an EJB or the user needs to take of starting a JTA user transaction. But back to the original question then: the message producing is NOT using transactions in my test. Only the consumers run in JMS transactions (one tx per message).
I have configured now a smaller page-size-bytes of 100k each. Although it creates a lot of files now this looks a bit better performing and a bit more stable now. The thing is that the system cannot cope with a massive load unless the producers are throttled down to a rate which is still processable for the servers when they are paging. Am I right that the producer throttling cannot be done dynamically based on the server load?
-
16. Re: Memory problems in paging mode makes server unresponsive
The thing is that the system cannot cope with a massive load unless the producers are throttled down to a rate which is still processable for the servers when they are paging.
Given the current design (paging as a palliative, not to be used like a database) this isn't surprising.
Am I right that the producer throttling cannot be done dynamically based on the server load?
Yes.
-
17. Re: Memory problems in paging mode makes server unresponsive
Right now if you use transaction on sending.. you will be using one object for every message you sent.
if you want infinite paging.. you have to send non transactionally, persistently.
-
18. Re: Memory problems in paging mode makes server unresponsive
Just a quick update on that topic. I have now upgraded the server to EAP6.2 which uses HornetQ 2.3.12. Paging looks much more stable in this version.
In my latest tests I encountered one strange phenomena. Besides everything runs smoothly (producers sent messages, consumers receive messages in time), one node in the cluster suddenly starts paging without a visible reason. The consumers are quick enough to fetch the messages from the queues. Looking at the message counters there is no increase of messages in the queues. But the server continues paging which has a remarkable influence on the producer performance. The worst thing is that the heap is filling up quickly. This will lead to OOM sooner or later. After the node restart the server returned to normal operations (paging files were removed very quickly).
Is there any reason why this happens although there is no need for it? Looks like unexpected behavior for me.
Thanks,
Marek
-
19. Re: Memory problems in paging mode makes server unresponsive
Still facing the problem that the server goes to OOM at some point (after paging has started some time before). Heap dump reveals 4GB of "org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl" instances which cannot be garbage collected somehow. According to GC roots there are millions of maps of "consumedPages" hold in these instances:
private final SortedMap<Long, PageCursorInfo> consumedPages = new TreeMap<Long, PageCursorInfo>();
Any idea why these objects cannot be garbage collected suddenly?
One thing which makes me suspicious: how does the server detect that it can stop paging? It seems that once paging has started it never stops although queues have been emptied after the consumer failures. It is always touching files in the paging directories:
104,0kiB 000117267.page
104,0kiB 000117268.page
72,0kiB 000117269.page
8,0kiB address.txt
Is a restart required for paging to stop?
-
20. Re: Memory problems in paging mode makes server unresponsive
Hello,
Did you get any clue or fix on the OOM with millions of consumedPages? I am getting exactly the same issue on EAP 6.2
-
21. Re: Memory problems in paging mode makes server unresponsive
Nick: the issue here is that when you hold transactions, it will also hold page-tx for these objects.
That's something we may change in a near future. but when you use transactions.. page can't be infinite.
We have a temporary cache for read messages... so we hold some SoftCache for read messages.
-
22. Re: Re: Memory problems in paging mode makes server unresponsive
Hi Clebert,
Thanks for your reply. However I'm not quite sure what you are saying.
Firstly let me explain what I am seeing in a bit more detail:
I am using EAP 6.2.
I have HornetQ deployed in a single JBoss instance acting as a separate message server. I have a couple of other Jboss instances referring to queues defined locally which are looked up via a remote netty connector onto the message server (this took me a while to work out how to setup but thanks to some of your other posts and some help from Redhat support I managed to get this working).
I have some JMS producer code running in JBoss instances which look up these queues (3 of them) and post messages to them. The queues are all consumed by MDBs.
The 3 queues are:
- index: drives Hibernate Search indexing MDB (this is on a separate server to the server producing the messages)
- job: drives an MDB that processes jobs (this is on a separate server to the server producing the messages)
- event: drives an event handler MDB (this is on the same server that produces the messages)
If I stop the index MDB server, stop the job MDB server and stop delivery the event handler MDB (I can't stop the server as this is the server producing the messages) and the application runs it produces messages for all 3 queues. Things happen as I would expect: looking at stats in JMS I see the message count climb for all 3 queues and after a while they start paging to disk.
When I start the other 2 MDB servers and start delivery for the event handler MDB I see the message count go down as the MDBs consume from the queue. The page count for the job and index queues drops to zero, BUT the page count for the event queue doesn't drop and continues to rise if more messages are produced (even though the message count remains at 0 or 1 as the messages are now being consumed).
I have all 3 MDBs configured the same - the only differences I can think of is that the message size for the events is possibly bigger than the other 2 queues and the producer/consumer are on the same Jboss instance.
A related scenario is if I start with no messages and no pages for the event queue and have the MDB consuming messages. Here if I start producing messages for a a few minutes the message count remains 0 or 1 as they are being consumed quickly and the page count remains zero. Then after a few minutes of running for some unknown reason (nothing seen in logs) the page count starts climbing for this queue only - although the message count remains 0 or 1 as the messages are still being consumed.
The thing you mention regarding transactions was what I had been thinking about - it does look like if I restart the message server it decides that the messages are no longer relevant and discards them (and the page count drops back to zero).
I am completely stumped as to what is causing this - mainly as I say that it seems to work fine for my other two queues and that it does for a while seem to run normally before suddenly and without a visible cause the page count spikes upwards.
Can you suggest anything I should look at to try and debug this further? I'm not sure how to work out why HornetQ thinks it should hang on to these messages. Is it possible that I have confused usage of transactions if the producer/consumer are on the same JBoss instance which is making use of the remote message server?
I have attached Jboss server config for the standalone message server and the server which is producing and consuming (via MDB) the event messages.
The event MDB is annotated with:
@MessageDriven(name = "EventListenerBean", activationConfig =
{
@ActivationConfigProperty(propertyName="destinationType", propertyValue = "javax.jms.Queue"),
@ActivationConfigProperty(propertyName="destination", propertyValue = "EventQueue"),
@ActivationConfigProperty(propertyName="maxSession", propertyValue = "50")
})
@Pool(value="mdb-strict-max-pool")
@TransactionManagement(TransactionManagementType.CONTAINER)
@TransactionAttribute(TransactionAttributeType.REQUIRED)
-
23. Re: Memory problems in paging mode makes server unresponsive
Marek,
we are using wildlfy 10 with artenis active mq. we are getting OOM and linux kernal is killing wildfly server. In the heap dump we are seeing lot of consumedPages in memory. Below is out configuration.
<address-setting name="#"
message-counter-history-day-limit="10" page-size-bytes="2097152"
max-size-bytes="10485760" redelivery-delay="2000" max-redelivery-delay="5000" redelivery-multiplier="2" max-delivery-attempts="5"/>
-
24. Re: Memory problems in paging mode makes server unresponsive
Are you acknowledging your messages when you consume from paging? otherwise they will build up as consumedPages.
-
25. Re: Memory problems in paging mode makes server unresponsive
we are creating Sessions with Session.AUTO_ACKNOWLEDGE as false
and delivery mode to be PERSISTENT. do we still have to ackowledge. One more thing on mdb
we mark it onMessage with @TransactionAttribute( TransactionAttributeType.NOT_SUPPORTED ) but we make a call to an stateless ejb
with @TransactionAttribute( TransactionAttributeType.REQUIRES_NEW ) .
-
26. Re: Memory problems in paging mode makes server unresponsive
sorry To Your questions we are not acknowledging the messages . Do we have to ??
-
27. Re: Memory problems in paging mode makes server unresponsive
You really ought to start a new forum thread rather than commenting on this one (which is several years old now). Explain your full use-case, the problem you believe you are observing, and what you expect to actually happen instead.
-
29. Re: Memory problems in paging mode makes server unresponsive
Justin,
I have created a new discussion for the issue. This resembles pretty much same as this issue. Below is the link.Please let me know what more information is required..
shiva