1 2 Previous Next 20 Replies Latest reply on Mar 3, 2016 9:14 AM by jbertram

hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

ericxiaomingzhao May 22, 2015 3:23 AM

Hello,

I got one problem that after the application process run for some days, the CPU/memory utilization will increase. use jmap dump and found that there are many (>65000) ServerMessageImpl objects existed in memory. this will cause out of memory after some days.

in our application, we set the ackMode to AUTO ACKNOWLEDGED like the following line:

topicSession = topicConnection.createSession(transacted,ackMode);

We didn't set the pre-acknowledge in hornetq-jms.xml like what the manual said.

Should we set to pre-acknowledged?

we found normal case, the message can be acknowledged(no ServerMessageImpl object existed in memory). but for unknown cases, the message can't be acknowledged(many serverMessageImpl object existed in memory).

Can someone help me out? what could be the possible reason to cause this issue?

1. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

jbertram May 22, 2015 10:20 AM (in response to ericxiaomingzhao)

What value are you using for the transacted variable when you create your session?
Actions
2. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

ericxiaomingzhao May 24, 2015 9:32 PM (in response to jbertram)

Hello Justin,

Thanks for your reply!
the transacted value is "false". is it correct for AUTO-ACKNOWLEDGE mode, or I have to use the PRE-ACKNOWLEDGE Mode?

Best Regards,
Eric.
Actions
3. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

jbertram May 26, 2015 9:59 AM (in response to ericxiaomingzhao)

is it correct for AUTO-ACKNOWLEDGE mode, or I have to use the PRE-ACKNOWLEDGE Mode?

Yes, it is proper to use AUTO-ACKNOWLEDGE mode.

Can you provide a reproducible test-case for this scenario?
Actions
4. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

ericxiaomingzhao May 26, 2015 9:16 PM (in response to jbertram)

Hello Justin,

thanks for your confirm, currently we don't know in which case we can reproduce this problem. we tried to send JMS message in our lab for example 1 message/second in 12 hours. then do jmap, but didn't get the problem. so it could be some error case scneario. since this was a customer site issue, we didn't know the scenario.
Actions
5. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

jbertram May 29, 2015 10:01 AM (in response to ericxiaomingzhao)

Message acknowledgment is tested quite thoroughly so in my opinion it's critical to have a clear description of the problematic use-case (ideally in the form of a reproducible test-case). At this point my guess is that the messages are simply aren't being removed because of some kind of bug in the application (i.e. not in HornetQ). If you can demonstrate otherwise I'd be happy to investigate further.
Actions
6. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

ericxiaomingzhao Jun 10, 2015 3:35 AM (in response to jbertram)

Justin, sorry for the late response.
We trying in our lab in last two weeks, and got the problem again, We suspected when large number of JMS message send to hornetq (and no consumer (PC GUI) connected), is it possible that hornetq store large number of JMS events and were not released.

our TTL is set to 20mins, is it possible TTL hanlding doesn't work, do we have anyway to check it?
where the JMS events was stored in hornetq? how to check and verify if it's the size was full?

Thanks in advance for your help.

Best Regards,
Eric.
Actions
7. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

jbertram Jun 11, 2015 12:55 PM (in response to ericxiaomingzhao)

our TTL is set to 20mins, is it possible TTL hanlding doesn't work, do we have anyway to check it?

If you have an expiry address set then you can look at any queue(s) bound to the expiry address to see the expired messages. Otherwise you can keep an eye on the message count for the queue(s) in question.

where the JMS events was stored in hornetq?

I'm not sure what you're asking about the storage of "JMS events." Can you clarify your question here?

how to check and verify if it's the size was full?

I'm not clear on this question either. How do you check if the size of what is full?

The bottom line here is...If you're pushing a high volume of messages to a server for 20 minutes with no consumers you're very likely to have memory issues.
Actions
8. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

ericxiaomingzhao Sep 6, 2015 2:31 AM (in response to jbertram)

Dear Justin,

We hit the problem on customer labs, finally we got the chance to dump the heap when out of memory.

our mx size was set to 2048M, but in heap dump, hornetq occupied almost of all the memory(92.63%), see the attachment.

in hornetq-configuration.xml, we have the following setting:
   <address-settings>
      
      <address-setting match="#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>10485760</max-size-bytes>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
         <address-full-policy>BLOCK</address-full-policy>
      </address-setting>
   </address-settings>

how to resolve this kind of issue? should we use PAGE instead of BLOCK?
does the message still existed after expired or in dead letter? the TTL of message is set to 15 mins when publishing the message.
Actions
9. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

jbertram Sep 7, 2015 1:24 PM (in response to ericxiaomingzhao)

Do you have a test-case that I can use to reproduce this?
Actions
10. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

ericxiaomingzhao Sep 8, 2015 4:59 AM (in response to jbertram)

Justin,

I am using JBoss 6.1.0 Final which includes Hornetq 2.2.5 Final version.
today we tried the following cases:
1). there are client connected to the server. the heap dump has zero number ServerMesageImpl object.
2). no client connected to the server. the heap dump has zero number ServerMesasgeImpl object.

then I tried to kill the client from task manager after the client connected to the server and found ServerMessageImpl object keep increasing in different heap dump.

Do you know which version I can upgrade to fix this issue?

Thanks,
Eric.
Actions
11. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

jbertram Sep 8, 2015 5:30 PM (in response to ericxiaomingzhao)

then I tried to kill the client from task manager after the client connected to the server and found ServerMessageImpl object keep increasing in different heap dump.

How long did you inspect the server to check the message count? If a client creates a non-durable subscription and then it gets disconnected for whatever reason (e.g. it crashes, network interruption, etc.) without closing that subscription then that subscription will remain until the TTL for the connection elapses and the server closes the connection and removes the subscription. Did you inspect the message count on the server both before and after the connection TTL elapsed?
Actions
12. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

ericxiaomingzhao Sep 8, 2015 11:32 PM (in response to jbertram)

Justin,

Thanks for your reply,

in JBOSS6.1.0 Final version, I only find ConnectionTTL in deplooy/jms-ra.rar/META-INF/ra.xml, it's commented out. we tried to set this value into 10min. seems no improvement. the problem still exists.


      <config-property>
        <description>The connection TTL</description>
        <config-property-name>ConnectionTTL</config-property-name>
        <config-property-type>java.lang.Long</config-property-type>
        <config-property-value></config-property-value>
      </config-property>

in hornetq-jms.xml. the connection-ttl is -1, that means the connection always there. but I remembered when we doing the development. it have some problem if it's not -1.

        <connection-factory name="NettyConnectionFactory">
                <xa>true</xa>
                <connectors>
                <connector-ref connector-name="netty"/>
                </connectors>
                <entries>
                        <entry name="/ConnectionFactory"/>
                        <entry name="/XAConnectionFactory"/>
                </entries>
                <connection-ttl>-1</connection-ttl>
        </connection-factory>

Is it related with one the above parameter?
Actions
13. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

jbertram Sep 9, 2015 9:09 AM (in response to ericxiaomingzhao)

in hornetq-jms.xml. the connection-ttl is -1, that means the connection always there. but I remembered when we doing the development. it have some problem if it's not -1.

        <connection-factory name="NettyConnectionFactory">

                <xa>true</xa>

                <connectors>n

                <connector-ref connector-name="netty"/>

                </connectors>

                <entries>

                        <entry name="/ConnectionFactory"/>

                        <entry name="/XAConnectionFactory"/>

                </entries>

                <connection-ttl>-1</connection-ttl>

        </connection-factory>

Is it related with one the above parameter?

Are the clients in your test using the NettyConnectionFactory? If so, that would explain the problem. A connection-ttl of -1 should never be used for connections that could potentially break as it could lead to message accumulation and/or connection accumulation.
Actions
14. Re: hornetq - many ServerMessageImpl objects existed in memory and not acknowledged

ericxiaomingzhao Sep 10, 2015 4:27 AM (in response to jbertram)

Thanks Justin,

We changed the connection-ttl into 1hour (3600000ms). it worked with this change.

One more question:
If No consumer connected, how hornetq handle the server message from Application side, seems it's discarded, right? since when I dumped, there is no ServerMessageImpl object in heap dump.

Best Regards,
Eric.
Actions

1 2 Previous Next

Go to original post