MessageReferenceImpl Memory Leak

[mackerman]

Hi, we have a dual server JBoss 7.1.1.Final system, using hornetq 2.2.14.Final for inter-server communication and event notification consumed via STOMP.  After hitting the system (it's a SIP server of sorts) pretty hard for a while, we see a memory leak on one of the servers.  If allowed to continue, the system locks up and spends all its time doing GC. As far as I can tell most, if not all, of the MessageReferenceImpl objects are for the topic (see below for configuration), though it's hard to tell with that many messages.  When a message is send to a topic by a server it will send it to the topic on both servers in the cluster - we do not make use of hornetQ clustering.  A .NET client will consume messages via STOMP from one, but not both, of the servers (via a load-balanced address).

Does anyone have any suggestions as to what the problem might be?

The instances we are seeing growing are:

thats 1.5G of data consumed by instances of those 2 classes!

our horneq-q.xml includes:

    <topic name="asyncEvents">

        <entry name="/topic/asyncEvents"/>

    </topic>

    <queue name="interServerQueue">

        <entry name="/queue/interServerQueue"/>

        <durable>false</durable>

    </queue>

Connections, Sessions and Producers are created like this:

HornetQConnectionFactory cf = HornetQJMSClient.createConnectionFactoryWithoutHA(getFactoryType(), getTransportConfiguration());

connection = cf.createConnection();

connection.start();

session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

destination = HornetQDestination.fromAddress(destinationName);

producer = session.createProducer(destination);

// set time to live for topics, queued messages should all be consumed

if (JMSFactoryType.TOPIC_CF.equals(getFactoryType()))

          producer.setTimeToLive(30000); 

producer.setDisableMessageID(true);

producer.setDisableMessageTimestamp(true);

Messages are sent like this:

TextMessage message = session.createTextMessage();

message.setStringProperty(org.hornetq.rest.HttpHeaderProperty.CONTENT_TYPE,

                    MediaType.APPLICATION_JSON);

message.setText(strmsg);

message.setStringProperty(propName, propValue);

try {

          producer.send(message);

          if (session.getTransacted())

                    session.commit();

} catch (Exception e) {

          log.error(...);

}

The system runs on CentOS 6.3, JBoss 7.1.1.Final, HornetQ 2.2.14.Final, JRE 1.6.0_30.

Thanks, Mitchell

[ataylor]

are you saying that you arent consuming messages from the server that locks up, if so, have you configured paging?

[mackerman]

It looks like the topic messages on the server that has a STOMP client attached (actually 2 clients) are not being cleaned up.  Topic messages only have a timeToLive of 30 seconds, so they should all have been cleaned up at some point, whether of not a subscriber received the messages.  Even after leaving the system running overnight with no load, the heap is not getting cleaned up.

I'm not really interested in paging messages to disk.  If there is no topic subscriber I just want the messages to expire and be removed.  Messsages in the queue should all be consumed in short order.