Low memory warning

[nbhatia]

I just switched from JBM2-Beta4 to HornetQ-Beta5 and I am starting to get the following warning message every few seconds:

18:33:07.515 WARNING [Thread-6] [org.hornetq.core.server.impl.MemoryManagerImpl] Less than 25%
free memory: 237.51 MiB
max memory: 950.69 MiB
total memory: 950.69 MiB
available memory: 24.98%

You are in danger of running out of RAM. Have you set paging parameters on your addresses? (See user manual "Paging" chapter)

Is this message something new in Beta5 or was it always there? Trying to figure out if I have introduced a memory leak somewhere.

[timfox]

Yes, this is a new warning in beta5.

[nbhatia]

I can see the JVM heap expand with time, even on a completely quiet system - so I definitely have a memory leak. I did a heap dump after about 1 hour of operation and analyzed it with Eclipse MAT. The "Leak Suspects" report is as follows:

Suspect 1: org.hornetq.core.remoting.impl.RemotingConnectionImpl - 16 instances - 258M (20.56%)
Suspect 2: org.hornetq.core.client.impl.ClientSessionImpl - 47,839 instances - 240M (19.16%)
Suspect 3: org.hornetq.core.server.impl.ServerSessionImpl - 47,839 instances - 148M (11.75%)
Suspect 4: org.hornetq.jms.client.HornetQConnection - 23,921 instances - 132M (10.55%)
---------
Total memory occupied by 4 suspects = 778M (62.02%)

Is there any clue here that I should go after? Are 47K sessions and 23K connections suspect?

To give you some context, this is a Spring-based web application (WAR packaged) running on JBoss AS 5.1. HornetQ is also running on the same instance of JBoss. I have configured spring to use a pool of 16 MessageListeners. The listeners themselves are using the JmsXA connectionfactory to get connections. So, if I understand correctly, connection pooling is being mangaged by the JCA adapter and is very efficient:

<bean id="customerListenerContainer"
 class="org.springframework.jms.listener.DefaultMessageListenerContainer">
 <property name="connectionFactory" ref="jmsConnectionFactory" />
 <property name="destination" ref="customerQueue" />
 <property name="messageListener" ref="customerListener" />
 <property name="concurrentConsumers" value="15" />
 <property name="transactionManager" ref="transactionManager" />
</bean>

<bean id="jmsConnectionFactory"
 class="org.springframework.jndi.JndiObjectFactoryBean">
 <property name="jndiName" value="java:/JmsXA" />
</bean>

Thanks.
Naresh

[clebert.suconic]

Are you sure you are closing your connections?

[nbhatia]

Since connections are managed by Spring, I have no control over opening/closing them. All I really specify is the customerListenerContainer as mentioned in my previous post and messages start flowing into my MessageListener.

(My app also sends messages to queues but that capability was not exercised in the measurement above. Even when I do send messages, I use the JmsTemplate provided by spring - I just tell it to send a message and it manages the connection. Again, I do not explicitly open/close a connection.)

[timfox]

If you're not closing it, then the spring code isn't closing it.

We really recommend you don't use the Spring JMSTemplate! It is a source of many headaches for us since it uses several anti-patterns and encourages a very poor use and understanding of the JMS API

Even the ActiveMQ guys have something to say about it here http://activemq.apache.org/jmstemplate-gotchas.html

I would go further, and say the Spring JMSTemplate is *never* appropriate for consuming messages, and can only be used safely for sending messages but only if you're using a JCA adapter. Even then though I'd recommend just coding against the JMS API yourself, since at the end of the day you'll probably have fewer problems.

If you can replicate a connection leak without using the Spring JMSTemplate, that's fine and we'll investigate. But we're not going to debug tons of Spring code just to find out they're doing another dumb thing! ;)

[nbhatia]

Fair enough :-). I will remove dependency on JmsTemplate and see if that takes away the leak.

[timfox]

I've just created this wiki page too:

http://www.jboss.org/community/wiki/CanIusetheSpringJMSTemplatewithHornetQ

[nbhatia]

Thanks Tim.

FYI, JmsTemplate is generally used for send only. Calls to receive JMS messages, as offered by JmsTemplate, are always synchronous - so not very attractive. Spring developers generally use the MessageListenerContainer to receive messages. Originally, I thought that this would be a mechanism to receive messages asynchronously, but going through the code, it appears that the JMS provider is being polled at regular intervals and the messages are still being received synchronously! There are too many layers of abstraction to figure out exactly what is going on, nonetheless, as mentioned in my earlier post, the memory leak is real.

I got too wound up in other things, but the next thing I will try is to create a simple pool of listeners (not using Spring) that will behave correctly with the JMS provider (unless somebody else has done this already - please let me know). One thing I don't want to do is to convert my application into an EAR just to use MDBs (that has severely increased my dev-test cycle on other applications).

[clebert.suconic]

"nbhatia" wrote:
as mentioned in my earlier post, the memory leak is real.

so far it seems the leak is on the JMSTemplate. I have personally tested a lot of scenarios with closing connections and leaks.

I'm not saying it's impossible to have something wrong. But if you find an issue on HornetQ we will fix it right away. We have no way to fix the JMSTemplate.

One thing I don't want to do is to convert my application into an EAR just to use MDBs (that has severely increased my dev-test cycle on other applications).

You don't need that.. you could just use regular MessageListeners.

[timfox]

"nbhatia" wrote:

FYI, JmsTemplate is generally used for send only.

Where are you using the JMSTemplate? Inside the application server or outside of it?

If inside, what connection factory are you looking up? java:/JmsXA or some other one?

[nbhatia]

JmsTemplate is being used inside the application server. Both JmsTempate and MessageListenerContainer are using the java:/JmsXA connection factory. The Spring configuration below:

<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate">
 <property name="connectionFactory" ref="jmsConnectionFactory" />
</bean>

<bean id="customerListenerContainer"
 class="org.springframework.jms.listener.DefaultMessageListenerContainer">
 <property name="connectionFactory" ref="jmsConnectionFactory" />
 <property name="destination" ref="customerQueue" />
 <property name="messageListener" ref="customerListener" />
 <property name="concurrentConsumers" value="15" />
 <property name="transactionManager" ref="transactionManager" />
</bean>

<bean id="jmsConnectionFactory"
 class="org.springframework.jndi.JndiObjectFactoryBean">
 <property name="jndiName" value="java:/JmsXA" />
</bean>

[timfox]

Ok, so as Clebert mentioned, you need to replicate the issue without using Spring.

If there is an issue then, we, as always will be more than happy to diagnose and fix it, if necessary, but we're not going to debug Spring code.

[clebert.suconic]

@nbhatia: I just did a quick test with MDBs... it looks like there is a connection leak on JCA. I've run some MDB examples (no spring code involved) and it leaked connections.

Let me do some work around this on monday.. and I will get back to you through this thread. Thanks for raising the issue.

[clebert.suconic]

https://jira.jboss.org/jira/browse/HORNETQ-139