-
1. Re: Low memory warning
timfox Sep 13, 2009 4:18 AM (in response to nbhatia)Yes, this is a new warning in beta5.
-
2. Re: Low memory warning
nbhatia Sep 13, 2009 5:31 PM (in response to nbhatia)I can see the JVM heap expand with time, even on a completely quiet system - so I definitely have a memory leak. I did a heap dump after about 1 hour of operation and analyzed it with Eclipse MAT. The "Leak Suspects" report is as follows:
Suspect 1: org.hornetq.core.remoting.impl.RemotingConnectionImpl - 16 instances - 258M (20.56%) Suspect 2: org.hornetq.core.client.impl.ClientSessionImpl - 47,839 instances - 240M (19.16%) Suspect 3: org.hornetq.core.server.impl.ServerSessionImpl - 47,839 instances - 148M (11.75%) Suspect 4: org.hornetq.jms.client.HornetQConnection - 23,921 instances - 132M (10.55%) --------- Total memory occupied by 4 suspects = 778M (62.02%)
Is there any clue here that I should go after? Are 47K sessions and 23K connections suspect?
To give you some context, this is a Spring-based web application (WAR packaged) running on JBoss AS 5.1. HornetQ is also running on the same instance of JBoss. I have configured spring to use a pool of 16 MessageListeners. The listeners themselves are using the JmsXA connectionfactory to get connections. So, if I understand correctly, connection pooling is being mangaged by the JCA adapter and is very efficient:<bean id="customerListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer"> <property name="connectionFactory" ref="jmsConnectionFactory" /> <property name="destination" ref="customerQueue" /> <property name="messageListener" ref="customerListener" /> <property name="concurrentConsumers" value="15" /> <property name="transactionManager" ref="transactionManager" /> </bean> <bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean"> <property name="jndiName" value="java:/JmsXA" /> </bean>
Thanks.
Naresh -
3. Re: Low memory warning
clebert.suconic Sep 13, 2009 11:05 PM (in response to nbhatia)Are you sure you are closing your connections?
-
4. Re: Low memory warning
nbhatia Sep 14, 2009 12:01 AM (in response to nbhatia)Since connections are managed by Spring, I have no control over opening/closing them. All I really specify is the customerListenerContainer as mentioned in my previous post and messages start flowing into my MessageListener.
(My app also sends messages to queues but that capability was not exercised in the measurement above. Even when I do send messages, I use the JmsTemplate provided by spring - I just tell it to send a message and it manages the connection. Again, I do not explicitly open/close a connection.) -
5. Re: Low memory warning
timfox Sep 14, 2009 4:58 AM (in response to nbhatia)If you're not closing it, then the spring code isn't closing it.
We really recommend you don't use the Spring JMSTemplate! It is a source of many headaches for us since it uses several anti-patterns and encourages a very poor use and understanding of the JMS API
Even the ActiveMQ guys have something to say about it here http://activemq.apache.org/jmstemplate-gotchas.html
I would go further, and say the Spring JMSTemplate is *never* appropriate for consuming messages, and can only be used safely for sending messages but only if you're using a JCA adapter. Even then though I'd recommend just coding against the JMS API yourself, since at the end of the day you'll probably have fewer problems.
If you can replicate a connection leak without using the Spring JMSTemplate, that's fine and we'll investigate. But we're not going to debug tons of Spring code just to find out they're doing another dumb thing! ;) -
6. Re: Low memory warning
nbhatia Sep 14, 2009 7:58 AM (in response to nbhatia)Fair enough :-). I will remove dependency on JmsTemplate and see if that takes away the leak.
-
7. Re: Low memory warning
timfox Sep 18, 2009 1:39 PM (in response to nbhatia)I've just created this wiki page too:
http://www.jboss.org/community/wiki/CanIusetheSpringJMSTemplatewithHornetQ -
8. Re: Low memory warning
nbhatia Sep 18, 2009 2:59 PM (in response to nbhatia)Thanks Tim.
FYI, JmsTemplate is generally used for send only. Calls to receive JMS messages, as offered by JmsTemplate, are always synchronous - so not very attractive. Spring developers generally use the MessageListenerContainer to receive messages. Originally, I thought that this would be a mechanism to receive messages asynchronously, but going through the code, it appears that the JMS provider is being polled at regular intervals and the messages are still being received synchronously! There are too many layers of abstraction to figure out exactly what is going on, nonetheless, as mentioned in my earlier post, the memory leak is real.
I got too wound up in other things, but the next thing I will try is to create a simple pool of listeners (not using Spring) that will behave correctly with the JMS provider (unless somebody else has done this already - please let me know). One thing I don't want to do is to convert my application into an EAR just to use MDBs (that has severely increased my dev-test cycle on other applications). -
9. Re: Low memory warning
clebert.suconic Sep 18, 2009 3:35 PM (in response to nbhatia)"nbhatia" wrote:
as mentioned in my earlier post, the memory leak is real.
so far it seems the leak is on the JMSTemplate. I have personally tested a lot of scenarios with closing connections and leaks.
I'm not saying it's impossible to have something wrong. But if you find an issue on HornetQ we will fix it right away. We have no way to fix the JMSTemplate.One thing I don't want to do is to convert my application into an EAR just to use MDBs (that has severely increased my dev-test cycle on other applications).
You don't need that.. you could just use regular MessageListeners. -
10. Re: Low memory warning
timfox Sep 18, 2009 3:47 PM (in response to nbhatia)"nbhatia" wrote:
FYI, JmsTemplate is generally used for send only.
Where are you using the JMSTemplate? Inside the application server or outside of it?
If inside, what connection factory are you looking up? java:/JmsXA or some other one? -
11. Re: Low memory warning
nbhatia Sep 18, 2009 5:12 PM (in response to nbhatia)JmsTemplate is being used inside the application server. Both JmsTempate and MessageListenerContainer are using the java:/JmsXA connection factory. The Spring configuration below:
<bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate"> <property name="connectionFactory" ref="jmsConnectionFactory" /> </bean> <bean id="customerListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer"> <property name="connectionFactory" ref="jmsConnectionFactory" /> <property name="destination" ref="customerQueue" /> <property name="messageListener" ref="customerListener" /> <property name="concurrentConsumers" value="15" /> <property name="transactionManager" ref="transactionManager" /> </bean> <bean id="jmsConnectionFactory" class="org.springframework.jndi.JndiObjectFactoryBean"> <property name="jndiName" value="java:/JmsXA" /> </bean>
-
12. Re: Low memory warning
timfox Sep 19, 2009 7:57 AM (in response to nbhatia)Ok, so as Clebert mentioned, you need to replicate the issue without using Spring.
If there is an issue then, we, as always will be more than happy to diagnose and fix it, if necessary, but we're not going to debug Spring code. -
13. Re: Low memory warning
clebert.suconic Sep 19, 2009 11:31 PM (in response to nbhatia)@nbhatia: I just did a quick test with MDBs... it looks like there is a connection leak on JCA. I've run some MDB examples (no spring code involved) and it leaked connections.
Let me do some work around this on monday.. and I will get back to you through this thread. Thanks for raising the issue. -