8 Replies Latest reply on Jun 18, 2011 10:00 AM by leosbitto

Memory Leak in ServerLocator.factories?

greyfairer2 Jun 16, 2011 9:19 AM

Hi all,

We use HornetQ using JMS API in a web application.

When a message needs to be sent, we use:

con = connectionFactory.createConnection();

con.start():

ses = con.createSession( false, Session.CLIENT_ACKNOWLEDGE );

producer=ses.createProducer(new HornetQQueue(name))

msg = ses.createObjectMessage(...);

producer.send(msg);

producer.close();

ses.close();

con.close();

After a few days running and sending lots of messages like this, we get an OutOfMemoryError.

Inspecting the heap dump, we see 200MB retained heap in 100.000 instances of ClientSessionFactoryImpl in the ServerLocator factories map.

If I see it correctly, everytime a HornetQConnection is created, a linked sessionFactory is created. This sessionFactory is created by the ServerLocator in the ConnectionFactory, and retained in the factories map.

When the connection is closed, the attached sessionFactory is closed as well, but it is not removed from the serverLocator's factories map.

The sessionFactory does have a link to the serverLocator, and ServerLocator has a method 'factoryClosed', but this method is not used.

Did you forget this?

1. Re: Memory Leak in ServerLocator.factories?

ataylor Jun 16, 2011 9:24 AM (in response to greyfairer2)

which version of HornetQ are you using, does this still occur with trunk.

Also, looks like your not using the correct factory, in JEE you should use the pooled factory, i.e. java:JmsXA
Actions
2. Re: Memory Leak in ServerLocator.factories?

greyfairer2 Jun 16, 2011 12:35 PM (in response to ataylor)
Sorry, forgot to mention the version. We have this in HornetQ 2.2.2.Final and also with yesterday's version of Branch_2_2.
We didn't try any other versions, but it seems that ServerLocator.factoryClosed() isn't called anywhere.

We're connecting to a remote server, using a remote JNDI lookup of '/ConnectionFactory'. Can you use a pooled factory remote?

It's used in a spring DefaultMessageListenerContainer managed with Spring managed transactions.

http://static.springsource.org/spring/docs/2.0.x/api/org/springframework/jms/listener/DefaultMessageListenerContainer.html
Note that this listener container will automatically reobtain all JMS handles for each transaction in case of an external transaction manager specified, for compatibility with all J2EE servers (in particular JBoss).

So with a default receiveTimeout of 1 second, this creates and closes a new connection every second.
If you close a connection, the attached clientSessionFactory should be removed from the ServerLocator automatically, I'd guess.

Greets, Geert.

Message was edited by: Geert Pante
Actions
3. Re: Memory Leak in ServerLocator.factories?

ataylor Jun 16, 2011 1:15 PM (in response to greyfairer2)

We're connecting to a remote server, using a remote JNDI lookup of '/ConnectionFactory'. Can you use a pooled factory remote?
Yes, just configure the resource adapter, take a look at some of the jca examples.

So with a default receiveTimeout of 1 second, this creates and closes a new connection every second.
If you close a connection, the attached clientSessionFactory should be removed from the ServerLocator automatically, I'd guess.
creating a new connection for every message sent/received is an anti pattern, connections are designed to be long lived objects, possible memory leak aside your going to get poor peformance and possible memory problems. This is why JCA is used, it pools connections to save you creating lots of new ones.
Actions
4. Re: Memory Leak in ServerLocator.factories?

clebert.suconic Jun 16, 2011 8:00 PM (in response to ataylor)

I'm fixing the factoryClosed call.

If you open a JIRA.. I will close it as soon as I commit it.

However, you shouldn't be creating that many sessionFactories anyway. you probably need a pool as Andy mentioned.
Actions
5. Re: Memory Leak in ServerLocator.factories?

greyfairer2 Jun 17, 2011 12:27 PM (in response to clebert.suconic)

OK, see HORNETQ-724

And indeed in DefaultMessageListenerContainer, you can make it cache the Connection, so that should solve it, too.
Actions
6. Re: Memory Leak in ServerLocator.factories?

leosbitto Jun 18, 2011 7:18 AM (in response to greyfairer2)

Geert Pante wrote:

And indeed in DefaultMessageListenerContainer, you can make it cache the Connection, so that should solve it, too.

The problem is that if you configure DefaultMessageListenerContainer to cache the Connection and you use the JCA ConnectionFactory, the resulting Session might not be enlisted in the XA transaction properly. I have solved this by writing my own ConnectionFactory, which wraps the real XAConnectionFactory from the JMS provider, and creates Connections which wrap the real XAConnections and handle the caching and enlisting of the Sessions to the XA transactions.
Actions
7. Re: Memory Leak in ServerLocator.factories?

ataylor Jun 18, 2011 8:48 AM (in response to leosbitto)

If you use the jca connection factory you dont have to cache it, its pooled anyway.

Im not sure what spring does bit the jca layer will automatically add the created session to the current transaction.
Actions
8. Re: Memory Leak in ServerLocator.factories?

leosbitto Jun 18, 2011 10:00 AM (in response to ataylor)

Andy Taylor wrote:

If you use the jca connection factory you dont have to cache it, its pooled anyway.

Im not sure what spring does bit the jca layer will automatically add the created session to the current transaction.
Spring's DefaultMessageListenerContainer only starts and ends the XA transaction (when configured to use the class org.springframework.transaction.jta.JtaTransactionManager), it does not enlist the XAResource (from XASession#getXAResource()) to the running transaction - it relies on the JCA ConnectionFactory to do this. However, at least with JBoss 4.3 EAP (that's what we have used - not tried EAP 5.x yet), when the client does not call ConnectionFactory#createConnection (and caches the Connection instead), the user JMS Session does not get enlisted to the running XA transaction. That's why I have added my comment to Geert Pante's comment "in DefaultMessageListenerContainer, you can make it cache the Connection".

Another reason why I wrote the caching myself is that I wanted to have full control over assigning of the cached Sessions with MessageConsumers to my threads, to avoid the situation when some MessageConsumers (with messages pushed from the JMS server to their client-side cache) were idling in the cache of the JCA layer and therefore adding unwanted latency to the processing of the messages.
Actions

Go to original post