HornetQ 2.2.5 Final + Jboss 6.0.0 Final memory leak?

[bjornko]

Our server crashed with out of memory exception after I let a script push files to it using curl to a REST interface for file upload. The file upload does some processing then creates a message on a queue. As there are no sessions between the curls, a new is created each time, and I quickly found that they are not being expired as they should. From what i've gathered from jvisualvm, the culprit is HornetQ.

There were some confusion regarding wether I should cache connections, sessions and producers, but this page convinced me that I should. We're in an EJB environment but we're not using a JCA managed connection factory.

Now if I didn't cache the session, I was leaking connection factories. When I wasn't caching producers, I was leaking ClientSessionImpl. Now I'm caching both, but I'm still accumulating ServerMessageImpl, though all queues are empty.

jvisualvm claims hornetq's are the largest objects.

Any help would be greatly appreciated.

[ataylor]

its hard to say with out seeing how you are caching your JMS resources, however i must ask why you arent using the JCA managed connection factory, its what you should be using.

[bjornko]

All my JMS resources are held in Threadlocals. I dispose of them if they start misbehaving and create new ones as needed. According to jvisualvm the retained size of my QueueManager is low.

I'll try to reconfigure things to use JCA. I didn't set up HornetQ so I don't know why things are the way they are.

[ataylor]

All my JMS resources are held in Threadlocals. I dispose of them if they start misbehaving and create new ones as needed. According to jvisualvm the retained size of my QueueManager is low.

That seems wrong to me, Typically app servers use thread pools so to avoid resources bleeding from on operation/invocation to another you should always clean them up at the end of the thread invocation, if you are cleaning them up then theres no caching. You should use the JCA, thats what its there for, to give you caching amongst other things

[bjornko]

Maybe I was mistaken.

Our configuration mirrors the example in hornetq-2.2.5.Final/examples/jca-config/server

It sets up a connection factory like this:

   <connection-factory name="ConnectionFactory">
      <connectors>
         <connector-ref connector-name="netty"/>
      </connectors>
      <entries>
         <entry name="ConnectionFactory"/>
         <entry name="XAConnectionFactory"/>
      </entries>
   </connection-factory>

Which i specifiy as resource in my QueueManager. Does that mean I am using JCA, and if so, why is it leaking? I leak wether I cache my JMS resources or not, but significantly more if I don't.

I tried specifying java:/JmsXA as connection manager, but that resulted in this, when trying to send a message. cause=InvalidDestinationException(Not a HornetQ Destination:HornetQQueue[ImportQueue-10])

[bjornko]

So I'm using JCA. I get hold of the ConnectionFactory like this:

   @Resource(mappedName = "/ConnectionFactory")
  ConnectionFactory connectionFactory;

And I send the message like this:

try {
     connection = connectionFactory.createConnection();
     session = connection.createSession( false, Session.AUTO_ACKNOWLEDGE );
     producer = session.createProducer( HornetQJMSClient.createQueue( name ) );
     Message message = session.createObjectMessage( holder );
     producer.send( message );
} finally {
     producer.close();
     session.close();
     connection.close();
}     

I'm leaving out some exception handling but you get the idea.

By not caching anything and closing everything, I'm finding that I'm leaking a lot less, and a lot fewer object types. Sessions time out properly. However, I'm still leaking

org.hornetq.utils.ClientSessionFactoryImpl. (Also NettyConnectionFactory, and OrderedExecutorFactory).

Looking at heap dumps I see that a lot of ClientSessionFactoryImpls have closed=true, but they are not getting GCed.

[ataylor]

no you arent using the JCA, you need to use the java:/JmsXA connection factory, fyi this is configured in jms-ds.xml file.

If its not working then your code/config is wrong, take a look at one of the javaee examples shipped with HornetQ

[bjornko]

Thanks for your help, Andy. I was able to figure it out eventually. The InvalidDestinationException when using JmsXA should've been a hint to google.

One of my hornetq maven dependencies (hornetq-jms) was not scoped as provided, so instead of using the bundled hornetq libs that come with jboss, I got a mix, with predictably unpredictable results.

After specifiying JmsXA as the ConnectionFactory and fixing the dependency there are no leaks.