Connection creation fails in hornetq 2.3.0

[prakruthi.v]

I am using hornetq 2.3.0 version in a standalone non-clustered mode and I am seeing this kind of error when creating a queue connection.  This problem happens rarely and randomly when there are a lot of messages processing at the same time. It works well otherwise.

Code snippet to create a connection :

this.qName = qName.getQName();

qConnectionFactory = getQConnectionFactory();

qConnection = qConnectionFactory.createQueueConnection();

qSession = qConnection.createQueueSession(false, Session.AUTO_ACKNOWLEDGE);

public static QueueConnectionFactory getQConnectionFactory() throws NamingException {

   if (connectionFactory == null) {

      connectionFactory = (ConnectionFactory) jndiLookup("java:/XAConnectionFactory");

   }

   return (QueueConnectionFactory) connectionFactory;

}

Exception observed:

Caused by: javax.jms.JMSException: Failed to create session factory

      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:587)

      at org.hornetq.jms.client.HornetQConnectionFactory.createQueueConnection(HornetQConnectionFactory.java:119)

      at org.hornetq.jms.client.HornetQConnectionFactory.createQueueConnection(HornetQConnectionFactory.java:114)

      at com.framework.GenericQueue.<init>(GenericQueue.java:79)

      at com.framework.SendQueue.<init>(SendQueue.java:58)

      at com.framework.SendQueue.replyMessage(SendQueue.java:100)

      at com.networkcontrol.NetworkControlServiceImpl$QueueHandler.handleMessage(NetworkControlServiceImpl.java:167)

      ... 9 more

Caused by: org.hornetq.api.core.HornetQInterruptedException: java.lang.InterruptedException

      at org.hornetq.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:244)

      at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:267)

      at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:194)

      at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable.send(ClientSessionFactoryImpl.java:1765)

      at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1369)

      at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:1069)

      at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:246)

      at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:828)

      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:583)

      ... 15 more

Caused by: java.lang.InterruptedException

      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)

      at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)

      at org.hornetq.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:175)

      ... 23 more

Netty connector has been configured. Configuration in hornetq-configuration.xml:

<configuration xmlns="urn:hornetq"

               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

               xsi:schemaLocation="urn:hornetq /schema/hornetq-configuration.xsd">

        <persistence-enabled>false</persistence-enabled>

        <security-enabled>false</security-enabled>

        <journal-directory>/tmp/hornetq</journal-directory>

   <!-- Connectors -->

   <connectors>

      <connector name="netty-connector">

         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>

        <param key="host"  value="172.19.134.2"/>

      </connector>

   </connectors>

   <!-- Acceptors -->

   <acceptors>

      <acceptor name="netty-acceptor">

         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>

        <param key="host"  value="172.19.134.2"/>

      </acceptor>

   </acceptors>

</configuration>

Can somebody help on this?

[jbertram]

I want to say that was either a Netty bug or a HornetQ bug.  In any case, I think it would be resolved if you moved to the latest release.

[prakruthi.v]

I am unable to reproduce this problem consistently. Is there a way to ensure the latest hornetQ or netty will resolve this?

[jbertram]

I believe the problem would be resolved with an upgrade, but I'm not 100% sure.  If you can't reproduce the issue consistently in order to test it on both platforms then it's obviously hard to prove that way as well.

What is the use-case when you see this crop up?  Are you creating lots of connections in a short amount of time?

[prakruthi.v]

When a message is sent, a connection is created and the connection is closed after the message is sent.

Approximately 10 to 20 messages per second are processed, which means 10-20 connections are created and closed per second.

[ataylor]

opening a connection for every message is an anti pattern, connections (and sessions) should be long lived. (the caveat here is if you are using the JCA managed connection factory)

[jbertram]

As Andy noted, opening and closing a connection for every message send is a huge anti-pattern (unless you're using JCA).  This is where we saw this issue crop up before.  I believe you should simply be able to adjust your code not to use this anti-pattern and the problem will go away.

Aside from the anti-pattern, we fixed the underlying issue with a Netty upgrade.  If I recall correctly this was the commit that fixed it.

[prakruthi.v]

Thanks Andy and Justin.
To ensure that the fix works with a Netty upgrade, is there any way possible to arrive at this race condition and reproduce this HornetQInterruptedException?

[jbertram]

Do reproduce the exception reliably I think you'd need to use something like Byteman to somewhat artificially induce the race.