Certain JMS topics just stop responding
All,
I'm using a standalone instance of HornetQ 2.2.5 in RHEL in a non-J2EE environment. I'm having a problem where after a certain period of time (could be days, could be a week or more) clients are unable to publish to a JMS topic (our application has 40+ JMS topics defined in the hornetq-jms.xml file accessible via JNDI). When this happens, however, often other clients have no problem publishing to other JMS topics (i.e. the whole HornetQ server doesn't become hosed). When the problem develops the clients sending to the topic in question end up blocking on the call messageProducer.send(objMessage). With the current client failure check set to 5 minutes I see the client unable to publish for approximately 5 minutes (I have a timeout when trying to publish) and I then see this error:
ERROR [2012-02-12 14:19:03,693] [Thread-2] [protocol.jms.lib.ATMSContext$ReconnectExceptionListener] -- Received JMS connection exception [errorCode: DISCONNECT]: javax.jms.JMSException: HornetQException[errorCode=3 message=Did not recei
ve data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@686490[local= /172.30.1.11:60603, remote=/172.30.1.10:10049]]
javax.jms.JMSException: HornetQException[errorCode=3 message=Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@686490[local= /172.30.1.11:60603, remote=/172.30.1.10:10049]]
at org.hornetq.jms.client.HornetQConnection$JMSFailureListener.connectionFailed(HornetQConnection.java:643)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:818)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:605)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable$1.run(ClientSessionFactoryImpl.java:1376)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: HornetQException[errorCode=3 message=Did not receive data from server for org.hornetq.core.remoting.impl.netty.NettyConnection@686490[local= /172.30.1.11:60603, remote=/172.30.1.10:10049]]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$PingRunnable.run(ClientSessionFactoryImpl.java:1366)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$ActualScheduledPinger.run(ClientSessionFactoryImpl.java:1337)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(Unknown Source)
at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(Unknown Source)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
... 3 more
WARN [2012-02-12 14:19:03,694] [Thread-2] [protocol.jms.lib.ATMSContext] -- Reconnecting...
Our particular application has no requirements for persistence so I've disabled all persistence via the <persistence-enabled>false</persistence-enabled>. I've also explicitly not defined a dead-letter-address or expiry-address so that these messages are (or should be) just dropped. Our requirements don't need all messages to be delivered so I've set the address-full-policy to DROP.
Currently we're using ObjectMessages when publishing (we at times tend to publish rather long arrays of Serializable objects) although the objects that do get published aren't overly complicated (just beans that are Serializable).
As an example here's the code we currently use to publish to a particular JMS topic:
/**
* Attempt to publish a message to the given JMS topic.
*
* @param jmsTopic
* @param messageName
* @param obj object to publish
* @throws JMSException
* @throws RemoteException
* @throws NamingException
*/
public synchronized static final void publishMessage(String jmsTopic, String messageName,
Serializable obj) throws JMSException, RemoteException, NamingException {
logger.debug("Publishing [topic: " + jmsTopic + ", messageName: " + messageName + "]");
if (jmsTopic == null) {
logger.warn("Null topic...ignoring publish request");
return;
}
// use a cached version of the JMS session object
Session session = sessionMap.get(jmsTopic);
if (session == null) {
session = ATMSContext.getProducingSession();
sessionMap.put(jmsTopic, session);
}
// get the message producer, creating one lazily as needed
MessageProducer messageProducer = producerMap.get(jmsTopic);
if (messageProducer == null) {
messageProducer = session.createProducer(ATMSContext.getDestination(jmsTopic));
// set the message expiration time
messageProducer.setTimeToLive(messageTimeToLive);
producerMap.put(jmsTopic, messageProducer);
}
ObjectMessage objMessage = session.createObjectMessage(obj);
objMessage.setStringProperty(ATMSContext.MESSAGE_NAME, messageName);
messageProducer.send(objMessage);
logger.debug("Successfully published [topic: " + jmsTopic + ", messageName: " + messageName
+ "]");
}
I unfortunately don't have any test cases that will immeidatley reproduce the problem (which is frustrating on my end). I'm including our current configurations as a reference. Any suggestions as to why sometimes only particular JMS topics are getting hosed?
Thanks,
Dave
-
hornetq.zip 3.8 KB
