Problem with HQ119014: Timed out waiting for response when sending packet 30

[pgrazek]

Hello,

I have a spring-boot application acting as a HornetQ server and ~10 client applications sending and receiving messages without any problem.

But there's one specific client that refuses to connect with the following exception:

org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: HQ119014: Timed out waiting for response when sending packet 30

  at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)

  at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:169)

  at org.springframework.jms.core.JmsTemplate.executeLocal(JmsTemplate.java:987)

  at org.springframework.jms.core.JmsTemplate.sendAndReceive(JmsTemplate.java:923)

  at coral.rapidreconciler.agent.jms.AgentConnectionJms.connectToValc(AgentConnectionJms.java:52)

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

  at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

  at java.lang.reflect.Method.invoke(Unknown Source)

  at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)

  at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

  at java.util.concurrent.FutureTask.runAndReset(Unknown Source)

  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)

  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

  at java.lang.Thread.run(Unknown Source)

Caused by: javax.jms.JMSException: HQ119014: Timed out waiting for response when sending packet 30

  at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:390)

  at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:891)

  at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:800)

  at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:337)

  at org.hornetq.jms.client.HornetQConnection.authorize(HornetQConnection.java:719)

  at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:762)

  at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:112)

  at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:107)

  at org.springframework.jms.core.JmsTemplate.executeLocal(JmsTemplate.java:976)

  ... 15 common frames omitted

Caused by: org.hornetq.api.core.HornetQConnectionTimedOutException: HQ119014: Timed out waiting for response when sending packet 30

  ... 24 common frames omitted

On the server this line appears on the log:

2016-01-29 16:22:19.389  WARN 4764 --- [re-check-thread] org.hornetq.core.client                  : HQ212037: Connection failure has been detected: HQ119014: Did not receive data from /xx.xxx.xx.xxx:61350. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

Any directions to discover what's wrong with this client? Firewall, proxy, network, resources, version?

[jbertram]

The packet type 30 is the CREATESESSION packet which corresponds with the stack-trace.  If other clients can connect without issue then I would suspect something environmental which is blocking your client somehow.  Have you tried moving that particular client to another machine which you know can communicate with the broker (e.g. localhost) to rule some kind of code issue?

Also, you might consider gathering thread dumps right before, during, and right after the problem is observed so you can see exactly what the broker is doing.

[pgrazek]

Yes, on another machine it does work. So it must be something on the environment.

I can telnet on the JMS server, so there's some connectivity.

Any ideia on what should I look for in thread dumps?

[jbertram]

In the thread dumps you'd look for threads servicing remote clients that look to be stuck (i.e. they are in the same method call across multiple thread dumps).