How to fix HornetQ REST web app lost connection
I have a HornetQ REST server running on EAP6.1. When I call topics/jms.topic.MyJMSTopic/pull-subscriptions, I get the following error in the EAP logs (see EAP Logs).
Just prior to these errors appearing, I get a message about client connections failing (see Client Connection Failures).
It seems clear that the HornetQ REST application lost its connection, but I can't see any options in HornetQ User Manual that would allow me to reestablish the connection. Restarting EAP does fix the issue.
Does anyone know how I might solve this issue without a restart of the application server?
(FYI in-VM connection failure [code=3] and [HORNETQ-944] Unexpected and random connection failures - JBoss Issue Tracker mention similar issues).
EAP Logs
[0m [31m11:45:28,970 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/pressgang-ccms-messaging].[default]] (http-/0.0.0.0:8180-8) JBWEB000236: Servlet.service() for servlet default threw exception: org.jboss.resteasy.spi.UnhandledException: java.lang.RuntimeException: HornetQException[errorType=INTERNAL_ERROR message=HQ119001: Failed to create session] at org.jboss.resteasy.core.SynchronousDispatcher.handleApplicationException(SynchronousDispatcher.java:365) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.handleException(SynchronousDispatcher.java:233) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.handleInvokerException(SynchronousDispatcher.java:209) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:557) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.invokePropagateNotFound(SynchronousDispatcher.java:169) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:212) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.doFilter(FilterDispatcher.java:59) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:156) [cors-filter-1.8.jar:1.8] at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:233) [cors-filter-1.8.jar:1.8] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web-7.2.0.Final-redhat-8.jar:7.2.0.Final-redhat-8] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) [jbossweb-7.2.0.Final-redhat-1.jar:7.2.0.Final-redhat-1] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: java.lang.RuntimeException: HornetQException[errorType=INTERNAL_ERROR message=HQ119001: Failed to create session] at org.hornetq.rest.topic.SubscriptionsResource.subscriptionExists(SubscriptionsResource.java:409) [hornetq-rest-2.4.1.Final.jar:] at org.hornetq.rest.topic.SubscriptionsResource.createSubscription(SubscriptionsResource.java:214) [hornetq-rest-2.4.1.Final.jar:] at sun.reflect.GeneratedMethodAccessor2377.invoke(Unknown Source) [:1.7.0_25] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_25] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_25] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:269) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:227) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:159) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:107) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:154) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:92) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542) [resteasy-jaxrs-2.3.6.Final-redhat-1.jar:2.3.6.Final-redhat-1] ... 21 more Caused by: HornetQException[errorType=INTERNAL_ERROR message=HQ119001: Failed to create session] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:928) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:363) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.rest.topic.SubscriptionsResource.subscriptionExists(SubscriptionsResource.java:402) [hornetq-rest-2.4.1.Final.jar:] ... 33 more Caused by: java.lang.IllegalStateException: Connection is null at org.hornetq.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:797) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] ... 35 more
Client Connection Failures
08:52:27,251 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 08:52:27,254 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 08:52:27,254 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea4ecfd9-dbaf-11e3-9dfe-7f969a935307 08:52:27,276 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea4ecfd9-dbaf-11e3-9dfe-7f969a935307 08:52:27,276 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea4ef6ea-dbaf-11e3-9dfe-7f969a935307 08:52:27,276 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea4ef6ea-dbaf-11e3-9dfe-7f969a935307 08:52:27,276 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea4fba3b-dbaf-11e3-9dfe-7f969a935307 08:52:27,277 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea4fba3b-dbaf-11e3-9dfe-7f969a935307 08:52:27,277 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea50085c-dbaf-11e3-9dfe-7f969a935307 08:52:27,277 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea50085c-dbaf-11e3-9dfe-7f969a935307 08:52:27,277 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea50567d-dbaf-11e3-9dfe-7f969a935307 08:52:27,278 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea50567d-dbaf-11e3-9dfe-7f969a935307 08:52:27,278 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea50a49e-dbaf-11e3-9dfe-7f969a935307 08:52:27,278 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea50a49e-dbaf-11e3-9dfe-7f969a935307 08:52:27,278 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea50cbaf-dbaf-11e3-9dfe-7f969a935307 08:52:27,278 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea50cbaf-dbaf-11e3-9dfe-7f969a935307 08:52:27,278 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea50f2c0-dbaf-11e3-9dfe-7f969a935307 08:52:27,279 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea50f2c0-dbaf-11e3-9dfe-7f969a935307 08:52:27,279 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea5167f1-dbaf-11e3-9dfe-7f969a935307 08:52:27,279 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea5167f1-dbaf-11e3-9dfe-7f969a935307 08:52:27,279 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea51b612-dbaf-11e3-9dfe-7f969a935307 08:52:27,279 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea51b612-dbaf-11e3-9dfe-7f969a935307 08:52:27,279 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea520433-dbaf-11e3-9dfe-7f969a935307 08:52:27,280 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea520433-dbaf-11e3-9dfe-7f969a935307 08:52:27,280 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea522b44-dbaf-11e3-9dfe-7f969a935307 08:52:27,280 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea522b44-dbaf-11e3-9dfe-7f969a935307 08:52:27,280 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea538ad5-dbaf-11e3-9dfe-7f969a935307 08:52:27,280 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea538ad5-dbaf-11e3-9dfe-7f969a935307 08:52:27,280 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea53b1e6-dbaf-11e3-9dfe-7f969a935307 08:52:27,280 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea53b1e6-dbaf-11e3-9dfe-7f969a935307 08:52:27,281 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea53d8f7-dbaf-11e3-9dfe-7f969a935307 08:52:27,281 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea53d8f7-dbaf-11e3-9dfe-7f969a935307 08:52:27,281 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea542718-dbaf-11e3-9dfe-7f969a935307 08:52:27,281 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea542718-dbaf-11e3-9dfe-7f969a935307 08:52:27,281 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea544e29-dbaf-11e3-9dfe-7f969a935307 08:52:27,281 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea544e29-dbaf-11e3-9dfe-7f969a935307 08:52:27,281 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea549c4a-dbaf-11e3-9dfe-7f969a935307 08:52:27,282 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea549c4a-dbaf-11e3-9dfe-7f969a935307 08:52:27,282 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea55117b-dbaf-11e3-9dfe-7f969a935307 08:52:27,282 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea55117b-dbaf-11e3-9dfe-7f969a935307 08:52:27,282 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session ea55fbdc-dbaf-11e3-9dfe-7f969a935307 08:52:27,282 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session ea55fbdc-dbaf-11e3-9dfe-7f969a935307 08:52:27,284 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT] 08:52:27,284 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 1db47e9c-dbd2-11e3-9dfe-7f969a935307 08:52:27,284 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 1db47e9c-dbd2-11e3-9dfe-7f969a935307 08:52:27,285 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222061: Client connection failed, clearing up resources for session 1db4a5ad-dbd2-11e3-9dfe-7f969a935307 08:52:27,285 WARN [org.hornetq.core.server] (hornetq-failure-check-thread) HQ222107: Cleared up resources for session 1db4a5ad-dbd2-11e3-9dfe-7f969a935307
10:26:13,319 WARN [org.hornetq.jms.server] (Thread-146 (HornetQ-client-global-threads-1451576061)) Notified of connection failure in xa discovery, we will retry on the next recovery: HornetQException[errorType=NOT_CONNECTED message=HQ119006: Channel disconnected] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] 10:26:13,320 WARN [org.hornetq.jms.server] (Thread-148 (HornetQ-client-global-threads-1451576061)) HQ122014: Notified of connection failure in xa recovery connectionFactory for provider ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=db530ea9-dbaf-11e3-9dfe-7f969a935307, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=db530ea9-dbaf-11e3-9dfe-7f969a935307, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null] will attempt reconnect on next pass: HornetQException[errorType=NOT_CONNECTED message=HQ119006: Channel disconnected] at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connectionDestroyed(ClientSessionFactoryImpl.java:418) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.core.remoting.impl.invm.InVMConnector$Listener$1.run(InVMConnector.java:214) [hornetq-server-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:106) [hornetq-core-client-2.3.1.Final-redhat-1.jar:2.3.1.Final-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
