9 Replies Latest reply on Feb 12, 2015 9:31 AM by Justin Bertram

    HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]

    Amos Feng Apprentice

      Hi all,

       

      We had met this issue when running the CI tests. I had checked the log files and it looks like the client lost the connection. see the following

      2014-11-19 21:38:59,625 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

      2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (hornetq-failure-check-thread) Connection removed 78dbd32a-7034-11e4-8c19-61c30e379d43 from server HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43: java.lang.Exception: trace

      at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:554)

      at org.hornetq.core.remoting.impl.invm.InVMAcceptor$Listener.connectionDestroyed(InVMAcceptor.java:278)

      at org.hornetq.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:118)

      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:619)

      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:354)

      at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:765)

       

      2014-11-19 21:38:59,626 INFO [org.hornetq.core.server] (hornetq-failure-check-thread) HQ221021: failed to remove connection

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=in-vm, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=787e49d8-7034-11e4-8c19-61c30e379d43, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null]::Subscribing Topology

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (Thread-14 (HornetQ-client-global-threads-20087331)) Client Connection failed, calling failure listeners and trying to reconnect, reconnectAttempts=0

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Sending packet nonblocking SubscribeClusterTopologyUpdatesMessage [clusterConnection=false, toString()=PACKET(SubscribeClusterTopologyUpdatesMessageV2)[type=113, channelID=0, packetObject=SubscribeClusterTopologyUpdatesMessageV2]] on channeID=0

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Writing buffer for channelID=0

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) returning null

      2014-11-19 21:38:59,628 DEBUG [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Trying to connect towards ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=in-vm, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=787e49d8-7034-11e4-8c19-61c30e379d43, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null]

      2014-11-19 21:38:59,629 DEBUG [org.codehaus.stomp.jms.ProtocolConverter] (StompConnect Transport: tcp:///127.0.0.1:60851) Caught an exception: : javax.jms.JMSException: Failed to create session factory

      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:673)

      at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:112)

      at org.codehaus.stomp.jms.ProtocolConverter.onStompConnect(ProtocolConverter.java:229)

      at org.codehaus.stomp.jms.ProtocolConverter.onStompFrame(ProtocolConverter.java:182)

      at org.codehaus.stomp.tcp.TcpTransport.run(TcpTransport.java:101)

      at java.lang.Thread.run(Thread.java:744)

      Caused by: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.]

      at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:901)

      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:669)

      ... 5 more

      It looks very strange and I wonder how this issue could happened.

      We use the default setting in the wildfly configuration.

      <jms-connection-factories>
                          <connection-factory name="InVmConnectionFactory">
                              <connectors>
                                  <connector-ref connector-name="in-vm"/>
                              </connectors>
                              <entries>
                                  <entry name="java:/ConnectionFactory"/>
                              </entries>
                              <client-failure-check-period>86400000</client-failure-check-period>
                              <connection-ttl>-1</connection-ttl>
                          </connection-factory>
                          <connection-factory name="RemoteConnectionFactory">
                              <connectors>
                                  <connector-ref connector-name="http-connector"/>
                              </connectors>
                              <entries>
                                  <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                              </entries>
                          </connection-factory>
                          <pooled-connection-factory name="hornetq-ra">
                              <transaction mode="xa"/>
                              <connectors>
                                  <connector-ref connector-name="in-vm"/>
                              </connectors>
                              <entries>
                                  <entry name="java:/JmsXA"/>
                                  <entry name="java:jboss/DefaultJMSConnectionFactory"/>
                              </entries>
                              <client-failure-check-period>86400000</client-failure-check-period>
                              <connection-ttl>-1</connection-ttl>
                          </pooled-connection-factory>
                      </jms-connection-factories>
      

       

      I just want to know if it would be helpful by setting the <reconnect-attempts> ?

       

      Thanks in any suggestion and feedback !

      Amos

        • 2. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
          Justin Bertram Master

          The version of HornetQ you're using (2.4.5.Final) should have the fix from https://issues.jboss.org/browse/HORNETQ-1314 which should disable connection monitoring for in-vm connections so that errors like this are not even possible (unless the client itself is setting a connection-ttl).  Can you isolate the test where this failure happened and reproduce it simply and reliably?

          • 3. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
            Amos Feng Apprentice

            Thanks Justin,

             

            This issue happens only once and it could be hard to reproduce. Anyway, I will double check the connection-ttl at first.

             

            Amos

            • 4. Re: Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
              Amos Feng Apprentice

              I can confirm that we do not set a connection-ttl and it should use "-1" as the default value. We just use the security credential with username and password.

               

              I doubt that the hornetq-failure-check-thread  running in the client side

              2014-11-19 21:38:59,624 DEBUG [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Trying reconnection attempt 0/1
              2014-11-19 21:38:59,624 DEBUG [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Trying to connect with connector = org.hornetq.core.remoting.impl.invm.InVMConnectorFactory@77a31c, parameters = {server-id=0} connector = org.hornetq.core.remoting.impl.invm.InVMConnector@75b4be
              2014-11-19 21:38:59,624 DEBUG [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Trying to connect at the main server using connector :TransportConfiguration(name=787e49d8-7034-11e4-8c19-61c30e379d43, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0
              2014-11-19 21:38:59,624 TRACE [org.hornetq.core.server] (StompConnect Transport: tcp:///127.0.0.1:60851) Connection created InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]
              2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1] on channeID=0
              2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Writing buffer for channelID=0
              2014-11-19 21:38:59,624 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::Sending inVM packet
              2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1]
              2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1] on channeID=0
              2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) Writing buffer for channelID=0
              2014-11-19 21:38:59,624 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.server] (Thread-13 (HornetQ-client-global-threads-20087331)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::Sending inVM packet
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-13 (HornetQ-client-global-threads-20087331)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1]
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.server] (Thread-13 (HornetQ-client-global-threads-20087331)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-1 (HornetQ-client-global-scheduled-threads-16640325)) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1] on channeID=0
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-1 (HornetQ-client-global-scheduled-threads-16640325)) Writing buffer for channelID=0
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::Sending inVM packet
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1]
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1] on channeID=0
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) Writing buffer for channelID=0
              2014-11-19 21:38:59,625 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done
              2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (Thread-13 (HornetQ-client-global-threads-20087331)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::Sending inVM packet
              2014-11-19 21:38:59,626 TRACE [org.hornetq.core.client] (Thread-13 (HornetQ-client-global-threads-20087331)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1]
              2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (Thread-13 (HornetQ-client-global-threads-20087331)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done
              2014-11-19 21:38:59,625 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
              2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (hornetq-failure-check-thread) Connection removed 78dbd32a-7034-11e4-8c19-61c30e379d43 from server HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43: java.lang.Exception: trace
              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:554)
              at org.hornetq.core.remoting.impl.invm.InVMAcceptor$Listener.connectionDestroyed(InVMAcceptor.java:278)
              at org.hornetq.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:118)
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:619)
              at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:354)
              at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:765)
              
              2014-11-19 21:38:59,626 INFO [org.hornetq.core.server] (hornetq-failure-check-thread) HQ221021: failed to remove connection
              

               

              see the above logs that it looks like normal processes before the hornetq-failure-check-thread throws the Exception. How does it detect the connection failure even if the connection-ttl setting with "-1" ?

              • 5. Re: Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                Justin Bertram Master

                I don't see in the code how it is possible for a connection to fail if the ttl is -1.  Can you paste your whole config?

                 

                Also, any chance you can provide a simple, reproducible test-case for this?

                • 6. Re: Re: Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                  Amos Feng Apprentice

                  sure, the whole standalone-blacktie.xml

                  <?xml version='1.0' encoding='UTF-8'?>
                  
                  
                  <server xmlns="urn:jboss:domain:3.0">
                  
                  
                      <extensions>
                          <extension module="org.jboss.as.clustering.infinispan"/>
                          <extension module="org.jboss.as.connector"/>
                          <extension module="org.jboss.as.deployment-scanner"/>
                          <extension module="org.jboss.as.ee"/>
                          <extension module="org.jboss.as.ejb3"/>
                          <extension module="org.jboss.as.jaxrs"/>
                          <extension module="org.jboss.as.jdr"/>
                          <extension module="org.jboss.as.jmx"/>
                          <extension module="org.jboss.as.jpa"/>
                          <extension module="org.jboss.as.jsf"/>
                          <extension module="org.jboss.as.jsr77"/>
                          <extension module="org.jboss.as.logging"/>
                          <extension module="org.jboss.as.mail"/>
                          <extension module="org.jboss.as.messaging"/>
                          <extension module="org.jboss.as.naming"/>
                          <extension module="org.jboss.as.pojo"/>
                          <extension module="org.jboss.as.remoting"/>
                          <extension module="org.jboss.as.sar"/>
                          <extension module="org.jboss.as.security"/>
                          <extension module="org.jboss.as.transactions"/>
                          <extension module="org.jboss.as.webservices"/>
                          <extension module="org.jboss.as.weld"/>
                          <extension module="org.wildfly.extension.batch"/>
                          <extension module="org.wildfly.extension.bean-validation"/>
                          <extension module="org.wildfly.extension.blacktie"/>
                          <extension module="org.wildfly.extension.io"/>
                          <extension module="org.wildfly.extension.request-controller"/>
                          <extension module="org.wildfly.extension.rts"/>
                          <extension module="org.wildfly.extension.undertow"/>
                          <extension module="org.wildfly.iiop-openjdk"/>
                      </extensions>
                  
                  
                  
                  
                      <management>
                          <security-realms>
                              <security-realm name="ManagementRealm">
                                  <authentication>
                                      <local default-user="$local" skip-group-loading="true"/>
                                      <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                                  </authentication>
                                  <authorization map-groups-to-roles="false">
                                      <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                                  </authorization>
                              </security-realm>
                              <security-realm name="ApplicationRealm">
                                  <authentication>
                                      <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
                                      <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                                  </authentication>
                                  <authorization>
                                      <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                                  </authorization>
                              </security-realm>
                          </security-realms>
                          <audit-log>
                              <formatters>
                                  <json-formatter name="json-formatter"/>
                              </formatters>
                              <handlers>
                                  <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
                              </handlers>
                              <logger log-boot="true" log-read-only="false" enabled="false">
                                  <handlers>
                                      <handler name="file"/>
                                  </handlers>
                              </logger>
                          </audit-log>
                          <management-interfaces>
                              <http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">
                                  <socket-binding http="management-http"/>
                              </http-interface>
                          </management-interfaces>
                          <access-control provider="simple">
                              <role-mapping>
                                  <role name="SuperUser">
                                      <include>
                                          <user name="$local"/>
                                      </include>
                                  </role>
                              </role-mapping>
                          </access-control>
                      </management>
                  
                  
                      <profile>
                          <subsystem xmlns="urn:jboss:domain:logging:3.0">
                              <console-handler name="CONSOLE">
                                  <level name="INFO"/>
                                  <formatter>
                                      <named-formatter name="COLOR-PATTERN"/>
                                  </formatter>
                              </console-handler>
                              <periodic-rotating-file-handler name="FILE" autoflush="true">
                                  <formatter>
                                      <named-formatter name="PATTERN"/>
                                  </formatter>
                                  <file relative-to="jboss.server.log.dir" path="server.log"/>
                                  <suffix value=".yyyy-MM-dd"/>
                                  <append value="true"/>
                              </periodic-rotating-file-handler>
                              <logger category="com.arjuna">
                                  <level name="WARN"/>
                              </logger>
                              <logger category="org.apache.tomcat.util.modeler">
                                  <level name="WARN"/>
                              </logger>
                              <logger category="org.jboss.as.config">
                                  <level name="DEBUG"/>
                              </logger>
                              <logger category="sun.rmi">
                                  <level name="WARN"/>
                              </logger>
                              <logger category="jacorb">
                                  <level name="WARN"/>
                              </logger>
                              <logger category="jacorb.config">
                                  <level name="ERROR"/>
                              </logger>
                              <logger category="org.jboss.ejb.client">
                                  <level name="ALL"/>
                              </logger>
                              <logger category="org.jboss.as.ejb3">
                                  <level name="ALL"/>
                              </logger>
                              <logger category="org.jboss.narayana.blacktie">
                                  <level name="ALL"/>
                              </logger>
                              <logger category="org.codehaus.stomp">
                                  <level name="ALL"/>
                              </logger>
                              <logger category="org.hornetq">
                                  <level name="ALL"/>
                              </logger>
                              <root-logger>
                                  <level name="INFO"/>
                                  <handlers>
                                      <handler name="CONSOLE"/>
                                      <handler name="FILE"/>
                                  </handlers>
                              </root-logger>
                              <formatter name="PATTERN">
                                  <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
                              </formatter>
                              <formatter name="COLOR-PATTERN">
                                  <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
                              </formatter>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:batch:1.0">
                              <job-repository>
                                  <in-memory/>
                              </job-repository>
                              <thread-pool>
                                  <max-threads count="10"/>
                                  <keepalive-time time="30" unit="seconds"/>
                              </thread-pool>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>
                          <subsystem xmlns="urn:jboss:domain:datasources:3.0">
                              <datasources>
                                  <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
                                      <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                                      <driver>h2</driver>
                                      <security>
                                          <user-name>sa</user-name>
                                          <password>sa</password>
                                      </security>
                                  </datasource>
                                  <drivers>
                                      <driver name="h2" module="com.h2database.h2">
                                          <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                                      </driver>
                                  </drivers>
                              </datasources>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">
                              <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:ee:3.0">
                              <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
                              <concurrent>
                                  <context-services>
                                      <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
                                  </context-services>
                                  <managed-thread-factories>
                                      <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
                                  </managed-thread-factories>
                                  <managed-executor-services>
                                      <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" core-threads="5" max-threads="25" keepalive-time="5000"/>
                                  </managed-executor-services>
                                  <managed-scheduled-executor-services>
                                      <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" core-threads="2" keepalive-time="3000"/>
                                  </managed-scheduled-executor-services>
                              </concurrent>
                              <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:ejb3:3.0">
                              <session-bean>
                                  <stateful default-access-timeout="5000" cache-ref="simple" passivation-disabled-cache-ref="simple"/>
                                  <singleton default-access-timeout="5000"/>
                              </session-bean>
                              <mdb>
                                  <resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:hornetq-ra.rar}"/>
                                  <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
                              </mdb>
                              <pools>
                                  <bean-instance-pools>
                                      <strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                                      <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                                  </bean-instance-pools>
                              </pools>
                              <caches>
                                  <cache name="simple"/>
                                  <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>
                              </caches>
                              <passivation-stores>
                                  <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>
                              </passivation-stores>
                              <async thread-pool-name="default"/>
                              <timer-service thread-pool-name="default" default-data-store="default-file-store">
                                  <data-stores>
                                      <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>
                                  </data-stores>
                              </timer-service>
                              <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>
                              <thread-pools>
                                  <thread-pool name="default">
                                      <max-threads count="10"/>
                                      <keepalive-time time="100" unit="milliseconds"/>
                                  </thread-pool>
                              </thread-pools>
                              <iiop enable-by-default="false" use-qualified-name="false"/>
                              <default-security-domain value="other"/>
                              <default-missing-method-permissions-deny-access value="true"/>
                              <log-system-exceptions value="true"/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:io:1.1">
                              <worker name="default"/>
                              <buffer-pool name="default"/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:infinispan:3.0">
                              <cache-container name="server" default-cache="default" module="org.wildfly.clustering.server">
                                  <local-cache name="default">
                                      <transaction mode="BATCH"/>
                                  </local-cache>
                              </cache-container>
                              <cache-container name="web" default-cache="passivation" module="org.wildfly.clustering.web.infinispan">
                                  <local-cache name="passivation">
                                      <transaction mode="BATCH"/>
                                      <file-store passivation="true" purge="false"/>
                                  </local-cache>
                                  <local-cache name="persistent">
                                      <transaction mode="BATCH"/>
                                      <file-store passivation="false" purge="false"/>
                                  </local-cache>
                              </cache-container>
                              <cache-container name="ejb" default-cache="passivation" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">
                                  <local-cache name="passivation">
                                      <transaction mode="BATCH"/>
                                      <file-store passivation="true" purge="false"/>
                                  </local-cache>
                                  <local-cache name="persistent">
                                      <transaction mode="BATCH"/>
                                      <file-store passivation="false" purge="false"/>
                                  </local-cache>
                              </cache-container>
                              <cache-container name="hibernate" default-cache="local-query" module="org.hibernate">
                                  <local-cache name="entity">
                                      <transaction mode="NON_XA"/>
                                      <eviction strategy="LRU" max-entries="10000"/>
                                      <expiration max-idle="100000"/>
                                  </local-cache>
                                  <local-cache name="local-query">
                                      <eviction strategy="LRU" max-entries="10000"/>
                                      <expiration max-idle="100000"/>
                                  </local-cache>
                                  <local-cache name="timestamps"/>
                              </cache-container>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:iiop-openjdk:1.0">
                              <orb socket-binding="iiop" ssl-socket-binding="iiop-ssl">
                                  <initializers security="identity" transactions="full"/>
                              </orb>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
                          <subsystem xmlns="urn:jboss:domain:jca:3.0">
                              <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
                              <bean-validation enabled="true"/>
                              <default-workmanager>
                                  <short-running-threads>
                                      <core-threads count="50"/>
                                      <queue-length count="50"/>
                                      <max-threads count="50"/>
                                      <keepalive-time time="10" unit="seconds"/>
                                  </short-running-threads>
                                  <long-running-threads>
                                      <core-threads count="50"/>
                                      <queue-length count="50"/>
                                      <max-threads count="50"/>
                                      <keepalive-time time="10" unit="seconds"/>
                                  </long-running-threads>
                              </default-workmanager>
                              <cached-connection-manager/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
                          <subsystem xmlns="urn:jboss:domain:jmx:1.3">
                              <expose-resolved-model/>
                              <expose-expression-model/>
                              <remoting-connector/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:jpa:1.1">
                              <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:jsf:1.0"/>
                          <subsystem xmlns="urn:jboss:domain:jsr77:1.0"/>
                          <subsystem xmlns="urn:jboss:domain:mail:2.0">
                              <mail-session name="default" jndi-name="java:jboss/mail/Default">
                                  <smtp-server outbound-socket-binding-ref="mail-smtp"/>
                              </mail-session>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:messaging:3.0">
                              <hornetq-server>
                                  <journal-file-size>102400</journal-file-size>
                  
                  
                                  <connectors>
                                      <http-connector name="http-connector" socket-binding="http">
                                          <param key="http-upgrade-endpoint" value="http-acceptor"/>
                                      </http-connector>
                                      <http-connector name="http-connector-throughput" socket-binding="http">
                                          <param key="http-upgrade-endpoint" value="http-acceptor-throughput"/>
                                          <param key="batch-delay" value="50"/>
                                      </http-connector>
                                      <in-vm-connector name="in-vm" server-id="0"/>
                                  </connectors>
                  
                  
                                  <acceptors>
                                      <http-acceptor http-listener="default" name="http-acceptor"/>
                                      <http-acceptor http-listener="default" name="http-acceptor-throughput">
                                          <param key="batch-delay" value="50"/>
                                          <param key="direct-deliver" value="false"/>
                                      </http-acceptor>
                                      <in-vm-acceptor name="in-vm" server-id="0"/>
                                  </acceptors>
                  
                  
                                  <security-settings>
                                      <security-setting match="jms.queue.BTR_SECURE">
                                          <permission type="send" roles="dynsub"/>
                                          <permission type="consume" roles="dynsub"/>
                                      </security-setting>
                                      <security-setting match="jms.queue.BTR_BTDomainAdmin">
                                          <permission type="send" roles="blacktie"/>
                                          <permission type="consume" roles="blacktie"/>
                                      </security-setting>
                                      <security-setting match="jms.queue.BTR_BTStompAdmin">
                                          <permission type="send" roles="blacktie"/>
                                          <permission type="consume" roles="blacktie"/>
                                      </security-setting>
                                      <security-setting match="#">
                                          <permission type="send" roles="guest"/>
                                          <permission type="consume" roles="guest"/>
                                          <permission type="createNonDurableQueue" roles="guest"/>
                                          <permission type="deleteNonDurableQueue" roles="guest"/>
                                      </security-setting>
                                  </security-settings>
                  
                  
                                  <address-settings>
                                      <address-setting match="#">
                                          <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                                          <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                                          <max-size-bytes>10485760</max-size-bytes>
                                          <page-size-bytes>2097152</page-size-bytes>
                                          <message-counter-history-day-limit>10</message-counter-history-day-limit>
                                      </address-setting>
                                  </address-settings>
                  
                  
                                  <jms-connection-factories>
                                      <connection-factory name="InVmConnectionFactory">
                                          <connectors>
                                              <connector-ref connector-name="in-vm"/>
                                          </connectors>
                                          <entries>
                                              <entry name="java:/ConnectionFactory"/>
                                          </entries>
                                          <client-failure-check-period>86400000</client-failure-check-period>
                                          <connection-ttl>-1</connection-ttl>
                                      </connection-factory>
                                      <connection-factory name="RemoteConnectionFactory">
                                          <connectors>
                                              <connector-ref connector-name="http-connector"/>
                                          </connectors>
                                          <entries>
                                              <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                                          </entries>
                                      </connection-factory>
                                      <pooled-connection-factory name="hornetq-ra">
                                          <transaction mode="xa"/>
                                          <connectors>
                                              <connector-ref connector-name="in-vm"/>
                                          </connectors>
                                          <entries>
                                              <entry name="java:/JmsXA"/>
                                              <entry name="java:jboss/DefaultJMSConnectionFactory"/>
                                          </entries>
                                          <client-failure-check-period>86400000</client-failure-check-period>
                                          <connection-ttl>-1</connection-ttl>
                                      </pooled-connection-factory>
                                  </jms-connection-factories>
                  
                  
                                  <jms-destinations>
                                      <jms-queue name="ExpiryQueue">
                                          <entry name="java:/jms/queue/ExpiryQueue"/>
                                      </jms-queue>
                                      <jms-queue name="DLQ">
                                          <entry name="java:/jms/queue/DLQ"/>
                                      </jms-queue>
                                      <jms-queue name="BTR_TestOne">
                                          <entry name="queue/BTR_TestOne"/>
                                          <entry name="java:jboss/exported/jms/queue/BTR_TestOne"/>
                                      </jms-queue>
                                      <jms-queue name="BTR_TestTwo">
                                          <entry name="queue/BTR_TestTwo"/>
                                          <entry name="java:jboss/exported/jms/queue/BTR_TestTwo"/>
                                      </jms-queue>
                                      <jms-queue name="BTC_ConvService">
                                          <entry name="queue/BTC_ConvService"/>
                                          <entry name="java:jboss/exported/jms/queue/BTC_ConvService"/>
                                      </jms-queue>
                                      <jms-queue name="BTR_JAVA_Converse">
                                          <entry name="queue/BTR_JAVA_Converse"/>
                                          <entry name="java:jboss/exported/jms/queue/BTR_JAVA_Converse"/>
                                      </jms-queue>
                                      <jms-topic name="BTR_JAVA_Topic">
                                          <entry name="topic/BTR_JAVA_Topic"/>
                                          <entry name="java:jboss/exported/jms/topic/BTR_JAVA_Topic"/>
                                      </jms-topic>
                                  </jms-destinations>
                              </hornetq-server>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:naming:2.0">
                              <remote-naming/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
                          <subsystem xmlns="urn:jboss:domain:remoting:3.0">
                              <endpoint worker="default"/>
                              <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:resource-adapters:3.0"/>
                          <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>
                          <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
                          <subsystem xmlns="urn:jboss:domain:security:1.2">
                              <security-domains>
                                  <security-domain name="other" cache-type="default">
                                      <authentication>
                                          <login-module code="Remoting" flag="optional">
                                              <module-option name="password-stacking" value="useFirstPass"/>
                                          </login-module>
                                          <login-module code="RealmDirect" flag="required">
                                              <module-option name="password-stacking" value="useFirstPass"/>
                                          </login-module>
                                      </authentication>
                                  </security-domain>
                                  <security-domain name="jboss-web-policy" cache-type="default">
                                      <authorization>
                                          <policy-module code="Delegating" flag="required"/>
                                      </authorization>
                                  </security-domain>
                                  <security-domain name="jboss-ejb-policy" cache-type="default">
                                      <authorization>
                                          <policy-module code="Delegating" flag="required"/>
                                      </authorization>
                                  </security-domain>
                              </security-domains>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:transactions:3.0">
                              <core-environment>
                                  <process-id>
                                      <uuid/>
                                  </process-id>
                              </core-environment>
                              <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
                              <jts/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:undertow:2.0">
                              <buffer-cache name="default"/>
                              <server name="default-server">
                                  <http-listener name="default" socket-binding="http"/>
                                  <host name="default-host" alias="localhost">
                                      <location name="/" handler="welcome-content"/>
                                      <filter-ref name="server-header"/>
                                      <filter-ref name="x-powered-by-header"/>
                                  </host>
                              </server>
                              <servlet-container name="default">
                                  <jsp-config/>
                                  <websockets/>
                              </servlet-container>
                              <handlers>
                                  <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
                              </handlers>
                              <filters>
                                  <response-header name="server-header" header-name="Server" header-value="WildFly/9"/>
                                  <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
                              </filters>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:webservices:2.0">
                              <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
                              <endpoint-config name="Standard-Endpoint-Config"/>
                              <endpoint-config name="Recording-Endpoint-Config">
                                  <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
                                      <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
                                  </pre-handler-chain>
                              </endpoint-config>
                              <client-config name="Standard-Client-Config"/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:weld:2.0"/>
                          <subsystem xmlns="urn:jboss:domain:rts:1.0">
                              <servlet server="default-server" host="default-host" socket-binding="http"/>
                          </subsystem>
                          <subsystem xmlns="urn:jboss:domain:blacktie:1.0">
                              <stompconnect connection-factoryname="java:/JmsXA" socket-binding="stomp"/>
                          </subsystem>
                      </profile>
                  
                  
                      <interfaces>
                          <interface name="management">
                              <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
                          </interface>
                          <interface name="public">
                              <inet-address value="${jboss.bind.address:127.0.0.1}"/>
                          </interface>
                          <interface name="unsecure">
                              <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
                          </interface>
                      </interfaces>
                  
                  
                      <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
                          <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
                          <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
                          <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
                          <socket-binding name="http" port="${jboss.http.port:8080}"/>
                          <socket-binding name="https" port="${jboss.https.port:8443}"/>
                          <socket-binding name="iiop" interface="unsecure" port="3528"/>
                          <socket-binding name="iiop-ssl" interface="unsecure" port="3529"/>
                          <socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9876}"/>
                          <socket-binding name="stomp" port="61613"/>
                          <socket-binding name="txn-recovery-environment" port="4712"/>
                          <socket-binding name="txn-status-manager" port="4713"/>
                          <outbound-socket-binding name="mail-smtp">
                              <remote-destination host="localhost" port="25"/>
                          </outbound-socket-binding>
                      </socket-binding-group>
                  
                  
                      <deployments>
                          <deployment name="blacktie-admin-services-ear-5.0.5.Final-SNAPSHOT.ear" runtime-name="blacktie-admin-services-ear-5.0.5.Final-SNAPSHOT.ear">
                              <content sha1="34b41da3c26d3335f027be3e22b36953d34510ee"/>
                          </deployment>
                      </deployments>
                  </server>
                  
                  • 7. Re: Re: Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                    Justin Bertram Master

                    The config looks fine to me.  Any luck with a simple, reproducible test-case?

                    • 8. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                      Tom Jenkinson Master

                      Hi,

                       

                      For invm, is it possible that we don't need:

                      <connection-ttl>-1</connection-ttl>

                      anymore, as it looks like the commit (HORNETQ-1314 - disable monitoring for in-vm connections ·  f80610d · hornetq/hornetq · GitHub) should make this unrequired?

                       

                      Also, for invm do you think we no longer need:

                      <client-failure-check-period>86400000</client-failure-check-period>

                       

                      Finally, what is it that causes:

                       

                      WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

                      2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (hornetq-failure-check-thread) Connection removed 78dbd32a-7034-11e4-8c19-61c30e379d43 from server HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43: java.lang.Exception: trace

                      at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:554)

                      at org.hornetq.core.remoting.impl.invm.InVMAcceptor$Listener.connectionDestroyed(InVMAcceptor.java:278)

                      at org.hornetq.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:118)

                      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:619)

                      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:354)

                      at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:765)

                       

                      I would imagine that if ttl was -1 it wasn't registered for checking anywhere for failure checking so we could hook up a debugger and see why that is happening if we had some pointers.


                      Thanks,

                      Tom

                      • 9. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                        Justin Bertram Master

                        The best place to start is probably the run() method of org.hornetq.core.remoting.server.impl.RemotingServiceImpl.FailureCheckAndFlushThread as it decides which connections should be closed based on activity.