9 Replies Latest reply on Feb 12, 2015 9:31 AM by jbertram

    HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]

    zhfeng

      Hi all,

       

      We had met this issue when running the CI tests. I had checked the log files and it looks like the client lost the connection. see the following

      2014-11-19 21:38:59,625 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

      2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (hornetq-failure-check-thread) Connection removed 78dbd32a-7034-11e4-8c19-61c30e379d43 from server HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43: java.lang.Exception: trace

      at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:554)

      at org.hornetq.core.remoting.impl.invm.InVMAcceptor$Listener.connectionDestroyed(InVMAcceptor.java:278)

      at org.hornetq.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:118)

      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:619)

      at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:354)

      at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:765)

       

      2014-11-19 21:38:59,626 INFO [org.hornetq.core.server] (hornetq-failure-check-thread) HQ221021: failed to remove connection

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=in-vm, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=787e49d8-7034-11e4-8c19-61c30e379d43, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null]::Subscribing Topology

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (Thread-14 (HornetQ-client-global-threads-20087331)) Client Connection failed, calling failure listeners and trying to reconnect, reconnectAttempts=0

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Sending packet nonblocking SubscribeClusterTopologyUpdatesMessage [clusterConnection=false, toString()=PACKET(SubscribeClusterTopologyUpdatesMessageV2)[type=113, channelID=0, packetObject=SubscribeClusterTopologyUpdatesMessageV2]] on channeID=0

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Writing buffer for channelID=0

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done

      2014-11-19 21:38:59,627 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) returning null

      2014-11-19 21:38:59,628 DEBUG [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Trying to connect towards ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=in-vm, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=787e49d8-7034-11e4-8c19-61c30e379d43, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, backupConfig=null]

      2014-11-19 21:38:59,629 DEBUG [org.codehaus.stomp.jms.ProtocolConverter] (StompConnect Transport: tcp:///127.0.0.1:60851) Caught an exception: : javax.jms.JMSException: Failed to create session factory

      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:673)

      at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:112)

      at org.codehaus.stomp.jms.ProtocolConverter.onStompConnect(ProtocolConverter.java:229)

      at org.codehaus.stomp.jms.ProtocolConverter.onStompFrame(ProtocolConverter.java:182)

      at org.codehaus.stomp.tcp.TcpTransport.run(TcpTransport.java:101)

      at java.lang.Thread.run(Thread.java:744)

      Caused by: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s). Tried with all available servers.]

      at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:901)

      at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:669)

      ... 5 more

      It looks very strange and I wonder how this issue could happened.

      We use the default setting in the wildfly configuration.

      <jms-connection-factories>
                          <connection-factory name="InVmConnectionFactory">
                              <connectors>
                                  <connector-ref connector-name="in-vm"/>
                              </connectors>
                              <entries>
                                  <entry name="java:/ConnectionFactory"/>
                              </entries>
                              <client-failure-check-period>86400000</client-failure-check-period>
                              <connection-ttl>-1</connection-ttl>
                          </connection-factory>
                          <connection-factory name="RemoteConnectionFactory">
                              <connectors>
                                  <connector-ref connector-name="http-connector"/>
                              </connectors>
                              <entries>
                                  <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                              </entries>
                          </connection-factory>
                          <pooled-connection-factory name="hornetq-ra">
                              <transaction mode="xa"/>
                              <connectors>
                                  <connector-ref connector-name="in-vm"/>
                              </connectors>
                              <entries>
                                  <entry name="java:/JmsXA"/>
                                  <entry name="java:jboss/DefaultJMSConnectionFactory"/>
                              </entries>
                              <client-failure-check-period>86400000</client-failure-check-period>
                              <connection-ttl>-1</connection-ttl>
                          </pooled-connection-factory>
                      </jms-connection-factories>
      

       

      I just want to know if it would be helpful by setting the <reconnect-attempts> ?

       

      Thanks in any suggestion and feedback !

      Amos

        • 1. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
          zhfeng

          the full server.log should be found on http://albany.eng.hst.ams2.redhat.com/view/Status/job/narayana/694/PROFILE=BLACKTIE,jdk=jdk7.latest,label=linux32el6/art… (it should be only available in the redhat internal network)

           

          Amos

          • 2. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
            jbertram

            The version of HornetQ you're using (2.4.5.Final) should have the fix from https://issues.jboss.org/browse/HORNETQ-1314 which should disable connection monitoring for in-vm connections so that errors like this are not even possible (unless the client itself is setting a connection-ttl).  Can you isolate the test where this failure happened and reproduce it simply and reliably?

            • 3. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
              zhfeng

              Thanks Justin,

               

              This issue happens only once and it could be hard to reproduce. Anyway, I will double check the connection-ttl at first.

               

              Amos

              • 4. Re: Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                zhfeng

                I can confirm that we do not set a connection-ttl and it should use "-1" as the default value. We just use the security credential with username and password.

                 

                I doubt that the hornetq-failure-check-thread  running in the client side

                2014-11-19 21:38:59,624 DEBUG [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Trying reconnection attempt 0/1
                2014-11-19 21:38:59,624 DEBUG [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Trying to connect with connector = org.hornetq.core.remoting.impl.invm.InVMConnectorFactory@77a31c, parameters = {server-id=0} connector = org.hornetq.core.remoting.impl.invm.InVMConnector@75b4be
                2014-11-19 21:38:59,624 DEBUG [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Trying to connect at the main server using connector :TransportConfiguration(name=787e49d8-7034-11e4-8c19-61c30e379d43, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0
                2014-11-19 21:38:59,624 TRACE [org.hornetq.core.server] (StompConnect Transport: tcp:///127.0.0.1:60851) Connection created InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]
                2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1] on channeID=0
                2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (StompConnect Transport: tcp:///127.0.0.1:60851) Writing buffer for channelID=0
                2014-11-19 21:38:59,624 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::Sending inVM packet
                2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1]
                2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1] on channeID=0
                2014-11-19 21:38:59,624 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) Writing buffer for channelID=0
                2014-11-19 21:38:59,624 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.server] (Thread-13 (HornetQ-client-global-threads-20087331)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::Sending inVM packet
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-13 (HornetQ-client-global-threads-20087331)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1]
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.server] (Thread-13 (HornetQ-client-global-threads-20087331)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-1 (HornetQ-client-global-scheduled-threads-16640325)) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1] on channeID=0
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-1 (HornetQ-client-global-scheduled-threads-16640325)) Writing buffer for channelID=0
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::Sending inVM packet
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1]
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) Sending packet nonblocking PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1] on channeID=0
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.client] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) Writing buffer for channelID=0
                2014-11-19 21:38:59,625 TRACE [org.hornetq.core.server] (Thread-9 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43-25631302-13495423)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done
                2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (Thread-13 (HornetQ-client-global-threads-20087331)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::Sending inVM packet
                2014-11-19 21:38:59,626 TRACE [org.hornetq.core.client] (Thread-13 (HornetQ-client-global-threads-20087331)) handling packet PACKET(Ping)[type=10, channelID=0, packetObject=Ping, connectionTTL=-1]
                2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (Thread-13 (HornetQ-client-global-threads-20087331)) InVMConnection [serverID=0, id=78dbd32a-7034-11e4-8c19-61c30e379d43]::packet sent done
                2014-11-19 21:38:59,625 WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]
                2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (hornetq-failure-check-thread) Connection removed 78dbd32a-7034-11e4-8c19-61c30e379d43 from server HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43: java.lang.Exception: trace
                at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:554)
                at org.hornetq.core.remoting.impl.invm.InVMAcceptor$Listener.connectionDestroyed(InVMAcceptor.java:278)
                at org.hornetq.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:118)
                at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:619)
                at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:354)
                at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:765)
                
                2014-11-19 21:38:59,626 INFO [org.hornetq.core.server] (hornetq-failure-check-thread) HQ221021: failed to remove connection
                

                 

                see the above logs that it looks like normal processes before the hornetq-failure-check-thread throws the Exception. How does it detect the connection failure even if the connection-ttl setting with "-1" ?

                • 5. Re: Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                  jbertram

                  I don't see in the code how it is possible for a connection to fail if the ttl is -1.  Can you paste your whole config?

                   

                  Also, any chance you can provide a simple, reproducible test-case for this?

                  • 6. Re: Re: Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                    zhfeng

                    sure, the whole standalone-blacktie.xml

                    <?xml version='1.0' encoding='UTF-8'?>
                    
                    
                    <server xmlns="urn:jboss:domain:3.0">
                    
                    
                        <extensions>
                            <extension module="org.jboss.as.clustering.infinispan"/>
                            <extension module="org.jboss.as.connector"/>
                            <extension module="org.jboss.as.deployment-scanner"/>
                            <extension module="org.jboss.as.ee"/>
                            <extension module="org.jboss.as.ejb3"/>
                            <extension module="org.jboss.as.jaxrs"/>
                            <extension module="org.jboss.as.jdr"/>
                            <extension module="org.jboss.as.jmx"/>
                            <extension module="org.jboss.as.jpa"/>
                            <extension module="org.jboss.as.jsf"/>
                            <extension module="org.jboss.as.jsr77"/>
                            <extension module="org.jboss.as.logging"/>
                            <extension module="org.jboss.as.mail"/>
                            <extension module="org.jboss.as.messaging"/>
                            <extension module="org.jboss.as.naming"/>
                            <extension module="org.jboss.as.pojo"/>
                            <extension module="org.jboss.as.remoting"/>
                            <extension module="org.jboss.as.sar"/>
                            <extension module="org.jboss.as.security"/>
                            <extension module="org.jboss.as.transactions"/>
                            <extension module="org.jboss.as.webservices"/>
                            <extension module="org.jboss.as.weld"/>
                            <extension module="org.wildfly.extension.batch"/>
                            <extension module="org.wildfly.extension.bean-validation"/>
                            <extension module="org.wildfly.extension.blacktie"/>
                            <extension module="org.wildfly.extension.io"/>
                            <extension module="org.wildfly.extension.request-controller"/>
                            <extension module="org.wildfly.extension.rts"/>
                            <extension module="org.wildfly.extension.undertow"/>
                            <extension module="org.wildfly.iiop-openjdk"/>
                        </extensions>
                    
                    
                    
                    
                        <management>
                            <security-realms>
                                <security-realm name="ManagementRealm">
                                    <authentication>
                                        <local default-user="$local" skip-group-loading="true"/>
                                        <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                                    </authentication>
                                    <authorization map-groups-to-roles="false">
                                        <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                                    </authorization>
                                </security-realm>
                                <security-realm name="ApplicationRealm">
                                    <authentication>
                                        <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
                                        <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                                    </authentication>
                                    <authorization>
                                        <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                                    </authorization>
                                </security-realm>
                            </security-realms>
                            <audit-log>
                                <formatters>
                                    <json-formatter name="json-formatter"/>
                                </formatters>
                                <handlers>
                                    <file-handler name="file" formatter="json-formatter" path="audit-log.log" relative-to="jboss.server.data.dir"/>
                                </handlers>
                                <logger log-boot="true" log-read-only="false" enabled="false">
                                    <handlers>
                                        <handler name="file"/>
                                    </handlers>
                                </logger>
                            </audit-log>
                            <management-interfaces>
                                <http-interface security-realm="ManagementRealm" http-upgrade-enabled="true">
                                    <socket-binding http="management-http"/>
                                </http-interface>
                            </management-interfaces>
                            <access-control provider="simple">
                                <role-mapping>
                                    <role name="SuperUser">
                                        <include>
                                            <user name="$local"/>
                                        </include>
                                    </role>
                                </role-mapping>
                            </access-control>
                        </management>
                    
                    
                        <profile>
                            <subsystem xmlns="urn:jboss:domain:logging:3.0">
                                <console-handler name="CONSOLE">
                                    <level name="INFO"/>
                                    <formatter>
                                        <named-formatter name="COLOR-PATTERN"/>
                                    </formatter>
                                </console-handler>
                                <periodic-rotating-file-handler name="FILE" autoflush="true">
                                    <formatter>
                                        <named-formatter name="PATTERN"/>
                                    </formatter>
                                    <file relative-to="jboss.server.log.dir" path="server.log"/>
                                    <suffix value=".yyyy-MM-dd"/>
                                    <append value="true"/>
                                </periodic-rotating-file-handler>
                                <logger category="com.arjuna">
                                    <level name="WARN"/>
                                </logger>
                                <logger category="org.apache.tomcat.util.modeler">
                                    <level name="WARN"/>
                                </logger>
                                <logger category="org.jboss.as.config">
                                    <level name="DEBUG"/>
                                </logger>
                                <logger category="sun.rmi">
                                    <level name="WARN"/>
                                </logger>
                                <logger category="jacorb">
                                    <level name="WARN"/>
                                </logger>
                                <logger category="jacorb.config">
                                    <level name="ERROR"/>
                                </logger>
                                <logger category="org.jboss.ejb.client">
                                    <level name="ALL"/>
                                </logger>
                                <logger category="org.jboss.as.ejb3">
                                    <level name="ALL"/>
                                </logger>
                                <logger category="org.jboss.narayana.blacktie">
                                    <level name="ALL"/>
                                </logger>
                                <logger category="org.codehaus.stomp">
                                    <level name="ALL"/>
                                </logger>
                                <logger category="org.hornetq">
                                    <level name="ALL"/>
                                </logger>
                                <root-logger>
                                    <level name="INFO"/>
                                    <handlers>
                                        <handler name="CONSOLE"/>
                                        <handler name="FILE"/>
                                    </handlers>
                                </root-logger>
                                <formatter name="PATTERN">
                                    <pattern-formatter pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
                                </formatter>
                                <formatter name="COLOR-PATTERN">
                                    <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
                                </formatter>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:batch:1.0">
                                <job-repository>
                                    <in-memory/>
                                </job-repository>
                                <thread-pool>
                                    <max-threads count="10"/>
                                    <keepalive-time time="30" unit="seconds"/>
                                </thread-pool>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:bean-validation:1.0"/>
                            <subsystem xmlns="urn:jboss:domain:datasources:3.0">
                                <datasources>
                                    <datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
                                        <connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE</connection-url>
                                        <driver>h2</driver>
                                        <security>
                                            <user-name>sa</user-name>
                                            <password>sa</password>
                                        </security>
                                    </datasource>
                                    <drivers>
                                        <driver name="h2" module="com.h2database.h2">
                                            <xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
                                        </driver>
                                    </drivers>
                                </datasources>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">
                                <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:ee:3.0">
                                <spec-descriptor-property-replacement>false</spec-descriptor-property-replacement>
                                <concurrent>
                                    <context-services>
                                        <context-service name="default" jndi-name="java:jboss/ee/concurrency/context/default" use-transaction-setup-provider="true"/>
                                    </context-services>
                                    <managed-thread-factories>
                                        <managed-thread-factory name="default" jndi-name="java:jboss/ee/concurrency/factory/default" context-service="default"/>
                                    </managed-thread-factories>
                                    <managed-executor-services>
                                        <managed-executor-service name="default" jndi-name="java:jboss/ee/concurrency/executor/default" context-service="default" hung-task-threshold="60000" core-threads="5" max-threads="25" keepalive-time="5000"/>
                                    </managed-executor-services>
                                    <managed-scheduled-executor-services>
                                        <managed-scheduled-executor-service name="default" jndi-name="java:jboss/ee/concurrency/scheduler/default" context-service="default" hung-task-threshold="60000" core-threads="2" keepalive-time="3000"/>
                                    </managed-scheduled-executor-services>
                                </concurrent>
                                <default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/ExampleDS" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:ejb3:3.0">
                                <session-bean>
                                    <stateful default-access-timeout="5000" cache-ref="simple" passivation-disabled-cache-ref="simple"/>
                                    <singleton default-access-timeout="5000"/>
                                </session-bean>
                                <mdb>
                                    <resource-adapter-ref resource-adapter-name="${ejb.resource-adapter-name:hornetq-ra.rar}"/>
                                    <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
                                </mdb>
                                <pools>
                                    <bean-instance-pools>
                                        <strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                                        <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                                    </bean-instance-pools>
                                </pools>
                                <caches>
                                    <cache name="simple"/>
                                    <cache name="distributable" passivation-store-ref="infinispan" aliases="passivating clustered"/>
                                </caches>
                                <passivation-stores>
                                    <passivation-store name="infinispan" cache-container="ejb" max-size="10000"/>
                                </passivation-stores>
                                <async thread-pool-name="default"/>
                                <timer-service thread-pool-name="default" default-data-store="default-file-store">
                                    <data-stores>
                                        <file-data-store name="default-file-store" path="timer-service-data" relative-to="jboss.server.data.dir"/>
                                    </data-stores>
                                </timer-service>
                                <remote connector-ref="http-remoting-connector" thread-pool-name="default"/>
                                <thread-pools>
                                    <thread-pool name="default">
                                        <max-threads count="10"/>
                                        <keepalive-time time="100" unit="milliseconds"/>
                                    </thread-pool>
                                </thread-pools>
                                <iiop enable-by-default="false" use-qualified-name="false"/>
                                <default-security-domain value="other"/>
                                <default-missing-method-permissions-deny-access value="true"/>
                                <log-system-exceptions value="true"/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:io:1.1">
                                <worker name="default"/>
                                <buffer-pool name="default"/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:infinispan:3.0">
                                <cache-container name="server" default-cache="default" module="org.wildfly.clustering.server">
                                    <local-cache name="default">
                                        <transaction mode="BATCH"/>
                                    </local-cache>
                                </cache-container>
                                <cache-container name="web" default-cache="passivation" module="org.wildfly.clustering.web.infinispan">
                                    <local-cache name="passivation">
                                        <transaction mode="BATCH"/>
                                        <file-store passivation="true" purge="false"/>
                                    </local-cache>
                                    <local-cache name="persistent">
                                        <transaction mode="BATCH"/>
                                        <file-store passivation="false" purge="false"/>
                                    </local-cache>
                                </cache-container>
                                <cache-container name="ejb" default-cache="passivation" module="org.wildfly.clustering.ejb.infinispan" aliases="sfsb">
                                    <local-cache name="passivation">
                                        <transaction mode="BATCH"/>
                                        <file-store passivation="true" purge="false"/>
                                    </local-cache>
                                    <local-cache name="persistent">
                                        <transaction mode="BATCH"/>
                                        <file-store passivation="false" purge="false"/>
                                    </local-cache>
                                </cache-container>
                                <cache-container name="hibernate" default-cache="local-query" module="org.hibernate">
                                    <local-cache name="entity">
                                        <transaction mode="NON_XA"/>
                                        <eviction strategy="LRU" max-entries="10000"/>
                                        <expiration max-idle="100000"/>
                                    </local-cache>
                                    <local-cache name="local-query">
                                        <eviction strategy="LRU" max-entries="10000"/>
                                        <expiration max-idle="100000"/>
                                    </local-cache>
                                    <local-cache name="timestamps"/>
                                </cache-container>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:iiop-openjdk:1.0">
                                <orb socket-binding="iiop" ssl-socket-binding="iiop-ssl">
                                    <initializers security="identity" transactions="full"/>
                                </orb>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
                            <subsystem xmlns="urn:jboss:domain:jca:3.0">
                                <archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
                                <bean-validation enabled="true"/>
                                <default-workmanager>
                                    <short-running-threads>
                                        <core-threads count="50"/>
                                        <queue-length count="50"/>
                                        <max-threads count="50"/>
                                        <keepalive-time time="10" unit="seconds"/>
                                    </short-running-threads>
                                    <long-running-threads>
                                        <core-threads count="50"/>
                                        <queue-length count="50"/>
                                        <max-threads count="50"/>
                                        <keepalive-time time="10" unit="seconds"/>
                                    </long-running-threads>
                                </default-workmanager>
                                <cached-connection-manager/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
                            <subsystem xmlns="urn:jboss:domain:jmx:1.3">
                                <expose-resolved-model/>
                                <expose-expression-model/>
                                <remoting-connector/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:jpa:1.1">
                                <jpa default-datasource="" default-extended-persistence-inheritance="DEEP"/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:jsf:1.0"/>
                            <subsystem xmlns="urn:jboss:domain:jsr77:1.0"/>
                            <subsystem xmlns="urn:jboss:domain:mail:2.0">
                                <mail-session name="default" jndi-name="java:jboss/mail/Default">
                                    <smtp-server outbound-socket-binding-ref="mail-smtp"/>
                                </mail-session>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:messaging:3.0">
                                <hornetq-server>
                                    <journal-file-size>102400</journal-file-size>
                    
                    
                                    <connectors>
                                        <http-connector name="http-connector" socket-binding="http">
                                            <param key="http-upgrade-endpoint" value="http-acceptor"/>
                                        </http-connector>
                                        <http-connector name="http-connector-throughput" socket-binding="http">
                                            <param key="http-upgrade-endpoint" value="http-acceptor-throughput"/>
                                            <param key="batch-delay" value="50"/>
                                        </http-connector>
                                        <in-vm-connector name="in-vm" server-id="0"/>
                                    </connectors>
                    
                    
                                    <acceptors>
                                        <http-acceptor http-listener="default" name="http-acceptor"/>
                                        <http-acceptor http-listener="default" name="http-acceptor-throughput">
                                            <param key="batch-delay" value="50"/>
                                            <param key="direct-deliver" value="false"/>
                                        </http-acceptor>
                                        <in-vm-acceptor name="in-vm" server-id="0"/>
                                    </acceptors>
                    
                    
                                    <security-settings>
                                        <security-setting match="jms.queue.BTR_SECURE">
                                            <permission type="send" roles="dynsub"/>
                                            <permission type="consume" roles="dynsub"/>
                                        </security-setting>
                                        <security-setting match="jms.queue.BTR_BTDomainAdmin">
                                            <permission type="send" roles="blacktie"/>
                                            <permission type="consume" roles="blacktie"/>
                                        </security-setting>
                                        <security-setting match="jms.queue.BTR_BTStompAdmin">
                                            <permission type="send" roles="blacktie"/>
                                            <permission type="consume" roles="blacktie"/>
                                        </security-setting>
                                        <security-setting match="#">
                                            <permission type="send" roles="guest"/>
                                            <permission type="consume" roles="guest"/>
                                            <permission type="createNonDurableQueue" roles="guest"/>
                                            <permission type="deleteNonDurableQueue" roles="guest"/>
                                        </security-setting>
                                    </security-settings>
                    
                    
                                    <address-settings>
                                        <address-setting match="#">
                                            <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                                            <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                                            <max-size-bytes>10485760</max-size-bytes>
                                            <page-size-bytes>2097152</page-size-bytes>
                                            <message-counter-history-day-limit>10</message-counter-history-day-limit>
                                        </address-setting>
                                    </address-settings>
                    
                    
                                    <jms-connection-factories>
                                        <connection-factory name="InVmConnectionFactory">
                                            <connectors>
                                                <connector-ref connector-name="in-vm"/>
                                            </connectors>
                                            <entries>
                                                <entry name="java:/ConnectionFactory"/>
                                            </entries>
                                            <client-failure-check-period>86400000</client-failure-check-period>
                                            <connection-ttl>-1</connection-ttl>
                                        </connection-factory>
                                        <connection-factory name="RemoteConnectionFactory">
                                            <connectors>
                                                <connector-ref connector-name="http-connector"/>
                                            </connectors>
                                            <entries>
                                                <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                                            </entries>
                                        </connection-factory>
                                        <pooled-connection-factory name="hornetq-ra">
                                            <transaction mode="xa"/>
                                            <connectors>
                                                <connector-ref connector-name="in-vm"/>
                                            </connectors>
                                            <entries>
                                                <entry name="java:/JmsXA"/>
                                                <entry name="java:jboss/DefaultJMSConnectionFactory"/>
                                            </entries>
                                            <client-failure-check-period>86400000</client-failure-check-period>
                                            <connection-ttl>-1</connection-ttl>
                                        </pooled-connection-factory>
                                    </jms-connection-factories>
                    
                    
                                    <jms-destinations>
                                        <jms-queue name="ExpiryQueue">
                                            <entry name="java:/jms/queue/ExpiryQueue"/>
                                        </jms-queue>
                                        <jms-queue name="DLQ">
                                            <entry name="java:/jms/queue/DLQ"/>
                                        </jms-queue>
                                        <jms-queue name="BTR_TestOne">
                                            <entry name="queue/BTR_TestOne"/>
                                            <entry name="java:jboss/exported/jms/queue/BTR_TestOne"/>
                                        </jms-queue>
                                        <jms-queue name="BTR_TestTwo">
                                            <entry name="queue/BTR_TestTwo"/>
                                            <entry name="java:jboss/exported/jms/queue/BTR_TestTwo"/>
                                        </jms-queue>
                                        <jms-queue name="BTC_ConvService">
                                            <entry name="queue/BTC_ConvService"/>
                                            <entry name="java:jboss/exported/jms/queue/BTC_ConvService"/>
                                        </jms-queue>
                                        <jms-queue name="BTR_JAVA_Converse">
                                            <entry name="queue/BTR_JAVA_Converse"/>
                                            <entry name="java:jboss/exported/jms/queue/BTR_JAVA_Converse"/>
                                        </jms-queue>
                                        <jms-topic name="BTR_JAVA_Topic">
                                            <entry name="topic/BTR_JAVA_Topic"/>
                                            <entry name="java:jboss/exported/jms/topic/BTR_JAVA_Topic"/>
                                        </jms-topic>
                                    </jms-destinations>
                                </hornetq-server>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:naming:2.0">
                                <remote-naming/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
                            <subsystem xmlns="urn:jboss:domain:remoting:3.0">
                                <endpoint worker="default"/>
                                <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:resource-adapters:3.0"/>
                            <subsystem xmlns="urn:jboss:domain:request-controller:1.0"/>
                            <subsystem xmlns="urn:jboss:domain:sar:1.0"/>
                            <subsystem xmlns="urn:jboss:domain:security:1.2">
                                <security-domains>
                                    <security-domain name="other" cache-type="default">
                                        <authentication>
                                            <login-module code="Remoting" flag="optional">
                                                <module-option name="password-stacking" value="useFirstPass"/>
                                            </login-module>
                                            <login-module code="RealmDirect" flag="required">
                                                <module-option name="password-stacking" value="useFirstPass"/>
                                            </login-module>
                                        </authentication>
                                    </security-domain>
                                    <security-domain name="jboss-web-policy" cache-type="default">
                                        <authorization>
                                            <policy-module code="Delegating" flag="required"/>
                                        </authorization>
                                    </security-domain>
                                    <security-domain name="jboss-ejb-policy" cache-type="default">
                                        <authorization>
                                            <policy-module code="Delegating" flag="required"/>
                                        </authorization>
                                    </security-domain>
                                </security-domains>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:transactions:3.0">
                                <core-environment>
                                    <process-id>
                                        <uuid/>
                                    </process-id>
                                </core-environment>
                                <recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
                                <jts/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:undertow:2.0">
                                <buffer-cache name="default"/>
                                <server name="default-server">
                                    <http-listener name="default" socket-binding="http"/>
                                    <host name="default-host" alias="localhost">
                                        <location name="/" handler="welcome-content"/>
                                        <filter-ref name="server-header"/>
                                        <filter-ref name="x-powered-by-header"/>
                                    </host>
                                </server>
                                <servlet-container name="default">
                                    <jsp-config/>
                                    <websockets/>
                                </servlet-container>
                                <handlers>
                                    <file name="welcome-content" path="${jboss.home.dir}/welcome-content"/>
                                </handlers>
                                <filters>
                                    <response-header name="server-header" header-name="Server" header-value="WildFly/9"/>
                                    <response-header name="x-powered-by-header" header-name="X-Powered-By" header-value="Undertow/1"/>
                                </filters>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:webservices:2.0">
                                <wsdl-host>${jboss.bind.address:127.0.0.1}</wsdl-host>
                                <endpoint-config name="Standard-Endpoint-Config"/>
                                <endpoint-config name="Recording-Endpoint-Config">
                                    <pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
                                        <handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
                                    </pre-handler-chain>
                                </endpoint-config>
                                <client-config name="Standard-Client-Config"/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:weld:2.0"/>
                            <subsystem xmlns="urn:jboss:domain:rts:1.0">
                                <servlet server="default-server" host="default-host" socket-binding="http"/>
                            </subsystem>
                            <subsystem xmlns="urn:jboss:domain:blacktie:1.0">
                                <stompconnect connection-factoryname="java:/JmsXA" socket-binding="stomp"/>
                            </subsystem>
                        </profile>
                    
                    
                        <interfaces>
                            <interface name="management">
                                <inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
                            </interface>
                            <interface name="public">
                                <inet-address value="${jboss.bind.address:127.0.0.1}"/>
                            </interface>
                            <interface name="unsecure">
                                <inet-address value="${jboss.bind.address.unsecure:127.0.0.1}"/>
                            </interface>
                        </interfaces>
                    
                    
                        <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
                            <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
                            <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9993}"/>
                            <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
                            <socket-binding name="http" port="${jboss.http.port:8080}"/>
                            <socket-binding name="https" port="${jboss.https.port:8443}"/>
                            <socket-binding name="iiop" interface="unsecure" port="3528"/>
                            <socket-binding name="iiop-ssl" interface="unsecure" port="3529"/>
                            <socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9876}"/>
                            <socket-binding name="stomp" port="61613"/>
                            <socket-binding name="txn-recovery-environment" port="4712"/>
                            <socket-binding name="txn-status-manager" port="4713"/>
                            <outbound-socket-binding name="mail-smtp">
                                <remote-destination host="localhost" port="25"/>
                            </outbound-socket-binding>
                        </socket-binding-group>
                    
                    
                        <deployments>
                            <deployment name="blacktie-admin-services-ear-5.0.5.Final-SNAPSHOT.ear" runtime-name="blacktie-admin-services-ear-5.0.5.Final-SNAPSHOT.ear">
                                <content sha1="34b41da3c26d3335f027be3e22b36953d34510ee"/>
                            </deployment>
                        </deployments>
                    </server>
                    
                    • 7. Re: Re: Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                      jbertram

                      The config looks fine to me.  Any luck with a simple, reproducible test-case?

                      • 8. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                        tomjenkinson

                        Hi,

                         

                        For invm, is it possible that we don't need:

                        <connection-ttl>-1</connection-ttl>

                        anymore, as it looks like the commit (HORNETQ-1314 - disable monitoring for in-vm connections ·  f80610d · hornetq/hornetq · GitHub) should make this unrequired?

                         

                        Also, for invm do you think we no longer need:

                        <client-failure-check-period>86400000</client-failure-check-period>

                         

                        Finally, what is it that causes:

                         

                        WARN [org.hornetq.core.client] (hornetq-failure-check-thread) HQ212037: Connection failure has been detected: HQ119014: Did not receive data from invm:0. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

                        2014-11-19 21:38:59,626 TRACE [org.hornetq.core.server] (hornetq-failure-check-thread) Connection removed 78dbd32a-7034-11e4-8c19-61c30e379d43 from server HornetQServerImpl::serverUUID=98490d2c-7031-11e4-8c19-61c30e379d43: java.lang.Exception: trace

                        at org.hornetq.core.remoting.server.impl.RemotingServiceImpl.connectionDestroyed(RemotingServiceImpl.java:554)

                        at org.hornetq.core.remoting.impl.invm.InVMAcceptor$Listener.connectionDestroyed(InVMAcceptor.java:278)

                        at org.hornetq.core.remoting.impl.invm.InVMConnection.close(InVMConnection.java:118)

                        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.internalClose(RemotingConnectionImpl.java:619)

                        at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:354)

                        at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$FailureCheckAndFlushThread.run(RemotingServiceImpl.java:765)

                         

                        I would imagine that if ttl was -1 it wasn't registered for checking anywhere for failure checking so we could hook up a debugger and see why that is happening if we had some pointers.


                        Thanks,

                        Tom

                        • 9. Re: HornetQNotConnectedException[errorType=NOT_CONNECTED message=HQ119007: Cannot connect to server(s)]
                          jbertram

                          The best place to start is probably the run() method of org.hornetq.core.remoting.server.impl.RemotingServiceImpl.FailureCheckAndFlushThread as it decides which connections should be closed based on activity.