6 Replies Latest reply on Jul 21, 2015 8:37 AM by Andrey zzz_public

    Timed out waiting to receive initial broadcast from cluster

    Hardy Ferentschik Novice

      Hi,

       

      I have a HornetQ server running within a Wildfly 9 instance. Its configuration is:

       

      <hornetq-server name="live">
          <persistence-enabled>true</persistence-enabled>
          <security-enabled>false</security-enabled>
          <cluster-user>HORNETQ.CLUSTER</cluster-user>
          <cluster-password>CHANGE ME!!</cluster-password>
          <statistics-enabled>true</statistics-enabled>
          <message-counter-sample-period>1000</message-counter-sample-period>
          <message-counter-max-day-history>3</message-counter-max-day-history>
          <backup>false</backup>
          <allow-failback>true</allow-failback>
          <failover-on-shutdown>false</failover-on-shutdown>
          <shared-store>false</shared-store>
          <journal-type>NIO</journal-type>
          <journal-file-size>102400</journal-file-size>
          <check-for-live-server>true</check-for-live-server>
          <backup-group-name>${hornetq.live.group.name}</backup-group-name>
          <paging-directory path="data/jms/live/paging"/>
          <bindings-directory path="data/jms/live/bindings"/>
          <journal-directory path="data/jms/live/journal"/>
          <large-messages-directory path="data/jms/live/large-messages"/>
          <connectors>
              <netty-connector name="netty" socket-binding="messaging">
                  <param key="use-nio" value="true"/>
              </netty-connector>
          </connectors>
          <acceptors>
              <netty-acceptor name="netty" socket-binding="messaging">
                  <param key="use-nio" value="true"/>
              </netty-acceptor>
          </acceptors>
          <broadcast-groups>
              <broadcast-group name="bg-group1">
                  <socket-binding>messaging-group</socket-binding>
                  <connector-ref>netty</connector-ref>
              </broadcast-group>
          </broadcast-groups>
          <discovery-groups>
              <discovery-group name="dg-group1">
                  <socket-binding>messaging-group</socket-binding>
              </discovery-group>
          </discovery-groups>
          <cluster-connections>
              <cluster-connection name="my-cluster">
                  <address>jms</address>
                  <connector-ref>netty</connector-ref>
                  <discovery-group-ref discovery-group-name="dg-group1"/>
              </cluster-connection>
          </cluster-connections>
          <security-settings>
              <security-setting match="#">
                  <permission roles="guest" type="send"/>
                  <permission roles="guest" type="consume"/>
                  <permission roles="guest" type="createNonDurableQueue"/>
                  <permission roles="guest" type="deleteNonDurableQueue"/>
              </security-setting>
          </security-settings>
          <address-settings>
              <address-setting match="#">
                  <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                  <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                  <max-size-bytes>10485760</max-size-bytes>
                  <page-size-bytes>2097152</page-size-bytes>
                  <message-counter-history-day-limit>10</message-counter-history-day-limit>
                  <redistribution-delay>1000</redistribution-delay>
              </address-setting>
          </address-settings>
          <jms-destinations>
              <jms-queue name="test queue">
                  <entry name="/jms/queue/test"/>
                  <entry name="java:jboss/exported/jms/queue/test"/>
              </jms-queue>
          </jms-destinations>
      </hornetq-server>
      

       

      In the end I want this to be part of a symmetrical HA JMS cluster between two Wildfly instances, but for now I am just starting this server.

       

      I also have a client app which I want to use to connect to using an EmbeddedJMS instance. I am using the following configuration:


      hornetq-configuration.xml


      <persistence-enabled>false</persistence-enabled>
       <security-enabled>false</security-enabled>
       <journal-file-size>102400</journal-file-size>
       <journal-min-files>1</journal-min-files>
      
       <cluster-user>HORNETQ.CLUSTER</cluster-user>
       <cluster-password>CHANGE ME!!</cluster-password>
      
       <connectors>
           <connector name="netty">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
               <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
               <param key="port" value="${hornetq.remoting.netty.port:5555}"/>
               <param key="use-nio" value="true"/>
           </connector>
       </connectors>
      
       <acceptors>
           <acceptor name="netty">
               <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
               <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
               <param key="port" value="${hornetq.remoting.netty.port:5555}"/>
               <param key="use-nio" value="true"/>
           </acceptor>
       </acceptors>
      
       <broadcast-groups>
           <broadcast-group name="bg-group1">
               <group-address>231.7.7.7</group-address>
               <group-port>9876</group-port>
               <!--<broadcast-period>5000</broadcast-period>-->
               <connector-ref>netty</connector-ref>
           </broadcast-group>
       </broadcast-groups>
      
       <discovery-groups>
           <discovery-group name="dg-group1">
               <group-address>231.7.7.7</group-address>
               <group-port>9876</group-port>
               <!--<refresh-timeout>10000</refresh-timeout>-->
           </discovery-group>
       </discovery-groups>
      
       <cluster-connections>
           <cluster-connection name="my-cluster">
               <address>jms</address>
               <connector-ref>netty</connector-ref>
               <discovery-group-ref discovery-group-name="dg-group1"/>
           </cluster-connection>
       </cluster-connections>
      
       <security-settings>
           <security-setting match="#">
               <permission type="send" roles="guest"/>
               <permission type="consume" roles="guest"/>
               <permission type="createNonDurableQueue" roles="guest"/>
               <permission type="deleteNonDurableQueue" roles="guest"/>
           </security-setting>
       </security-settings>
      
       <address-settings>
           <address-setting match="jms.#">
               <dead-letter-address>jms.queue.DLQ</dead-letter-address>
               <expiry-address>jms.queue.ExpiryQueue</expiry-address>
               <redelivery-delay>0</redelivery-delay>
               <max-size-bytes>10485760</max-size-bytes>
               <page-size-bytes>2097152</page-size-bytes>
               <address-full-policy>PAGE</address-full-policy>
               <message-counter-history-day-limit>10</message-counter-history-day-limit>
               <redistribution-delay>1000</redistribution-delay>
           </address-setting>
       </address-settings>
      

       

      hornetq-jms.xml

       <connection-factory name="ConnectionFactory">
           <discovery-group-ref discovery-group-name="dg-group1"/>
           <entries>
               <entry name="ConnectionFactory"/>
           </entries>
       </connection-factory>
      

       

      It seems the two server connect. I get for example this in the Wildfly log:

       

      17:19:29,831 INFO  [org.hornetq.core.server] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=50f5b655-261f-11e5-834f-8375d06ca8cd-1757495714)) HQ221027: Bridge ClusterConnectionBridge@c3eab6 [name=sf.my-cluster.092db3b1-24c7-11e5-88d3-ef26683f74ff, queue=QueueImpl[name=sf.my-cluster.092db3b1-24c7-11e5-88d3-ef26683f74ff, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=50f5b655-261f-11e5-834f-8375d06ca8cd]]@65c3fe73 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@c3eab6 [name=sf.my-cluster.092db3b1-24c7-11e5-88d3-ef26683f74ff, queue=QueueImpl[name=sf.my-cluster.092db3b1-24c7-11e5-88d3-ef26683f74ff, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=50f5b655-261f-11e5-834f-8375d06ca8cd]]@65c3fe73 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5555&use-nio=true&host=localhost], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@493642740[nodeUUID=50f5b655-261f-11e5-834f-8375d06ca8cd, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?use-nio=true&port=5445&host=localhost, address=jms, server=HornetQServerImpl::serverUUID=50f5b655-261f-11e5-834f-8375d06ca8cd])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5555&use-nio=true&host=localhost], discoveryGroupConfiguration=null]] is connected
      

       

      The problem occurs when I retrieve the connection factory defined in hornetq-jms.xml. I keep getting a timeout exception:

       

      javax.jms.JMSRuntimeException: Failed to create session factory
          at org.hornetq.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:98)
          at org.hornetq.jms.client.HornetQConnectionFactory.createContext(HornetQConnectionFactory.java:149)
          at org.hornetq.jms.client.HornetQConnectionFactory.createContext(HornetQConnectionFactory.java:130)
          at org.jboss.as.quickstarts.jms.JMSClient.sendMessage(JMSClient.java:140)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:483)
          at asg.cliche.ShellCommand.invoke(ShellCommand.java:69)
          at asg.cliche.Shell.processCommand(Shell.java:294)
          at asg.cliche.Shell.processLine(Shell.java:276)
          at asg.cliche.Shell.commandLoop(Shell.java:231)
          at org.jboss.as.quickstarts.jms.JMSClient.main(JMSClient.java:173)
      Caused by: javax.jms.JMSException: Failed to create session factory
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:673)
          at org.hornetq.jms.client.HornetQConnectionFactory.createContext(HornetQConnectionFactory.java:140)
          ... 11 more
      Caused by: HornetQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=HQ119012: Timed out waiting to receive initial broadcast from cluster]
          at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:839)
          at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:669)
          ... 12 more
      

       

      I've seen similar posts and questions and often the problem is about UDP networking and multicasts. I am on Mac and added the multicast routes for the loopback device. Using sudo tcpdump -i lo0 'ip[16]>=231' I can also monitor multicast traffic and I can see traffic from both servers, but still I get this timeout. I also tried a programmatic session connection like this:

       

      final String groupAddress = "231.7.7.7";
      final int groupPort = 9876;
      ServerLocator locator = HornetQClient.createServerLocatorWithHA(
         new DiscoveryGroupConfiguration(
        groupAddress, groupPort,
         10000,
         new UDPBroadcastGroupConfiguration( groupAddress, groupPort, null, -1 )
        )
      );
      ClientSessionFactory factory = locator.createSessionFactory();
      ClientSession session1 = factory.createSession();
      

       

      Same effect. The HornetQ version I am using is 2.4.7.Final (Wildfly and Client app). I am running out of ideas regarding the cause of the problem. Any help appreciated.