6 Replies Latest reply on Jul 21, 2015 8:37 AM by zzz_public

Timed out waiting to receive initial broadcast from cluster

hardy.ferentschik Jul 14, 2015 11:46 AM

Hi,

I have a HornetQ server running within a Wildfly 9 instance. Its configuration is:

<hornetq-server name="live">
    <persistence-enabled>true</persistence-enabled>
    <security-enabled>false</security-enabled>
    <cluster-user>HORNETQ.CLUSTER</cluster-user>
    <cluster-password>CHANGE ME!!</cluster-password>
    <statistics-enabled>true</statistics-enabled>
    <message-counter-sample-period>1000</message-counter-sample-period>
    <message-counter-max-day-history>3</message-counter-max-day-history>
    <backup>false</backup>
    <allow-failback>true</allow-failback>
    <failover-on-shutdown>false</failover-on-shutdown>
    <shared-store>false</shared-store>
    <journal-type>NIO</journal-type>
    <journal-file-size>102400</journal-file-size>
    <check-for-live-server>true</check-for-live-server>
    <backup-group-name>${hornetq.live.group.name}</backup-group-name>
    <paging-directory path="data/jms/live/paging"/>
    <bindings-directory path="data/jms/live/bindings"/>
    <journal-directory path="data/jms/live/journal"/>
    <large-messages-directory path="data/jms/live/large-messages"/>
    <connectors>
        <netty-connector name="netty" socket-binding="messaging">
            <param key="use-nio" value="true"/>
        </netty-connector>
    </connectors>
    <acceptors>
        <netty-acceptor name="netty" socket-binding="messaging">
            <param key="use-nio" value="true"/>
        </netty-acceptor>
    </acceptors>
    <broadcast-groups>
        <broadcast-group name="bg-group1">
            <socket-binding>messaging-group</socket-binding>
            <connector-ref>netty</connector-ref>
        </broadcast-group>
    </broadcast-groups>
    <discovery-groups>
        <discovery-group name="dg-group1">
            <socket-binding>messaging-group</socket-binding>
        </discovery-group>
    </discovery-groups>
    <cluster-connections>
        <cluster-connection name="my-cluster">
            <address>jms</address>
            <connector-ref>netty</connector-ref>
            <discovery-group-ref discovery-group-name="dg-group1"/>
        </cluster-connection>
    </cluster-connections>
    <security-settings>
        <security-setting match="#">
            <permission roles="guest" type="send"/>
            <permission roles="guest" type="consume"/>
            <permission roles="guest" type="createNonDurableQueue"/>
            <permission roles="guest" type="deleteNonDurableQueue"/>
        </security-setting>
    </security-settings>
    <address-settings>
        <address-setting match="#">
            <dead-letter-address>jms.queue.DLQ</dead-letter-address>
            <expiry-address>jms.queue.ExpiryQueue</expiry-address>
            <max-size-bytes>10485760</max-size-bytes>
            <page-size-bytes>2097152</page-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <redistribution-delay>1000</redistribution-delay>
        </address-setting>
    </address-settings>
    <jms-destinations>
        <jms-queue name="test queue">
            <entry name="/jms/queue/test"/>
            <entry name="java:jboss/exported/jms/queue/test"/>
        </jms-queue>
    </jms-destinations>
</hornetq-server>

In the end I want this to be part of a symmetrical HA JMS cluster between two Wildfly instances, but for now I am just starting this server.

I also have a client app which I want to use to connect to using an EmbeddedJMS instance. I am using the following configuration:

hornetq-configuration.xml

<persistence-enabled>false</persistence-enabled>
 <security-enabled>false</security-enabled>
 <journal-file-size>102400</journal-file-size>
 <journal-min-files>1</journal-min-files>

 <cluster-user>HORNETQ.CLUSTER</cluster-user>
 <cluster-password>CHANGE ME!!</cluster-password>

 <connectors>
     <connector name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyConnectorFactory</factory-class>
         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
         <param key="port" value="${hornetq.remoting.netty.port:5555}"/>
         <param key="use-nio" value="true"/>
     </connector>
 </connectors>

 <acceptors>
     <acceptor name="netty">
         <factory-class>org.hornetq.core.remoting.impl.netty.NettyAcceptorFactory</factory-class>
         <param key="host" value="${hornetq.remoting.netty.host:localhost}"/>
         <param key="port" value="${hornetq.remoting.netty.port:5555}"/>
         <param key="use-nio" value="true"/>
     </acceptor>
 </acceptors>

 <broadcast-groups>
     <broadcast-group name="bg-group1">
         <group-address>231.7.7.7</group-address>
         <group-port>9876</group-port>
         <!--<broadcast-period>5000</broadcast-period>-->
         <connector-ref>netty</connector-ref>
     </broadcast-group>
 </broadcast-groups>

 <discovery-groups>
     <discovery-group name="dg-group1">
         <group-address>231.7.7.7</group-address>
         <group-port>9876</group-port>
         <!--<refresh-timeout>10000</refresh-timeout>-->
     </discovery-group>
 </discovery-groups>

 <cluster-connections>
     <cluster-connection name="my-cluster">
         <address>jms</address>
         <connector-ref>netty</connector-ref>
         <discovery-group-ref discovery-group-name="dg-group1"/>
     </cluster-connection>
 </cluster-connections>

 <security-settings>
     <security-setting match="#">
         <permission type="send" roles="guest"/>
         <permission type="consume" roles="guest"/>
         <permission type="createNonDurableQueue" roles="guest"/>
         <permission type="deleteNonDurableQueue" roles="guest"/>
     </security-setting>
 </security-settings>

 <address-settings>
     <address-setting match="jms.#">
         <dead-letter-address>jms.queue.DLQ</dead-letter-address>
         <expiry-address>jms.queue.ExpiryQueue</expiry-address>
         <redelivery-delay>0</redelivery-delay>
         <max-size-bytes>10485760</max-size-bytes>
         <page-size-bytes>2097152</page-size-bytes>
         <address-full-policy>PAGE</address-full-policy>
         <message-counter-history-day-limit>10</message-counter-history-day-limit>
         <redistribution-delay>1000</redistribution-delay>
     </address-setting>
 </address-settings>

hornetq-jms.xml

 <connection-factory name="ConnectionFactory">
     <discovery-group-ref discovery-group-name="dg-group1"/>
     <entries>
         <entry name="ConnectionFactory"/>
     </entries>
 </connection-factory>

It seems the two server connect. I get for example this in the Wildfly log:

17:19:29,831 INFO  [org.hornetq.core.server] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=50f5b655-261f-11e5-834f-8375d06ca8cd-1757495714)) HQ221027: Bridge ClusterConnectionBridge@c3eab6 [name=sf.my-cluster.092db3b1-24c7-11e5-88d3-ef26683f74ff, queue=QueueImpl[name=sf.my-cluster.092db3b1-24c7-11e5-88d3-ef26683f74ff, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=50f5b655-261f-11e5-834f-8375d06ca8cd]]@65c3fe73 targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@c3eab6 [name=sf.my-cluster.092db3b1-24c7-11e5-88d3-ef26683f74ff, queue=QueueImpl[name=sf.my-cluster.092db3b1-24c7-11e5-88d3-ef26683f74ff, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=50f5b655-261f-11e5-834f-8375d06ca8cd]]@65c3fe73 targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5555&use-nio=true&host=localhost], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@493642740[nodeUUID=50f5b655-261f-11e5-834f-8375d06ca8cd, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?use-nio=true&port=5445&host=localhost, address=jms, server=HornetQServerImpl::serverUUID=50f5b655-261f-11e5-834f-8375d06ca8cd])) [initialConnectors=[TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5555&use-nio=true&host=localhost], discoveryGroupConfiguration=null]] is connected

The problem occurs when I retrieve the connection factory defined in hornetq-jms.xml. I keep getting a timeout exception:

javax.jms.JMSRuntimeException: Failed to create session factory
    at org.hornetq.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:98)
    at org.hornetq.jms.client.HornetQConnectionFactory.createContext(HornetQConnectionFactory.java:149)
    at org.hornetq.jms.client.HornetQConnectionFactory.createContext(HornetQConnectionFactory.java:130)
    at org.jboss.as.quickstarts.jms.JMSClient.sendMessage(JMSClient.java:140)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at asg.cliche.ShellCommand.invoke(ShellCommand.java:69)
    at asg.cliche.Shell.processCommand(Shell.java:294)
    at asg.cliche.Shell.processLine(Shell.java:276)
    at asg.cliche.Shell.commandLoop(Shell.java:231)
    at org.jboss.as.quickstarts.jms.JMSClient.main(JMSClient.java:173)
Caused by: javax.jms.JMSException: Failed to create session factory
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:673)
    at org.hornetq.jms.client.HornetQConnectionFactory.createContext(HornetQConnectionFactory.java:140)
    ... 11 more
Caused by: HornetQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=HQ119012: Timed out waiting to receive initial broadcast from cluster]
    at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:839)
    at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:669)
    ... 12 more

I've seen similar posts and questions and often the problem is about UDP networking and multicasts. I am on Mac and added the multicast routes for the loopback device. Using sudo tcpdump -i lo0 'ip[16]>=231' I can also monitor multicast traffic and I can see traffic from both servers, but still I get this timeout. I also tried a programmatic session connection like this:

final String groupAddress = "231.7.7.7";
final int groupPort = 9876;
ServerLocator locator = HornetQClient.createServerLocatorWithHA(
   new DiscoveryGroupConfiguration(
  groupAddress, groupPort,
   10000,
   new UDPBroadcastGroupConfiguration( groupAddress, groupPort, null, -1 )
  )
);
ClientSessionFactory factory = locator.createSessionFactory();
ClientSession session1 = factory.createSession();

Same effect. The HornetQ version I am using is 2.4.7.Final (Wildfly and Client app). I am running out of ideas regarding the cause of the problem. Any help appreciated.

1. Re: Timed out waiting to receive initial broadcast from cluster

gaohoward Jul 14, 2015 8:46 PM (in response to hardy.ferentschik)

what are the values of the socket-bindings:
socket-binding="messaging"
socket-binding="messaging-group"

in your wildfly config?
Actions
2. Re: Timed out waiting to receive initial broadcast from cluster

hardy.ferentschik Jul 15, 2015 3:45 AM (in response to gaohoward)

<socket-binding multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9876}" name="messaging-group" port="0"/>
<socket-binding name="messaging" port="5445"/>
Actions
3. Re: Timed out waiting to receive initial broadcast from cluster

gaohoward Jul 15, 2015 9:54 AM (in response to hardy.ferentschik)

hmm, turns out some default values are different in wildfly than in standalone hornetq. I tried

new UDPBroadcastGroupConfiguration(groupAddress, groupPort, "127.0.0.1", 0)

with your client code and I'm able to connect to wildfly.
Actions
4. Re: Timed out waiting to receive initial broadcast from cluster

gaohoward Jul 15, 2015 10:26 AM (in response to gaohoward)

well actually it's not default values for port, as you can see the port is explicitly set to '0' in your config.
Actions
5. Re: Timed out waiting to receive initial broadcast from cluster

hardy.ferentschik Jul 20, 2015 12:10 PM (in response to gaohoward)

I got my code to work as well now. However, as you say, there is a inconsistency somewhere. Programmatically I can do:

new UDPBroadcastGroupConfiguration(groupAddress, groupPort, "127.0.0.1", 0)

but for the XML config of local-bind-port '0' is not a valid value. -1 has to be used. Also in the Wildfly socket binding configuration '0' is used. I guess I got confused there. Using

<broadcast-groups>
    <broadcast-group name="bg-group1">
        <local-bind-address>localhost</local-bind-address>
        <local-bind-port>-1</local-bind-port>
        <group-address>231.7.7.7</group-address>
        <group-port>9876</group-port>
        <connector-ref>netty</connector-ref>
    </broadcast-group>
</broadcast-groups>

<discovery-groups>
    <discovery-group name="dg-group1">
        <local-bind-address>localhost</local-bind-address>
        <local-bind-port>-1</local-bind-port>
        <group-address>231.7.7.7</group-address>
        <group-port>9876</group-port>
    </discovery-group>
</discovery-groups>

things also work via the factory bound to JNDI.
Actions
6. Re: Timed out waiting to receive initial broadcast from cluster

zzz_public Jul 21, 2015 8:37 AM (in response to hardy.ferentschik)

If you want to have cluster I think you have to specify hosta and ports for connectors, acceptors, etc in initial configuration <hornetq-server name="live">...
Actions

Go to original post