12 Replies Latest reply on Sep 29, 2016 10:08 AM by jbertram

Embedded HornetQ Clustering in AWS using JGroup-S3_PING

musheka Mar 7, 2016 2:59 PM

Hi All,

I have setup two standalone instances of our application in wildfly 9.0.1 in two private subnets(in 2 different AZ's) of a VPC in AWS. We use embedded HornetQ server in the standalone.xml.

We have setup JGroup with S3_PING as a protocol for broadcast and discovery.However the clustering of these horbetQ servers in these two instances is failing with timeout errors.

We have ensured that we can ping,telnet,curl(http) from each of these instances to another. Strange that the hornetq-clustering is alone failing.I have included the Jgroup and messaging section of the standalone.xml from each of these machines. I have also included some hornetQ debug log.I can see connection attempts are being made from each of the machine instance. The cluster connection unfortunately doesn't happen. I can also see the entries being clearly made in the s3 bucket in AWS. Both the hostnames are included there.

Jgroup setup in both the instances:

<property name="access_key">xxxxxxx</property>

<property name="secret_access_key">xxxxx</property>

</protocol>

<property name="use_mcast_xmit">false</property>

<property name="use_mcast_xmit_req">false</property>

</protocol>

</stack>

</stacks>

Machine-1-10.16.0.91-Messaging section of standalone.xml is attached

Machine-2-10.16.4.28-Messaging section of standalone.xml is attached.

DEBUG LOG from both machines is attached.

Screen shot of AWS S3 entry is also attached.

1. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

jbertram Mar 8, 2016 10:20 AM (in response to musheka)

I'd post your question to the JGroups forum. This looks like a JGroups issue rather than a HornetQ one. I realize it is ultimately impacting the ability of the HornetQ server to form a cluster, but the JGroups community would be able to assist you better with this particular issue since it is JGroups that is emitting these warning messages. I don't see any logging directly related to HornetQ. As a HornetQ developer there's just not much I can do with what you've provided.
Actions
2. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

musheka Mar 8, 2016 8:35 PM (in response to jbertram)

Thanks for your reply Justin. To avoid Jgroups, I set up the two standalone instances to use static connectors for the HORNETQ clustering. However during the cluster establishment, the connection is failing due to exceptions. I turned the TRACE ON and collected the below logs.Any suggestion based on the below logs should be helpful. I will also send the standalone xml for these instances. Not sure why HornetQ is also trying to make the connection at port 10.16.0.91:57207 when the http port setup in standalone.xml was 5447. The port 5447 was readily available for listening as shown in the attached screenshot.
STACK TRACE:--

2016-03-08 22:47:12,882 DEBUG [org.hornetq.core.server] (ServerService Thread Pool -- 58) ClusterManagerImpl[server=HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2]@1211249137 defining cluster connection towards [TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91]

2016-03-08 22:47:17,092 DEBUG [org.hornetq.core.server] (ServerService Thread Pool -- 58) ClusterConnectionImpl@1513487194[nodeUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2, connector=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5447&host=10-16-0-91&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, address=jms, server=HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2]Creating a serverLocator for [TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91]

2016-03-08 22:47:17,146 DEBUG [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector@7fa85188::Submitting connect towards Connector [initialConnector=TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91]

2016-03-08 22:47:17,146 DEBUG [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) Connector [initialConnector=TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91]::Trying to connect to ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91, backupConfig=null]

2016-03-08 22:47:17,151 DEBUG [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) Trying to connect with connector = org.hornetq.core.remoting.impl.netty.NettyConnectorFactory@32d78717, parameters = {port=5448, host=10.16.4.28, http-upgrade-enabled=true, http-upgrade-endpoint=http-acceptor, local-address=10.16.0.91} connector = NettyConnector [host=10.16.4.28, port=5448, httpEnabled=false, httpUpgradeEnabled=true, useServlet=false, servletPath=/messaging/HornetQServlet, sslEnabled=false, useNio=true]

2016-03-08 22:47:17,263 DEBUG [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) Trying to connect at the main server using connector :TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91

2016-03-08 22:47:17,263 DEBUG [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) Remote destination: /10.16.4.28:5448
Host: 10.16.4.28
2016-03-08 22:47:17,659 TRACE [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91, backupConfig=null]::Subscribing Topology
2016-03-08 22:47:17,664 TRACE [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) returning RemotingConnectionImpl [clientID=null, nodeID=null, transportConnection=org.hornetq.core.remoting.impl.netty.NettyConnection@5a829cee[local= /10.16.0.91:57207, remote=/10.16.4.28:5448]]

2016-03-08 22:47:17,665 TRACE [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) returning RemotingConnectionImpl [clientID=null, nodeID=null, transportConnection=org.hornetq.core.remoting.impl.netty.NettyConnection@5a829cee[local= /10.16.0.91:57207, remote=/10.16.4.28:5448]]

2016-03-08 22:47:17,671 DEBUG [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) Returning ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91, backupConfig=null] after 1 retries on StaticConnector ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null]

2016-03-08 22:47:17,671 DEBUG [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) ClientSessionFactoryImpl received backup update for live/backup pair = TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91 / null but it didn't belong to TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91
serverLocator=ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null]: java.lang.Exception

2016-03-08 22:47:17,675 TRACE [org.hornetq.core.client] (Thread-0 (HornetQ-client-global-threads-1898637621)) NodeUp ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null]::nodeID=3b0c38c2-dca1-11e5-9089-b92605690ef2, connectorPair=Pair[a=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5447&host=10-16-0-91&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, b=null]: java.lang.Exception: trace
serverLocator=ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null]: java.lang.Exception

2016-03-08 22:47:17,678 TRACE [org.hornetq.core.client] (Thread-0 (HornetQ-client-global-threads-1898637621)) NodeUp ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null]::nodeID=3b0c38c2-dca1-11e5-9089-b92605690ef2, connectorPair=Pair[a=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5447&host=10-16-0-91&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, b=null]: java.lang.Exception: trace
Actions
3. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

musheka Mar 8, 2016 9:28 PM (in response to musheka)

Justin- I am including the messaging and socket binding configuration for both the instances below.

messaging and socket binding config - 10.16.0.91
        <subsystem xmlns="urn:jboss:domain:messaging:3.0">
            <hornetq-server>
                <clustered>true</clustered>
                <persistence-enabled>true</persistence-enabled>
                <security-enabled>false</security-enabled>
                <journal-min-files>10</journal-min-files>
                <paging-directory path="/docs/root/esp-etl-nr/standalone/data/hornetq-persistence/paging"/>
                <bindings-directory path="/docs/root/esp-etl-nr/standalone/data/hornetq-persistence/bindings"/>
                <journal-directory path="/docs/root/esp-etl-nr/standalone/data/hornetq-persistence/journal"/>

                <connectors>
                    <http-connector name="http-connector" socket-binding="http">
                        <param key="http-upgrade-endpoint" value="http-acceptor"/>
                    </http-connector>
                    <http-connector name="server1-connector" socket-binding="httpmessaging1">
                        <param key="http-upgrade-endpoint" value="http-acceptor"/>
                    </http-connector>
                    <in-vm-connector name="in-vm" server-id="0"/>
                </connectors>

                <acceptors>
                    <http-acceptor http-listener="default" name="http-acceptor"/>
                    <in-vm-acceptor name="in-vm" server-id="0"/>
                </acceptors>

                <cluster-connections>
                    <cluster-connection name="my-cluster">
                        <address>jms</address>
                        <connector-ref>http-connector</connector-ref>
                        <check-period>1000</check-period>
                        <connection-ttl>1000</connection-ttl>
                        <retry-interval>500</retry-interval>
                        <reconnect-attempts>3</reconnect-attempts>
                        <use-duplicate-detection>true</use-duplicate-detection>
                        <forward-when-no-consumers>false</forward-when-no-consumers>
                        <static-connectors>
                            <connector-ref>
                                server1-connector
                            </connector-ref>
                        </static-connectors>
                    </cluster-connection>
                </cluster-connections>

                <security-settings>
                    <security-setting match="#">
                        <permission type="send" roles="guest"/>
                        <permission type="consume" roles="guest"/>
                        <permission type="createNonDurableQueue" roles="guest"/>
                        <permission type="deleteNonDurableQueue" roles="guest"/>
                    </security-setting>
                </security-settings>

                <address-settings>
                    <address-setting match="#">
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <max-size-bytes>10485760</max-size-bytes>
                        <page-size-bytes>2097152</page-size-bytes>
                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                        <redistribution-delay>1000</redistribution-delay>
                    </address-setting>
                </address-settings>

                <jms-connection-factories>
                    <connection-factory name="InVmConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/ConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <connection-factory name="RemoteConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="http-connector"/>
                        </connectors>
                        <entries>
                            <entry name="RemoteConnectionFactory"/>
                            <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <pooled-connection-factory name="hornetq-ra">
                        <transaction mode="xa"/>
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/JmsXA"/>
                            <entry name="java:jboss/DefaultJMSConnectionFactory"/>
                        </entries>
                    </pooled-connection-factory>
                </jms-connection-factories>

                <jms-destinations>
                    <jms-queue name="ESP-Queue">
                        <entry name="java:/queue/EspQueue"/>
                        <entry name="java:jboss/exported/jms/queue/EspQueue"/>
                        <durable>true</durable>
                    </jms-queue>
                    <jms-queue name="HighPriorityESP-Queue">
                        <entry name="java:/queue/HighPriorityEspQueue"/>
                        <entry name="java:jboss/exported/jms/queue/HighPriorityEspQueue"/>
                        <durable>true</durable>
                    </jms-queue>
                    <jms-queue name="ESP-Incremental-Queue">
                        <entry name="java:/queue/EspIncrementalQueue"/>
                        <entry name="java:jboss/exported/jms/queue/EspIncrementalQueue"/>
                        <durable>true</durable>
                    </jms-queue>
                </jms-destinations>
            </hornetq-server>
        </subsystem>

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
        <socket-binding name="ajp" port="10204"/>
        <socket-binding name="http" port="5447"/>
        <socket-binding name="https" port="10206"/>
        <socket-binding name="jacorb" interface="unsecure" port="10207"/>
        <socket-binding name="jacorb-ssl" interface="unsecure" port="10208"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="10209"/>
        <socket-binding name="jgroups-tcp-fd" port="10210"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="10211"/>
        <socket-binding name="messaging" port="10212"/>
        <socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9877}"/>
        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
        <socket-binding name="txn-recovery-environment" port="10216"/>
        <socket-binding name="txn-status-manager" port="10217"/>
        <outbound-socket-binding name="httpmessaging1">
            <remote-destination host="10.16.4.28" port="5448"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
    </socket-binding-group>

messaging and socket binding config - 10.16.4.28
<subsystem xmlns="urn:jboss:domain:messaging:3.0">
            <hornetq-server>
                <clustered>true</clustered>
                <persistence-enabled>true</persistence-enabled>
                <security-enabled>false</security-enabled>
                <journal-min-files>10</journal-min-files>
                <paging-directory path="/docs/root/esp-etl-nr/standalone/data/hornetq-persistence/paging"/>
                <bindings-directory path="/docs/root/esp-etl-nr/standalone/data/hornetq-persistence/bindings"/>
                <journal-directory path="/docs/root/esp-etl-nr/standalone/data/hornetq-persistence/journal"/>

                <connectors>
                    <http-connector name="http-connector" socket-binding="http">
                        <param key="http-upgrade-endpoint" value="http-acceptor"/>
                    </http-connector>
                    <http-connector name="server0-connector" socket-binding="httpmessaging1">
                        <param key="http-upgrade-endpoint" value="http-acceptor"/>
                    </http-connector>
                    <in-vm-connector name="in-vm" server-id="0"/>
                </connectors>

                <acceptors>
                    <http-acceptor http-listener="default" name="http-acceptor"/>
                    <in-vm-acceptor name="in-vm" server-id="0"/>
                </acceptors>

                <cluster-connections>
                    <cluster-connection name="my-cluster">
                        <address>jms</address>
                        <connector-ref>http-connector</connector-ref>
                        <check-period>1000</check-period>
                        <connection-ttl>1000</connection-ttl>
                        <retry-interval>500</retry-interval>
                        <reconnect-attempts>3</reconnect-attempts>
                        <use-duplicate-detection>true</use-duplicate-detection>
                        <forward-when-no-consumers>false</forward-when-no-consumers>
                        <static-connectors>
                            <connector-ref>
                                server0-connector
                            </connector-ref>
                        </static-connectors>
                    </cluster-connection>
                </cluster-connections>

                <security-settings>
                    <security-setting match="#">
                        <permission type="send" roles="guest"/>
                        <permission type="consume" roles="guest"/>
                        <permission type="createNonDurableQueue" roles="guest"/>
                        <permission type="deleteNonDurableQueue" roles="guest"/>
                    </security-setting>
                </security-settings>

                <address-settings>
                    <address-setting match="#">
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <max-size-bytes>10485760</max-size-bytes>
                        <page-size-bytes>2097152</page-size-bytes>
                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                        <redistribution-delay>1000</redistribution-delay>
                    </address-setting>
                </address-settings>

                <jms-connection-factories>
                    <connection-factory name="InVmConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/ConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <connection-factory name="RemoteConnectionFactory">
                        <connectors>
                            <connector-ref connector-name="http-connector"/>
                        </connectors>
                        <entries>
                            <entry name="RemoteConnectionFactory"/>
                            <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
                        </entries>
                    </connection-factory>
                    <pooled-connection-factory name="hornetq-ra">
                        <transaction mode="xa"/>
                        <connectors>
                            <connector-ref connector-name="in-vm"/>
                        </connectors>
                        <entries>
                            <entry name="java:/JmsXA"/>
                            <entry name="java:jboss/DefaultJMSConnectionFactory"/>
                        </entries>
                    </pooled-connection-factory>
                </jms-connection-factories>

                <jms-destinations>
                    <jms-queue name="ESP-Queue">
                        <entry name="java:/queue/EspQueue"/>
                        <entry name="java:jboss/exported/jms/queue/EspQueue"/>
                        <durable>true</durable>
                    </jms-queue>
                    <jms-queue name="HighPriorityESP-Queue">
                        <entry name="java:/queue/HighPriorityEspQueue"/>
                        <entry name="java:jboss/exported/jms/queue/HighPriorityEspQueue"/>
                        <durable>true</durable>
                    </jms-queue>
                    <jms-queue name="ESP-Incremental-Queue">
                        <entry name="java:/queue/EspIncrementalQueue"/>
                        <entry name="java:jboss/exported/jms/queue/EspIncrementalQueue"/>
                        <durable>true</durable>
                    </jms-queue>
                </jms-destinations>
            </hornetq-server>
        </subsystem>

<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
        <socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
        <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
        <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
        <socket-binding name="ajp" port="10204"/>
        <socket-binding name="http" port="5448"/>
        <socket-binding name="https" port="10206"/>
        <socket-binding name="jacorb" interface="unsecure" port="10207"/>
        <socket-binding name="jacorb-ssl" interface="unsecure" port="10208"/>
        <socket-binding name="jgroups-mping" port="0" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
        <socket-binding name="jgroups-tcp" port="10209"/>
        <socket-binding name="jgroups-tcp-fd" port="10210"/>
        <socket-binding name="jgroups-udp" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
        <socket-binding name="jgroups-udp-fd" port="10211"/>
        <socket-binding name="messaging" port="10212"/>
        <socket-binding name="messaging-group" port="0" multicast-address="${jboss.messaging.group.address:231.7.7.7}" multicast-port="${jboss.messaging.group.port:9877}"/>
        <socket-binding name="modcluster" port="0" multicast-address="224.0.1.105" multicast-port="23364"/>
        <socket-binding name="txn-recovery-environment" port="10216"/>
        <socket-binding name="txn-status-manager" port="10217"/>
        <outbound-socket-binding name="httpmessaging1">
            <remote-destination host="10.16.0.91" port="5447"/>
        </outbound-socket-binding>
        <outbound-socket-binding name="mail-smtp">
            <remote-destination host="localhost" port="25"/>
        </outbound-socket-binding>
      </socket-binding-group>
Actions
4. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

jbertram Mar 8, 2016 9:53 PM (in response to musheka)

I don't see any exception which might indicate what the problem would be. Also, I don't see HornetQ trying to connect to 10.16.0.91:57207 anywhere.
Actions
5. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

musheka Mar 8, 2016 10:42 PM (in response to jbertram)

In the logs , you can see lot of places where the port for remoting netty connection for 10.16.0.91 is 57207. Example is below. However in the standalone.xml , the http port is set to 5447.

2016-03-08 22:47:17,664 TRACE [org.hornetq.core.client] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2-1544750951)) returning RemotingConnectionImpl [clientID=null, nodeID=null, transportConnection=org.hornetq.core.remoting.impl.netty.NettyConnection@5a829cee[local= /10.16.0.91:57207, remote=/10.16.4.28:5448]]

<socket-binding name="http" port="5447"/>

Also towards the end of the logs , you can see the "java.lang.Exception" statement in the logs in previous message.
Actions
6. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

musheka Mar 9, 2016 12:01 AM (in response to musheka)
Justin,
As a supplement , I am also attaching the complete log file(in Server.zip) that includes the exception log stack traces. I am also including some notable exceptions/stack traces below.

2016-03-09 04:21:08,492 TRACE [org.hornetq.core.client] (ServerService Thread Pool -- 58) Topology@558ee0ef CREATE: java.lang.Exception: trace
at org.hornetq.core.client.impl.Topology.<init>(Topology.java:69) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.server.cluster.impl.ClusterConnectionImpl.<init>(ClusterConnectionImpl.java:152) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.cluster.ClusterManager.deployClusterConnection(ClusterManager.java:790) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.cluster.ClusterManager.deploy(ClusterManager.java:229) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart1(HornetQServerImpl.java:1610) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.access$1100(HornetQServerImpl.java:172) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreLiveActivation.run(HornetQServerImpl.java:2182) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:452) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:488) [hornetq-jms-server-2.4.7.Final.jar:]
at org.jboss.as.messaging.jms.JMSService.doStart(JMSService.java:170) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at org.jboss.as.messaging.jms.JMSService.access$000(JMSService.java:62) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at org.jboss.as.messaging.jms.JMSService$1.run(JMSService.java:96) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_05]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]
at org.jboss.threads.JBossThread.run(JBossThread.java:320) [jboss-threads-2.2.0.Final.jar:2.2.0.Final]

2016-03-09 04:21:08,496 TRACE [org.hornetq.core.server] (ServerService Thread Pool -- 58) ClusterConnection.start at ClusterConnectionImpl@1485089[nodeUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2, connector=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5447&host=10-16-0-91&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, address=jms, server=HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2]: java.lang.Exception: trace
at org.hornetq.core.server.cluster.ClusterManager.deployClusterConnection(ClusterManager.java:805) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.cluster.ClusterManager.deploy(ClusterManager.java:229) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart1(HornetQServerImpl.java:1610) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.access$1100(HornetQServerImpl.java:172) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreLiveActivation.run(HornetQServerImpl.java:2182) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:452) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:488) [hornetq-jms-server-2.4.7.Final.jar:]
at org.jboss.as.messaging.jms.JMSService.doStart(JMSService.java:170) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at org.jboss.as.messaging.jms.JMSService.access$000(JMSService.java:62) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at org.jboss.as.messaging.jms.JMSService$1.run(JMSService.java:96) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_05]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]
at org.jboss.threads.JBossThread.run(JBossThread.java:320) [jboss-threads-2.2.0.Final.jar:2.2.0.Final]

2016-03-09 04:21:08,828 TRACE [org.hornetq.core.server] (ServerService Thread Pool -- 58) Adding binding LocalQueueBinding [address=jms.queue.ESP-Queue, queue=QueueImpl[name=jms.queue.ESP-Queue, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2]]@590aa5a5, filter=null, name=jms.queue.ESP-Queue, clusterName=jms.queue.ESP-Queue3b0c38c2-dca1-11e5-9089-b92605690ef2] with address = jms.queue.ESP-Queue: java.lang.Exception: trace
at org.hornetq.core.postoffice.impl.SimpleAddressManager.addBinding(SimpleAddressManager.java:68) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.postoffice.impl.WildcardAddressManager.addBinding(WildcardAddressManager.java:96) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.postoffice.impl.PostOfficeImpl.addBinding(PostOfficeImpl.java:457) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.PostOfficeJournalLoader.initQueues(PostOfficeJournalLoader.java:160) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.loadJournals(HornetQServerImpl.java:1841) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.initialisePart2(HornetQServerImpl.java:1688) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.access$1400(HornetQServerImpl.java:172) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl$SharedStoreLiveActivation.run(HornetQServerImpl.java:2207) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.core.server.impl.HornetQServerImpl.start(HornetQServerImpl.java:452) [hornetq-server-2.4.7.Final.jar:]
at org.hornetq.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:488) [hornetq-jms-server-2.4.7.Final.jar:]
at org.jboss.as.messaging.jms.JMSService.doStart(JMSService.java:170) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at org.jboss.as.messaging.jms.JMSService.access$000(JMSService.java:62) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at org.jboss.as.messaging.jms.JMSService$1.run(JMSService.java:96) [wildfly-messaging-9.0.1.Final.jar:9.0.1.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_05]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]
at org.jboss.threads.JBossThread.run(JBossThread.java:320) [jboss-threads-2.2.0.Final.jar:2.2.0.Final]

2016-03-09 04:21:12,670 TRACE [org.jboss.modules] (MSC service thread 1-2) Finding class org.jboss.as.weld.deployment.WeldPortableExtensions from Module "org.jboss.as.jsf:main" from local module loader @14899482 (finder: local module finder @21588809 (roots: /docs/local/wildfly-9.0.1.Final/modules,/docs/local/wildfly-9.0.1.Final/modules/system/layers/base))
2016-03-09 04:21:12,670 TRACE [org.jboss.modules] (MSC service thread 1-2) Finding local class org.jboss.as.weld.deployment.WeldPortableExtensions from Module "org.jboss.as.weld:main" from local module loader @14899482 (finder: local module finder @21588809 (roots: /docs/local/wildfly-9.0.1.Final/modules,/docs/local/wildfly-9.0.1.Final/modules/system/layers/base))
2016-03-09 04:21:12,669 TRACE [org.hornetq.core.client] (Thread-3 (HornetQ-client-global-threads-80408863)) Node 3b0c38c2-dca1-11e5-9089-b92605690ef2 going up, connector = Pair[a=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5447&host=10-16-0-91&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, b=null], isLast=false csf created at
serverLocator=ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null]: java.lang.Exception
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.<init>(ClientSessionFactoryImpl.java:206) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.createConnectors(ServerLocatorImpl.java:1940) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl$StaticConnector.connect(ServerLocatorImpl.java:1815) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:663) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl.connect(ServerLocatorImpl.java:653) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl$3.run(ServerLocatorImpl.java:631) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104) [hornetq-core-client-2.4.7.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]

2016-03-09 04:21:12,670 TRACE [org.jboss.modules] (MSC service thread 1-2) Loading class org.jboss.as.weld.deployment.WeldPortableExtensions locally from Module "org.jboss.as.weld:main" from local module loader @14899482 (finder: local module finder @21588809 (roots: /docs/local/wildfly-9.0.1.Final/modules,/docs/local/wildfly-9.0.1.Final/modules/system/layers/base))
2016-03-09 04:21:12,670 TRACE [org.jboss.modules] (MSC service thread 1-2) Attempting to define class org.jboss.as.weld.deployment.WeldPortableExtensions in Module "org.jboss.as.weld:main" from local module loader @14899482 (finder: local module finder @21588809 (roots: /docs/local/wildfly-9.0.1.Final/modules,/docs/local/wildfly-9.0.1.Final/modules/system/layers/base))
2016-03-09 04:21:12,670 TRACE [org.hornetq.core.client] (Thread-3 (HornetQ-client-global-threads-80408863)) NodeUp ServerLocatorImpl (identity=(main-ClusterConnection::HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2)) [initialConnectors=[TransportConfiguration(name=server1-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5448&host=10-16-4-28&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor&local-address=10-16-0-91], discoveryGroupConfiguration=null]::nodeID=3b0c38c2-dca1-11e5-9089-b92605690ef2, connectorPair=Pair[a=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5447&host=10-16-0-91&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, b=null]: java.lang.Exception: trace
at org.hornetq.core.client.impl.ServerLocatorImpl.notifyNodeUp(ServerLocatorImpl.java:1589) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1664) [hornetq-core-client-2.4.7.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]

2016-03-09 04:21:12,670 TRACE [org.hornetq.core.client] (Thread-2 (HornetQ-client-global-threads-80408863)) Node 3b0c38c2-dca1-11e5-9089-b92605690ef2 going up, connector = Pair[a=null, b=null], isLast=true csf created at
serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=5b708885-e5ae-11e5-9a14-8ba8fd2e4c62, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null]: java.lang.Exception
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.<init>(ClientSessionFactoryImpl.java:206) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:863) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.jms.server.recovery.RecoveryDiscovery.start(RecoveryDiscovery.java:62) [hornetq-jms-server-2.4.7.Final.jar:]
at org.hornetq.jms.server.recovery.HornetQRecoveryRegistry.register(HornetQRecoveryRegistry.java:106) [hornetq-jms-server-2.4.7.Final.jar:]
at org.hornetq.jms.server.recovery.HornetQRegistryBase.register(HornetQRegistryBase.java:43) [hornetq-jms-server-2.4.7.Final.jar:]
at org.hornetq.ra.recovery.RecoveryManager.register(RecoveryManager.java:60)
at org.hornetq.ra.HornetQResourceAdapter.setup(HornetQResourceAdapter.java:1752)
at org.hornetq.ra.HornetQResourceAdapter.start(HornetQResourceAdapter.java:246)
at org.jboss.jca.deployers.common.AbstractResourceAdapterDeployer.startContext(AbstractResourceAdapterDeployer.java:340)
at org.jboss.jca.deployers.common.AbstractResourceAdapterDeployer.createObjectsAndInjectValue(AbstractResourceAdapterDeployer.java:1959)
at org.jboss.as.connector.services.resourceadapters.ResourceAdapterActivatorService$ResourceAdapterActivator.doDeploy(ResourceAdapterActivatorService.java:171)
at org.jboss.as.connector.services.resourceadapters.ResourceAdapterActivatorService.start(ResourceAdapterActivatorService.java:115)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]

2016-03-09 04:21:12,676 TRACE [org.hornetq.core.client] (Thread-2 (HornetQ-client-global-threads-80408863)) NodeUp ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=5b708885-e5ae-11e5-9a14-8ba8fd2e4c62, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0], discoveryGroupConfiguration=null]::nodeID=3b0c38c2-dca1-11e5-9089-b92605690ef2, connectorPair=Pair[a=TransportConfiguration(name=5b7e9249-e5ae-11e5-9a14-8ba8fd2e4c62, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, b=null]: java.lang.Exception: trace
at org.hornetq.core.client.impl.ServerLocatorImpl.notifyNodeUp(ServerLocatorImpl.java:1589) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1664) [hornetq-core-client-2.4.7.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]

2016-03-09 04:21:12,676 TRACE [org.hornetq.core.client] (Thread-2 (HornetQ-client-global-threads-80408863)) Topology@47331ff5::NewMemberAdd nodeId=3b0c38c2-dca1-11e5-9089-b92605690ef2 member = TopologyMember[name = default, connector=Pair[a=TransportConfiguration(name=5b7e9249-e5ae-11e5-9a14-8ba8fd2e4c62, factory=org-hornetq-core-remoting-impl-invm-InVMConnectorFactory) ?server-id=0, b=null]]: java.lang.Exception: trace
at org.hornetq.core.client.impl.Topology.updateMember(Topology.java:211) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl.notifyNodeUp(ServerLocatorImpl.java:1594) [hornetq-core-client-2.4.7.Final.jar:]
at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1664) [hornetq-core-client-2.4.7.Final.jar:]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_05]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_05]

server.zip 1.4 MB
Actions
7. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

jbertram Mar 9, 2016 9:14 AM (in response to musheka)

In the logs , you can see lot of places where the port for remoting netty connection for 10.16.0.91 is 57207. Example is below. However in the standalone.xml , the http port is set to 5447.

As the log message indicates, the use of 10.16.0.91:57207 is local. In other words, this is where the connection is being established from and the connection is going to the remote address of 10.16.4.28:5448. I don't see a problem here.

Also towards the end of the logs , you can see the "java.lang.Exception" statement in the logs in previous message.

All the exceptions you've noted are intentionally logged for debugging purposes. They do not indicate that an error has occurred. Notice that they are logged at DEBUG and TRACE levels. If an actual error had occurred it would be logged at ERROR level.

Again, I don't see a problem here.
Actions
8. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

musheka Mar 9, 2016 10:19 AM (in response to jbertram)

Ok. If there are no errors, I am not sure why I dont see a "Cluster Connection" message like below. In all the logs it has only been attempting to make a connection.
I would like to see a message like below. But it never happens.
1.) Is there any restriction on the OS . Will the clustering work on Amazon Linux OS machines? Any network interfaces that need to be set?
We have ensured the connectivity between the instances using ping and telnet at ports 5447 and 5448.
2.) Do you see any issues in the standalone.xml configuration that i sent??

Because the same setup worked in our company's intranet but is failing in AWS private subnets. Really strange!!

HQ221027: Bridge ClusterConnectionBridge@c2083af [name=sf.my-cluster.127ca753-dca2-11e5-a14e-0d94d082830b, queue=QueueImpl[name=sf.my-cluster.127ca753-dca2-11e5-a14e-0d94d082830b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2]]@3ab8523b targetConnector=ServerLocatorImpl (identity=(Cluster-connection-bridge::ClusterConnectionBridge@c2083af [name=sf.my-cluster.127ca753-dca2-11e5-a14e-0d94d082830b, queue=QueueImpl[name=sf.my-cluster.127ca753-dca2-11e5-a14e-0d94d082830b, postOffice=PostOfficeImpl [server=HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2]]@3ab8523b targetConnector=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5446&host=mongosvc-d01&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]]::ClusterConnectionImpl@1120925516[nodeUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2, connector=TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=10205&host=mongoetl-d01&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor, address=jms, server=HornetQServerImpl::serverUUID=3b0c38c2-dca1-11e5-9089-b92605690ef2])) [initialConnectors=[TransportConfiguration(name=http-connector, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5446&host=mongosvc-d01&http-upgrade-enabled=true&http-upgrade-endpoint=http-acceptor], discoveryGroupConfiguration=null]] is connected
Actions
9. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

jbertram Mar 9, 2016 10:53 AM (in response to musheka)

Ok. If there are no errors, I am not sure why I dont see a "Cluster Connection" message like below. In all the logs it has only been attempting to make a connection.

I would like to see a message like below. But it never happens.

I'm not sure why you aren't seeing it either. If you can demonstrate something clear and conclusive with a log file or reproducible test-case then I could probably help more (and by log I don't mean a 130,000-line file that you just attach to the case).

Is there any restriction on the OS .

I think that would be a question for your cloud provider.

Will the clustering work on Amazon Linux OS machines?

I believe that it will, but I have no personal experience with it so I can't say for sure.

Any network interfaces that need to be set?

I don't know of any, but as I said before I have no personal experience with Amazon's cloud.

Do you see any issues in the standalone.xml configuration that i sent??

Because the same setup worked in our company's intranet but is failing in AWS private subnets. Really strange!!

No, I don't see any issues with the configuration. By your own admission it is working in one environment and not in another. That suggests to me an environmental issue, and I think you'd need to pursue that with your cloud provider.
Actions
10. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

musheka Mar 12, 2016 1:03 AM (in response to jbertram)

OK thanks for your time and reply!!.
The solution from our end was to move to wildfly10 and use the activeMQ Artemis. It worked very well and clustering was not an issue. Not sure what was incompatible between wildfly9 hornetQ and amazon Linux machines.
Actions
11. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

ravi.soni Sep 28, 2016 11:33 PM (in response to musheka)

hi,

There is one bug open on HornetQ and jgroup channel implementation which making cluster and working completely impossible, and its not even fixed on Wildfly10 too, i am using EAP 6.4.6 and to make it working i needed to recompile EAP from source with custom patch on messaging system to make use of jgroups + s3 ping on AWS to work.

Wildfly bug [WFLY-3805] JGroupsBroadcastGroupConfiguration Serialization Problem - JBoss Issue Tracker
HornetQ bug [HORNETQ-1389] JGroupsBroadcastGroupConfiguration Serialization Problem - JBoss Issue Tracker

both are closed with out proper fix, so if you stuck on this case, only option to upgrade to activeMQ Artemist with wildfly 10 or compile wildfly code and patch form this pull request from hornetq bug.

Ravi
Actions
12. Re: Embedded HornetQ Clustering in AWS using JGroup-S3_PING

jbertram Sep 29, 2016 10:08 AM (in response to ravi.soni)

Wildfly 10 is using Apache ActiveMQ Artemis. If there is a clustering issue for which you have a patch please report/submit it on the Artemis JIRA. Thanks!
Actions

Go to original post