1 2 Previous Next 22 Replies Latest reply on Jul 7, 2015 9:39 AM by Mark Swanson

    Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)

    Kiran Nunna Newbie

      We are using messaging:2.0 (HornetQ) for our project and see an issue related to HornetQ clustering,

       

      Env:

      2 nodes env and each node is in different Machines. i.e. Node-A in Machine-A and another node called Node-B in Machine-B.

      Server: Wildfly-8

      Mode: Domain Mode

       

      Scenario:

      • Node-A of Machine A starts up
      • Node-B of Machine-B starts up
      • Based on the logs, I can see HornetQ clustering is successfully configured  (configuration attached)
      • Now we have a tool which produces the Messages and put in HornetQ
      • And Node-A is reading/processing the messages successfully but Node-B is sitting idle (No messages are processed at Node-B)

       

      Summarized Problem Statement:

      All the messages are processed only in Node-A and Node-B is not processing any messages.

       

      What we are looking:

      Both the nodes (Node-A and Node-B) should process the messages which are in HornetQ Queue.

       

      Misc Information:

      Attached the logs and configurations file.

        • 1. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
          Justin Bertram Master

          The first thing I notice is that it appears you're binding your servers to 0.0.0.0.  Is that correct?  If so, I recommend you change your configuration so that each server binds to its specific IP address or hostname.  I believe binding to 0.0.0.0 is causing your cluster to malfunction because each node is broadcasting to the other node a connector which points to 0.0.0.0 which is meaningless to the other node.  In essence each node is forming a cluster connection with itself.

          • 2. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
            Justin Bertram Master

            I also noticed this:

             

            <cluster-connection name="hornetq-internal-cluster">
              <address>hornetq.#</address>
              <connector-ref>http-connector</connector-ref>
              <retry-interval>500</retry-interval>
              <use-duplicate-detection>true</use-duplicate-detection>
              <forward-when-no-consumers>true</forward-when-no-consumers>
              <max-hops>1</max-hops>
              <discovery-group-ref discovery-group-name="icp-discovery-group"/>
            </cluster-connection>
            

             

            As the documentation states, the <address> for a cluster-connection does not use wild-card matching so this cluster-connection is probably completely useless.  Even if you removed the ".#" from the <address> I think it would cause problems because it would load-balance internal notification messages.  Can you clarify why specifically you configured this cluster-connection?

            • 3. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
              Kiran Nunna Newbie

              Justin Bertram wrote:

               

              The first thing I notice is that it appears you're binding your servers to 0.0.0.0.  Is that correct?  If so, I recommend you change your configuration so that each server binds to its specific IP address or hostname.  I believe binding to 0.0.0.0 is causing your cluster to malfunction because each node is broadcasting to the other node a connector which points to 0.0.0.0 which is meaningless to the other node.  In essence each node is forming a cluster connection with itself.

              I see the INFO message that you specified in the log and it is as follows:

              2015-06-26 07:05:42,813 INFO  (ServerService Thread Pool -- 66) [org.hornetq.jms.server] HQ121005: Invalid "host" value "0.0.0.0" detected for "http-connector" connector. Switching to "APSED3095". If this new address is incorrect please manually configure the connector to use the proper one.

              But as it already switching the binding to the master host node (in this case apsed3095), we ignored it. Anyway, currently I made the changes as follows:

              <connectors>

                    <http-connector name="http-connector" socket-binding="hornetq-socket-binding">

                         <param key="http-upgrade-endpoint" value="http-acceptor"/>

                    </http-connector>

                    <in-vm-connector name="in-vm" server-id="0"/>

              </connectors>

              ---------

              <socket-binding name="hornetq-socket-binding" port="${jboss.http.port:8080}" interface="unsecure" />

              ---------

              <interface name="unsecure">

                   <inet-address value="${jboss.bind.address.unsecure:apsed3095}"/>

              </interface>

              • 4. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                Kiran Nunna Newbie

                Justin Bertram wrote:

                 

                I also noticed this:

                 

                1. <cluster-connection name="hornetq-internal-cluster"> 
                2.   <address>hornetq.#</address> 
                3.   <connector-ref>http-connector</connector-ref> 
                4.   <retry-interval>500</retry-interval> 
                5.   <use-duplicate-detection>true</use-duplicate-detection> 
                6.   <forward-when-no-consumers>true</forward-when-no-consumers> 
                7.   <max-hops>1</max-hops> 
                8.   <discovery-group-ref discovery-group-name="icp-discovery-group"/> 
                9. </cluster-connection> 

                 

                As the documentation states, the <address> for a cluster-connection does not use wild-card matching so this cluster-connection is probably completely useless.  Even if you removed the ".#" from the <address> I think it would cause problems because it would load-balance internal notification messages.  Can you clarify why specifically you configured this cluster-connection?

                We took these configuration from JBOSS4 hornetQ configurations. Do you think this configuration is misleading the stuff? Do we need to remove that?

                • 5. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                  Justin Bertram Master

                  Have you observed any difference in behavior since changing the interface where the server binds?

                  • 6. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                    Justin Bertram Master

                    Why was this configuration in JBoss AS 4?  Was HornetQ clustered when it was deployed in JBoss AS 4?

                     

                    I think the configuration is likely useless and therefore should be removed, but it would be good to understand why it was there in the first place.

                    • 8. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                      Kiran Nunna Newbie

                      Justin Bertram wrote:

                       

                      Why was this configuration in JBoss AS 4?  Was HornetQ clustered when it was deployed in JBoss AS 4?

                       

                      I think the configuration is likely useless and therefore should be removed, but it would be good to understand why it was there in the first place.

                      ok, I will try that out. Do we have any sample hornetq configuration with 2 nodes (in multiple machines) that process message queues successfully? Again our env: Wildfly8.2.0Final with HornetQ2.4.5

                       

                      Thanks in advance.

                      • 9. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                        Justin Bertram Master

                        Do we have any sample hornetq configuration with 2 nodes (in multiple machines) that process message queues successfully? Again our env: Wildfly8.2.0Final with HornetQ2.4.5

                        Yes.  The "full-ha" profile in domain.xml should successfully cluster.

                         

                        To be clear, there has to be at least one consumer on any node where you expect to see messages sent in the cluster by default.  Do you have at least one consumer on both of your nodes?

                        • 10. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                          Kiran Nunna Newbie

                          Yes.  The "full-ha" profile in domain.xml should successfully cluster.

                          Oh great, so where can I get the sample domain.xml. Will it be available with download the wildfly8?

                           

                          Do you have at least one consumer on both of your nodes?

                          Yes Justin. We have one consumer in both nodes.

                          • 11. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                            Justin Bertram Master

                            Oh great, so where can I get the sample domain.xml. Will it be available with download the wildfly8?

                            Yes, the "full-ha" profile comes in the domain.xml shipped with Wildfly.

                            • 12. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                              Mark Swanson Newbie

                              Justin,

                               

                              I can assist with some history here. We are porting our application from JBoss 4 / HornetQ 2.4.4 (grafted / hacked) to JBoss 8. As part of clustering problems with JB4 we reached out to the HornetQ team and were told to add this cluster connection (that I believe is not documented anywhere in the HornetQ docs). Your comment of "internal notifications" sounds familiar. In any case, one of our engineers asked the question of why our cluster was not working, and we were instructed to add this block. Kiran states, "we took this from our JB4 configuration", which is true. But we only added it because we were told to. We really don't understand why it was needed, or what it does.

                               

                              We did finally get clustering working on JB4. But only for UDP. We have customers running clusters that cannot multicast between nodes, so as part of our port to JBoss 8, we're also attempting to make *everything* TCP, from discovery to message delivery. Hence "tcpping", and the special JGroups stack.

                               

                              Our application is a homogenous cluster of MDB beans listening for messages that then call session beans, etc. as per accepted practices. We've worked to make the HornetQ RA / MDB session pool / thread pool counts the same to avoid funny behavior and thrashing. No errors in the logs, but no messages distributed to nodes other than our "primary" either.

                               

                              So that's where we are. If that triggers any thoughts. I'd sure like to understand what the hornetq.# cluster configuration is for, and why we may have  been told to put it in, and what affect might have if we take it out (better? worse?).

                               

                              Thanks,

                               

                              Mark

                              • 13. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                                Justin Bertram Master

                                We are porting our application from JBoss 4 / HornetQ 2.4.4 (grafted / hacked) to JBoss 8. As part of clustering problems with JB4 we reached out to the HornetQ team and were told to add this cluster connection (that I believe is not documented anywhere in the HornetQ docs). Your comment of "internal notifications" sounds familiar. In any case, one of our engineers asked the question of why our cluster was not working, and we were instructed to add this block. Kiran states, "we took this from our JB4 configuration", which is true. But we only added it because we were told to. We really don't understand why it was needed, or what it does.

                                I'm a member of the HornetQ engineering team, and I still don't understand why you'd need that cluster connection.  As far as I can tell it is meaningless now and always has been unless you had addresses that literally started with "hornetq.#" (e.g. "hornetq.#foo").  I'd be curious to see the original forum thread related to this configuration decision (assuming and hoping this exchange happened on the forum).  As always, applying configuration without understanding what it does is dangerous.

                                 

                                We have customers running clusters that cannot multicast between nodes, so as part of our port to JBoss 8, we're also attempting to make *everything* TCP, from discovery to message delivery. Hence "tcpping", and the special JGroups stack.

                                As far as HornetQ clustering goes, you don't need to use JGroups to get TCP based discovery.  You can simply use static discovery as discussed in the documentation.  I think it's more straight-forward to understand and configure than JGroups.

                                 

                                At the end of the day I'd recommend starting with a working cluster configuration (e.g. one of the clustered configurations that ship with Wildfly) and work backwards from there.  Don't add anything you don't absolutely need or understand.  A homogeneous cluster is a basic use-case that should work out-of-the-box.

                                • 14. Re: Wildfly: HornetQ Clustering, nodes doesnt recieve messages (or nodes are idle)
                                  Mark Swanson Newbie

                                  Justin,

                                   

                                  We investigated that very idea (static clustering), but we need to run in domain mode, not "clustered standalone" mode (for one basic reason: auto-deployment of our EAR application). And the example was a standalone.xml example. For this static connector definition to work in domain mode, instead of just having connections to all the *other* nodes, it would have to define connections to *all* nodes (which for each node would include "self"). This appears to be an anti-pattern, as described in the WildFly docs thusly:

                                   

                                   

                                  17.5.4 . Switch UDP to TCP for HornetQ Clustering

                                  1. Remove the broadcast-groups and discovery-groups [OK, no problem]

                                  2. Optionally, remove the "messaging-group" socket-binding: [OK, got it]

                                  3. Configure the appropriate Netty connector(s) - one for each of the other nodes in the cluster. [ah. there's the problem]

                                   

                                  There is even this explicit statement:

                                   

                                  Note: Do not configure a node with a connection to itself. This is considered as a misconfiguration.

                                   

                                  So we're confused where to start. The example with static clustering is for standalone.xml. Domain mode requires a single, homogenous configuration file to go to all nodes, but the docs say configuring a static connection to "self" is bad. This is why we went to JGroups and TCP as the only alternative we could think of.

                                   

                                  Given our domain-mode constraint, how would you suggest we proceed? Is there an example for "homogenous TCP-only domain-mode cluster?"

                                   

                                  Thanks,

                                   

                                  Mark

                                   

                                  P.S. - I could probably dig up the conversation we had with your team. It was with yet another engineer. I was not directly part of that conversation. I'd also be interested to read it .

                                   

                                  1 2 Previous Next