9 Replies Latest reply on May 17, 2017 11:19 AM by jbertram

    Why do we configure folders such as journal-directory for a data replication based high availability for message?

    wwang2016

      Hi,

       

      I am investigating HA (high availability) for messaging in a wildfly(active) - wildfly(active) cluster. For shared-store approach, we need to configure the following:

       

      bindings-directory

      journal-directory

      large-messages-directory

      paging-directory

       

      This is because each wildfly instance will have a default/active and a backup/standby activemq servers. The directories are specified so that in case of a wildfly instance is down, the messages will be copied to another active wildfly instance.

       

      However, for data replication approach, do we also need to configure these folders? My assumption is that they are not necessary since messages are replicated in real time to the default folders.

       

      Based on the wildfly-messaging-activemq_1_0.xsd, I can not see if these folders should or should not be configured for data replication scenario. If we do need to configure these directories, what are the rationales?

       

      Thanks,

       

      Wayne

        • 1. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
          jbertram

          The directories are specified so that in case of a wildfly instance is down, the messages will be copied to another active wildfly instance.

          That's not quite accurate. Messages are *shared* between the live and backup brokers (hence the label _shared storage_). Message are not "copied" from the live to the backup. In fact, that's one of the benefits of the shared-store approach - there is no overhead from data duplication.

           

          However, for data replication approach, do we also need to configure these folders?

          No, it's not strictly necessary to configure the folder locations.

           

          My assumption is that they are not necessary since messages are replicated in real time to the default folders.

          That's correct.

           

          Based on the wildfly-messaging-activemq_1_0.xsd, I can not see if these folders should or should not be configured for data replication scenario.

          The schema regarding the folders is no different for replication or shared-storage. The requirement to configure the folders for shared-storage is functional (i.e. so that two brokers can actually share the folders).

          • 2. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
            wwang2016

            Hi Justin,

             

            Based on your comments, I understand that the folder configuration is not necessary in both shared-store and data-replication approaches. However, sometimes we may NOT want to utilize the default message folders, and would rather configure a different sets of message folders.

             

            In a two-node scenario, wildfly (Active) - wildfly (Active), I can configure live-backup pair in each wildfly instance. I normally configured it as:

             

            wildfly #1 (default activemq) message folders <-> wildfly #2 (backup activemq) message folders

            wildfly #1 (backup activemq) message folders <-> wildfly #2 (default activemq) message folders

             

            If I do not configured it as above, will the cluster members automatically knows they will use the default message folders to share the messages?

             

            However, in a three wildfly instances scenario, If I do not configure the folder explicitly as:

             

            wildfly #1 (default message folder, backup for wildfly #2, backup for wildfly #3)

            wildfly #2 (default message folder, backup for wildfly #1, backup for wildfly #3)

            wildfly #3 (default message folder, backup for wildfly #1, backup for wildfly #2)

             

            How do these cluster members know how to share the messages without explicitly specifying the message folders?

             

            Thanks,

             

            Wayne

            • 3. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
              jbertram

              Based on your comments, I understand that the folder configuration is not necessary in both shared-store and data-replication approaches. However, sometimes we may NOT want to utilize the default message folders, and would rather configure a different sets of message folders.

              Configuring the folders in the shared-store use-case isn't necessary from a schema perspective, but it is necessary from a functional perspective. In other words, two brokers can't actually share the same storage if you don't configure the folders appropriately.  As I explained previously, in the shared-store use-case brokers actually share the same storage (e.g. configured on an NFS mount or SAN).  That's not possible if you leave the default folder configurations.

               

              If I do not configured it as above, will the cluster members automatically knows they will use the default message folders to share the messages?

              I'm not sure I understand your question.  If you leave the default folder configurations then there will be no shared storage since the brokers will not be using the same folders.  Brokers don't "automatically" know to use the same folders.  You have to explicitly configure them to use the same folders to properly support the shared-store use-case.

               

              How do these cluster members know how to share the messages without explicitly specifying the message folders?

              They don't.

              • 4. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
                wwang2016

                Hi Justin,

                 

                I understand now that we DO need to configure these message folders for shared store approach. Actually, I configured the message folders in all my previous tests for shared store approach, and it worked fine. I was just confusing about the minOccurs= 0 for all the message directories in the schema file.

                 

                My original question was about configuration of message folders for data replication approach, and I configured the messages folders for data replication approach the same way as that for shared store approach in all my previous tests. The tests seemed working fine. However, I was wondering if I did not configure the message folders, and would that work for data replication approach in wildfly (active) - wildfly (active) configuration? You mentioned that it is not necessary.

                 

                So in a scenario of two wildfly instance such as

                wildfly #1 (live activemq   / backup activemq)

                wildfly #2 (live activemq   / backup activemq)

                 

                If I do not configure the message folders explicitly, do these message servers in two active wildfly instances know how to get message replicated between these two wildfly instances?

                 

                What if we have three wildfly instances

                 

                wildfly #1 (live activemq   / backup activemq)

                wildfly #2 (live activemq   / backup activemq)

                wildfly #3 (live activemq   / backup activemq)

                 

                Will messages get replicated to from one server to other two server withoug explicitly configuring the message folders?

                 

                What would happen in the case of one wildfly instance is down while in the middle of processing messages?

                 

                Thanks,

                 

                Wayne

                • 5. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
                  jbertram

                  ...I configured the messages folders for data replication approach the same way as that for shared store approach in all my previous tests.

                  I'm struggling to understand why you would configure the folders the same for both shared-store and replication.

                   

                  To be clear, shared-store high availability use-case requires the live and backup brokers to share the same physical storage (e.g. via NFS, SAN, CIFS, etc.).  In this use-case no messages are copied from the live to the backup; messages are simply written to the shared storage device and are therefore available to both brokers.  However, in some environments it's not possible or desirable to configure shared storage so replication is an option.

                   

                  In the replication use-case no messages are physically shared.  Messages are copied across the network from the live to the backup.  Therefore both the live and the backup have a copy of the same messages.

                   

                  I hope you are tracking with me now and understand that It makes no sense to configure the folders the same for both shared-store and replication.

                   

                  The tests seemed working fine.

                  I've never imagined testing replication when the brokers are actually using a shared store so I can't say how the brokers would behave in that scenario, but I believe they would not function properly.  In any event, this is not a recommended configuration.

                   

                  However, I was wondering if I did not configure the message folders, and would that work for data replication approach in wildfly (active) - wildfly (active) configuration? You mentioned that it is not necessary.

                  To be clear, high availability (e.g. as implemented via replication) is only supported between a live (i.e. active) and a backup (i.e. passive) broker.  There is no support HA with active - active brokers.

                   

                  In terms of folder configuration, you don't have to configure the folders when using replication since there's no need to ensure that both the live and backup are using the same physical storage.

                   

                  If I do not configure the message folders explicitly, do these message servers in two active wildfly instances know how to get message replicated between these two wildfly instances?

                  Assuming you have configured everything appropriately the live and backup brokers will discover each other on the network, indicate what role they want to play in the live/backup relationship, and the live broker(s) will begin replicating its messages to the backup broker(s).

                   

                  Will messages get replicated to from one server to other two server withoug explicitly configuring the message folders?

                  Assuming you have replication configured correctly, yes.  Remember, replication doesn't require shared storage; that essentially the whole point of replication.

                   

                  What would happen in the case of one wildfly instance is down while in the middle of processing messages?

                  You'd have to be more specific about the use-case in order for me to comment.

                   

                   

                  In general, if you're still unclear about what exactly shared-store and replication are all about I would encourage you to spend some time in the documentation reading about both and even running the various HA examples shipped with Artemis.

                  • 6. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
                    wwang2016

                    Hi Justin,

                     

                    I would like to clarify the wildfly (active) - wildfly (active) configuration. We would like to set up a cluster of wildfly instances/ servers, and all these servers will be active at the same time so that anytime one instance is down, session data, cache etc will fail over to another/other wildfly instances. Let us call this true clustering (vs HA singleton where one wildfly is active, the other is standby).

                     

                    To ensure message high availability, we also need to set up / configure activemq subsystem so that if a wildfly instance is down while in the middle of message processing, we would like another wildfly instance can pick up messages that are queued, but not processed. However, without additional configuration, I believe the messages will not be visible to other wildfly instance. Therefore, I plan to configure live-backup pair in each wildfly instance with shared store approach or data replication.

                     

                    Is that the proper way to go?

                     

                    In this scenario, what is the best practice in configuring activemq subsystem in the case of

                     

                    (1) a cluster of two wildfly instances

                    (2) a cluster of three or more wildfly instances?

                     

                    Thanks,

                     

                    Wayne

                    • 7. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
                      jbertram

                      Let us call this true clustering (vs HA singleton where one wildfly is active, the other is standby).

                      There's already a name for this - colocated topology. In this topology every Wildfly instance has 2 (or more) broker instances where one is live and the other(s) serves as a backup for another, remote live server.

                       

                      To ensure message high availability, we also need to set up / configure activemq subsystem so that if a wildfly instance is down while in the middle of message processing, we would like another wildfly instance can pick up messages that are queued, but not processed. However, without additional configuration, I believe the messages will not be visible to other wildfly instance. Therefore, I plan to configure live-backup pair in each wildfly instance with shared store approach or data replication.

                       

                      Is that the proper way to go?

                      Yes.  If you want high availability then you'll need to configure live/backup pairs via shared-store or replication.

                       

                      In this scenario, what is the best practice in configuring activemq subsystem in the case of

                       

                      (1) a cluster of two wildfly instances

                      (2) a cluster of three or more wildfly instances?

                      In my opinion shared-storage is the best option as there is no overhead from replicating message data across the network and there's little to no risk of split brain (a situation where a backup activates illegitimately and there are 2 live servers with the same essential data).

                       

                      If you go with replication then the more live/backup pairs you have the better as it will mitigate the risk of split brain.

                      • 8. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
                        wwang2016

                        Hi Justin,

                         

                        I have some concern over configuring multiple active wildfly instances in a cluster to ensure HA for messaging.

                         

                        Scenario #1: two wildfly instances

                        I could set it like:

                         

                        (1) shared store approach

                        wildfly #1 (live #1 + backup #2)

                        wildfly #2 (live #2 + backup #1)

                         

                        The folders specified for live #1 is the same as that in backup #1

                        The folders specified for live #2 is the same as that in backup #2

                         

                        (2) data replication approach

                        wildfly #1 (replication-master)

                        wildfly #2 (replication-slave)

                         

                        No message folder to be configured

                         

                        Scenario #2: three wildfly instances

                         

                        (1) share store approach

                        wildfly #1 (live #1 + backup #2 + backup #3)

                        wildfly #2 (live #2 + backup #1 + backup #3)

                        wildfly #3 (live #3 + backup #1 + backup #2)

                         

                        This is to ensure if any wildfly instance is down, the messages will get picked up by any one of the remaining wildfly instances. However, I can see an issue with two backup with one live since there can be potential of repeated processing

                        for example, wildfly #1 is down, the backup #1 of wildfly #2 and wildfly #3 can access the messages.

                         

                        (2) how to set it up with three wildfly instances?

                        Two makes a pair. How to make three in pairing?

                         

                        Scenario #3: four wildfly instances

                         

                        We may not go with the way Scenario #2 is trying to do. It does not seem to scale.

                         

                        Is the following configuration better?

                         

                        wildfly #1 (live #1 + backup #2) and wildfly #2 (live #2 + backup #1)  // this is a pair

                        wildfly #3 (live #3 + backup #4) and wildfly #4 (live #4 + backup #3)  // this is a another pair

                         

                        There is an issue with the above configuration: if wildfly #1 is down, messages will fail over to wildfly #2. But if wildfly #2 is down, the messages will have no way to fail over.

                         

                        I have not found a lot of information as to setting up many wildfly instances in terms of HA for messaging. What do you think we should do in production?

                         

                        Thanks,

                         

                        Wayne

                        • 9. Re: Why do we configure folders such as journal-directory for a data replication based high availability for message?
                          jbertram

                          "Better" or "worse" really depends on your use-case, performance requirements, etc.  Your question about what to use in production is really too broad to tackle in a forum like this.

                           

                          Couple of general points:

                          • Configuring instances of the broker inside Wildfly really isn't conceptually different than configuring instances of the broker outside Wildfly.  Wildfly just acts as a "container" for the instances, but in terms of high availability they really operate the same inside or outside.
                          • Fail-back semantics don't work when there are multiple backups for a single live.  My personal opinion is that such configurations are a bit clumsy anyway and if you think that it's likely both a live and a backup will fail then you should look to more robust hardware (e.g. with redundant power supplies, redundant NICs, etc.).  If you colocate a live broker with multiple backups then in worst case scenario you'll have 3 or more live brokers running in the same Wildfly instance which isn't likely to function well anyway, especially if load is high.

                           

                          Hope that helps.