1 2 3 Previous Next 34 Replies Latest reply on Nov 25, 2003 7:18 AM by ivelin.ivanov

    JMS Clustering in JBoss 3.2

    ivelin.ivanov

      The core development group requested contribution to JBossMQ in the 3.2 branch.
      The primary reason is that JBossMQ in 4.0 will not be available for some time.
      JBoss 3.2 is expected to be the production branch for the foreseable future.

      Design discussions will occur on this thread.

        • 1. 3798921

          This comment was posted by Bela Ban on jboss-group mailing list. I'm reposting it here to facilitate discussion.

          > For JBoss clustering, we've been trying to avoid
          > having the remote client use JavaGroups. This is
          > because the client must be a member of the
          > Group/Channel to do anything. Maybe this has
          > changed in the current version of JG. Sacha? Bela? >comments?

          JavaGroups can be configured to use a 'services' architecture where there are 2 groups: a lightweight client-group, in which all clients and one or more servers are member, and a server-group in which only the
          servers are members. Clients use the client-group to talk to a server, which in turn uses the server-group, e.g. to replicate.

          However, as mentioned in my prev email, there are issues, e.g.:

          * Great number of clients, and/or clients not necessarily in the same multicast enabled network
          * Clients don't need to talk to each other, don't need to know about each other (e.g. no need to get notified when a new client joins/leaves/crashes).

          I'd suggest using JavaGroups for server-server communication, not client-server comm (although the latter is supported as well).

          --
          Bela Ban

          • 2. Re: JMS Clustering in JBoss 3.2

            Again, I can help if needed but it's difficult to start from scratsh.

            Ivelin, can you provide me some guidelines (links, doc, wiki page, whatever) ?

            Regards,

            Stephane

            • 3. MSSQL DS on jboss 2.4.3
              ivelin.ivanov

              Hello Everyone,

              I relize that i am talking about a really old version of JBOSS, however, i was hoping that someone would be be able to help me out.

              this is my setting in jboss.jcml


              <!--com.microsoft.jdbc.sqlserver.SQLServerDriver -->
              com.microsoft.jdbcx.sqlserver.SQLServerDataSource


              <!-- org.jboss.pool.jdbc.xa.wrapper.XADataSourceImpl -->
              org.jboss.pool.jdbc.xa.wrapper.XADataSourceImpl
              jdbc/WBTPCDataSource
              jdbc:microsoft:sqlserver://localhost:1433;DatabaseName=db_name;SelectMethod=Cursor
              user
              password
              3


              this is the error that i get:

              [INFO,jdbc/WBTPCDataSource] Starting
              [INFO,jdbc/WBTPCDataSource] XA Connection pool jdbc/WBTPCDataSource bound to java:/jdbc/WBTPCDataSource
              [ERROR,jdbc/WBTPCDataSource] Stopped
              java.lang.RuntimeException: No ManagedConnections Available!
              at org.jboss.pool.ObjectPool.getObject(ObjectPool.java:687)
              at org.jboss.pool.jdbc.xa.XAPoolDataSource.getConnection(XAPoolDataSource.java:242)
              at org.jboss.jdbc.XADataSourceLoader.startService(XADataSourceLoader.java:445)
              at org.jboss.util.ServiceMBeanSupport.start(ServiceMBeanSupport.java:103)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.lang.reflect.Method.invoke(Unknown Source)
              at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1628)
              at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1523)
              at org.jboss.configuration.ConfigurationService$ServiceProxy.invoke(ConfigurationService.java:967)
              at $Proxy0.start(Unknown Source)
              at org.jboss.util.ServiceControl.start(ServiceControl.java:79)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.lang.reflect.Method.invoke(Unknown Source)
              at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1628)
              at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1523)
              at org.jboss.Main.(Main.java:209)
              at org.jboss.Main$1.run(Main.java:110)
              at java.security.AccessController.doPrivileged(Native Method)
              at org.jboss.Main.main(Main.java:106)
              [ERROR,ConfigurationService] Unexpected error
              java.lang.RuntimeException: No ManagedConnections Available!
              at org.jboss.pool.ObjectPool.getObject(ObjectPool.java:687)
              at org.jboss.pool.jdbc.xa.XAPoolDataSource.getConnection(XAPoolDataSource.java:242)
              at org.jboss.jdbc.XADataSourceLoader.startService(XADataSourceLoader.java:445)
              at org.jboss.util.ServiceMBeanSupport.start(ServiceMBeanSupport.java:103)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.lang.reflect.Method.invoke(Unknown Source)
              at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1628)
              at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1523)
              at org.jboss.configuration.ConfigurationService$ServiceProxy.invoke(ConfigurationService.java:967)
              at $Proxy0.start(Unknown Source)
              at org.jboss.util.ServiceControl.start(ServiceControl.java:79)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
              at java.lang.reflect.Method.invoke(Unknown Source)
              at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1628)
              at com.sun.management.jmx.MBeanServerImpl.invoke(MBeanServerImpl.java:1523)
              at org.jboss.Main.(Main.java:209)
              at org.jboss.Main$1.run(Main.java:110)
              at java.security.AccessController.doPrivileged(Native Method)
              at org.jboss.Main.main(Main.java:106


              what does this mean?

              • 4. Re: JMS Clustering in JBoss 3.2
                ivelin.ivanov

                Hi,

                Load-balancing/failover are "easy" providing you can get
                the replication to work. Sacha has the right idea, it should
                be a sliding scale depending on requirements.

                I want the configuration to be chosen at destination level
                rather than global server level.

                Here are possible configurations:

                No replication
                like jbossmq now

                Singleton cold failover
                all state is in a single point if one server goes down,
                another picks up where the other left off
                losing any client state and non-persistent messages

                Singleton hot failover
                Each server has a JMS server implementation
                but only one server serves a destination at any one time. If you
                connect to the wrong server, you are "re-routed" to the active
                server for that destination.
                At least non-persistent messages are lost on failover
                and maybe client state as well.
                This gives you some load balancing because
                not all destinations are served by the same server.
                Some optimizations can put the senders/receivers on the
                same server where possible.
                This needs a DTM if there are multiple
                destinations in the same transaction on different servers.

                Replication based upon state in one point.
                (EntityBean approach) each server always goes to the shared state.
                You lose the performance gain of non-persistent messages.

                Full replication
                may or not be transactional (Stateful) each
                server has a copy of the state.

                The state does not just include the messages. There are also:

                Messages waiting acknowledgement.
                If the client disappears the server(s) should put these back in the queue.
                Durable subscriptions and clients logged on
                it will get confused if two servers have different views of this information

                Incomplete Transactions
                Messages that have been received or added in a transaction
                that is incomplete. This should be rolledback, but then
                so should other resources in the transaction.

                Connection specific Objects.
                Besides what has already been mention,
                there will be temporary queues/topics
                and non-durable subscriptions that should be deleted when
                the client disappears. These fall into the non-persistent
                message category i.e. maybe you don't want the overhead
                of replicating this state.

                One way to reduce some of the replication would be
                to ask the client to store a copy of some of the state
                (e.g. unacknowledged messages).
                On a failover the new server could ask the client for the
                current state.

                The two main use cases we need to support are
                (I don't discount others):

                Non-persistent messages
                as fast as possible, if the message is not delivered tough!

                Guaranteed delivery
                in the order the messages arrived at the JMS server(s)
                unless priority overrides the ordering
                (As a subset this guarantees delivery in the order
                sent by any single message producer)

                And remember KISS :-)

                Regards,
                Adrian

                • 5. Re: JMS Clustering in JBoss 3.2
                  rtvkuijk

                  Do we already have a list of accurate requirements for the JMS Clustering? We want clustering for topics, queues, persistent subscriptions, etc.?

                  From the client perspective, we have two cases: publisher and subscriber.

                  As a publisher, it is easy to both load-balances messages (all code is almost there) to a set of nodes or, to only send it to one node and failover in case of problem. Easy.

                  As a subscriber, it is easy to receive messages from any server.

                  So, IMHO, the difficulty is not really on the client side (except, as Adrian said, for Tx management in case of problem: we wouldn't be able to silently failover to another node as it would need a DTM)

                  The problem is more in the coordination of the servers. I don't remember exactly, but I think that JMS has some constraints wrt message priority, ordering, etc. To be verified, I don't remember.

                  I think we have essentialy three cases that, in fact, miming the EJB behaviour: SLSB, SFSB, EB.

                  1) In some cases, we want very high throughput and if some messages are lost, that's not a big deal, that's really the SLSB-analogy: you have 20 publisher feeding a queue and 20 consummers acting on these messages. In this case, the load can be totally distributed and if a node crash, that's not a big deal. the communication between the nodes is nil (except for administrative work). That's the goal of the Headless-JMS implementation based on JG I think.

                  2) In some cases, we want to be able to failover in case a server dies, but in a best-effort fashion. It is the SFSB mechanism. In this case, you would have JMS-nodes be paired and replicate the state, messages, etc. most of the time in-memory. In this case, let's say we have 6 servers, we would really have, for each topic/queue, a master server and one or more slave server (to be sure not to deliver a message twice). Thus, if we have 30 queues, each server would be the master for 5 queues and be the slave for 5 other queues.

                  3) The last case is for "critical" messaging. In this case, as for entity beans, you have to delegate the store to some kind of centralized repository that has to deal with its own HA. Each message is stored in the back-end DB (if the repository is a DB) and we must have some kind of synchronization between nodes (could be distributed lock mechanism at the JMS level, master-slave concept or DB pessimistic locking at the DB-level for example)

                  Anyway, I have always been skeptical about HA-JMS not because it would be hard to do, but simply because I have never been able to meet someone that has straight requirements (mostly blabla most of the time). Once we have real requirements, it should be quick as we can reuse A LOT: smart proxies are there, failover logic is there, locking strategies are there, in-memory state replication is there (even sub-partition code is still hanging around), etc.

                  Cheers,



                  sacha

                  • 6. Re: JMS Clustering in JBoss 3.2
                    ivelin.ivanov

                    FYI,
                    there is an earlier related topic that might hold interesting ideas:
                    http://www.jboss.org/modules/bb/index.html?module=bb&op=viewtopic&t=

                    • 7. Re: JMS Clustering in JBoss 3.2
                      ivelin.ivanov

                      The initial consensus is to use HASingleton for fault tolerant implementation of a clustered JMS server.

                      After I spent some time studying the JBossMQ code. looking at the problem from different perspectives and brainstorming with different people, I thought that I should mention an idea that was presented by a coleague of mine - Rick Grashel <rgrashel@austin.rr.com>.

                      Instead of picking one JMS server as a master for the whole cluster partition, each node can run its own independent copy of a JMS server.

                      See the next message for details.

                      • 8. Re: JMS Clustering in JBoss 3.2
                        ivelin.ivanov

                        The idea is as follows:

                        Each node will run its autonomous copy of a JMS server.
                        Each node will bind in its local JNDI an HAConnectionFactory and an XAHAConnectionFactory backed by a HAServiceIL.

                        When clients lookup their local HA connection factory and initiate a connection, the implementation will create an HAClientIL which will iterate over all member nodes and will connect to them. It will use JVMClientIL to connect to the local server and uil2.UILClientIL to connect to each remote node. These multiple connections can be later replaced by a single multicast connection, which will require a correspoding broadcasting ServerIL.

                        When a new node joins the cluster partition, each HAClientIL will connect to it.
                        When a node leaves the cluster, each HAClientIL will remove its connection to the node.

                        The advantages of this solution are:

                        1) No single point of failure

                        2) Sending messages is in VM. Only retrieval is remote.
                        This is more efficient compared to the singleton JMS server solution, when both sending a message to the singleton JMS server as well as receiving it require remote invocation and serialization.

                        3) No failover overhead.
                        As opposed to the singleton JMS where all clients have to reconnect to the new singleton when the current singleton fails.

                        4) Load balancing.
                        Each node has a fare chance of carying some message load.


                        Disadvantages:

                        1) No single view of the JMS cluster.
                        A new MBean has to be build to provide information about the overall status of the JMS cluster.

                        2) What else?


                        Thoughts?

                        • 9. Re: JMS Clustering in JBoss 3.2

                          It just won't work without replication.

                          This is the key point for HA messaging.
                          Something has to takeover when a server crashes and
                          have the same view of the data.

                          You also cannot do transparent failover inside
                          a transaction. You have to fail the transaction before
                          failing over.

                          If you are doing replication, I don't see how
                          load balancing allows you scale efficiently.

                          Stateful systems are always hard to scale efficiently.
                          You can make them highly available.

                          Regards,
                          Adrian

                          • 10. Re: JMS Clustering in JBoss 3.2
                            rgrashel

                            > It just won't work without replication.

                            This is absolutely true. There must be some form of replication in order for things to happen. Any kind of a stateful solution has to have replication. The best you can do here is to choose a method of stateful persistence/replication that is highly efficient.

                            > If you are doing replication, I don't see how
                            > load balancing allows you scale efficiently.

                            Replication and load-balancing are orthogonal. Replication is a given overhead that has to be allowed for. No matter what. That's an obvious given. Load balancing of messages can allow a system to scale because while all the cluster members have the same view of the data, no single member of the cluster is _handling_ an unbalanced load of those messages.

                            Replication is really an overhead that is fairly even and predictable. But in a large-scale, high-load environment there are many factors involved in message handling which can cause a system to degrade if load balancing is not performed.

                            Conceptually, this really shouldn't be a question of how to do clustering. There are tons of implementations out there that do it... and do it well. Any idea I have is really just ideas based upon clustering approaches in systems that already work. This really is simply a matter of choosing a clustering approach that is proven in event systems, and seeing applying it to JBoss.

                            -- Rick

                            • 11. Re: JMS Clustering in JBoss 3.2
                              nphelps

                              Clustering JMS is hard... I have no idea how to do it, but I've given it some thought. I don't think it makes sense to send the message to all nodes in the cluster from the ClientIL as you incur point-to-point communication to each node for each send. Much better to do the replication on the sever side (were you'll have to do it anyway when a new node starts up). Plus, you'll have to manage the removal of messages from the destination once the delivery has occurred to ensure that a message delivered by a node that fails right after delivery, is not redelivered by the node which the client fails over to. The positive ack should go to the new node though... Interesting stuff, and I think very difficult to implement. Also, Adrian mentioned in an e-mail that you don't really need to replicate non persistent messages. If the node they were delivered to fails, too bad... that is what non persistent messages are all about.

                              • 12. Re: JMS Clustering in JBoss 3.2
                                slaboure

                                Do we already have a list of accurate requirements for the JMS Clustering? We want clustering for topics, queues, persistent subscriptions, etc.?

                                From the client perspective, we have two cases: publisher and subscriber.

                                As a publisher, it is easy to both load-balances messages (all code is almost there) to a set of nodes or, to only send it to one node and failover in case of problem. Easy.

                                As a subscriber, it is easy to receive messages from any server.

                                So, IMHO, the difficulty is not really on the client side (except, as Adrian said, for Tx management in case of problem: we wouldn't be able to silently failover to another node as it would need a DTM)

                                The problem is more in the coordination of the servers. I don't remember exactly, but I think that JMS has some constraints wrt message priority, ordering, etc. To be verified, I don't remember.

                                I think we have essentialy three cases that, in fact, miming the EJB behaviour: SLSB, SFSB, EB.

                                1) In some cases, we want very high throughput and if some messages are lost, that's not a big deal, that's really the SLSB-analogy: you have 20 publisher feeding a queue and 20 consummers acting on these messages. In this case, the load can be totally distributed and if a node crash, that's not a big deal. the communication between the nodes is nil (except for administrative work). That's the goal of the Headless-JMS implementation based on JG I think.

                                2) In some cases, we want to be able to failover in case a server dies, but in a best-effort fashion. It is the SFSB mechanism. In this case, you would have JMS-nodes be paired and replicate the state, messages, etc. most of the time in-memory. In this case, let's say we have 6 servers, we would really have, for each topic/queue, a master server and one or more slave server (to be sure not to deliver a message twice). Thus, if we have 30 queues, each server would be the master for 5 queues and be the slave for 5 other queues.

                                3) The last case is for "critical" messaging. In this case, as for entity beans, you have to delegate the store to some kind of centralized repository that has to deal with its own HA. Each message is stored in the back-end DB (if the repository is a DB) and we must have some kind of synchronization between nodes (could be distributed lock mechanism at the JMS level, master-slave concept or DB pessimistic locking at the DB-level for example)

                                Anyway, I have always been skeptical about HA-JMS not because it would be hard to do, but simply because I have never been able to meet someone that has straight requirements (mostly blabla most of the time). Once we have real requirements, it should be quick as we can reuse A LOT: smart proxies are there, failover logic is there, locking strategies are there, in-memory state replication is there (even sub-partition code is still hanging around), etc.

                                Cheers,



                                sacha

                                • 13. Re: JMS Clustering in JBoss 3.2

                                  Hi,

                                  Load-balancing/failover are "easy" providing you can get
                                  the replication to work. Sacha has the right idea, it should
                                  be a sliding scale depending on requirements.

                                  I want the configuration to be chosen at destination level
                                  rather than global server level.

                                  Here are possible configurations:

                                  No replication
                                  like jbossmq now

                                  Singleton cold failover
                                  all state is in a single point if one server goes down,
                                  another picks up where the other left off
                                  losing any client state and non-persistent messages

                                  Singleton hot failover
                                  Each server has a JMS server implementation
                                  but only one server serves a destination at any one time. If you
                                  connect to the wrong server, you are "re-routed" to the active
                                  server for that destination.
                                  At least non-persistent messages are lost on failover
                                  and maybe client state as well.
                                  This gives you some load balancing because
                                  not all destinations are served by the same server.
                                  Some optimizations can put the senders/receivers on the
                                  same server where possible.
                                  This needs a DTM if there are multiple
                                  destinations in the same transaction on different servers.

                                  Replication based upon state in one point.
                                  (EntityBean approach) each server always goes to the shared state.
                                  You lose the performance gain of non-persistent messages.

                                  Full replication
                                  may or not be transactional (Stateful) each
                                  server has a copy of the state.

                                  The state does not just include the messages. There are also:

                                  Messages waiting acknowledgement.
                                  If the client disappears the server(s) should put these back in the queue.
                                  Durable subscriptions and clients logged on
                                  it will get confused if two servers have different views of this information

                                  Incomplete Transactions
                                  Messages that have been received or added in a transaction
                                  that is incomplete. This should be rolledback, but then
                                  so should other resources in the transaction.

                                  Connection specific Objects.
                                  Besides what has already been mention,
                                  there will be temporary queues/topics
                                  and non-durable subscriptions that should be deleted when
                                  the client disappears. These fall into the non-persistent
                                  message category i.e. maybe you don't want the overhead
                                  of replicating this state.

                                  One way to reduce some of the replication would be
                                  to ask the client to store a copy of some of the state
                                  (e.g. unacknowledged messages).
                                  On a failover the new server could ask the client for the
                                  current state.

                                  The two main use cases we need to support are
                                  (I don't discount others):

                                  Non-persistent messages
                                  as fast as possible, if the message is not delivered tough!

                                  Guaranteed delivery
                                  in the order the messages arrived at the JMS server(s)
                                  unless priority overrides the ordering
                                  (As a subset this guarantees delivery in the order
                                  sent by any single message producer)

                                  And remember KISS :-)

                                  Regards,
                                  Adrian

                                  • 14. Re: JMS Clustering in JBoss 3.2
                                    ivelin.ivanov

                                    Glad to see that Adrian and Sacha joined this thread.
                                    Makes me feel more confident that we won't go spagetty crazy when it comes down to coding.

                                    While digesting their messages, I just want to briefly respond to Nathan's note:

                                    > I don't think it makes sense to send the message to all
                                    > nodes in the cluster from the ClientIL as you incur ?
                                    > point-to-point communication to each node for each send.

                                    There is a misunderstanding at the algorithm level, probably because I am not using the correct JBossMQ interface names.

                                    The suggestion was not to send the messages to all the nodes, but only to the local server.
                                    All cluster nodes are registered as listeners remotely to the local server, and they will all receive the message in a pub/sub mode or only one of them will in a P2P mode.

                                    Does this make sense?

                                    If it does, which components of the current JBossMQ implementation would handle the described logic if not the ClientIL?

                                    Ivelin

                                    1 2 3 Previous Next