1 2 Previous Next 24 Replies Latest reply on Dec 2, 2006 5:56 PM by Kurt Stam

    Design of a Content Based Routing Service

    Kurt Stam Master

      I've been looking at the donated cbr code (see http://www.jboss.com/index.html?module=bb&op=viewtopic&t=86937, thx for that btw :)) and I think that for us to use it we need to make the following changes:

      1. Technology:
      a. upgrade to the latest JBossRules (3.x)
      b. move towards the new java-like rules definition
      c. move to a plain java implementation

      2. Integration
      a. Name the CBR as an action and define a processor for it, or in the architecture to come, define the CBR as a Service and register it's ServiceBinding in the Registry
      b. Add the CBR action as a (routing) action in the action chain.

      3. Questions
      a. Should the CBR service return a service/action name, or should it actually forward the message straight to it.
      b. Do we support routing to one or multiple destinations for the next release?
      c. Our messages are either of type Serializable or XML, will we support the content based routing of both types (or just XML)?
      d. What happens when no rules match? Do we have to create a 'catch all' rule, or is it ok to return 'sorry'. What will happen next? Will the message simply go to the next action?

      Dave I'd love to hear your, or anyone elses, thoughts on any of this.

      --Kurt

        • 1. Re: Design of a Content Based Routing Service
          Dave Dunkin Newbie

          Upgrading to JBossRules 3 will mean rewriting the DSL, which is fine. I haven't looked at how it's done in the new version, but it was dead simple to write a DSL for Drools. What do you mean by "plain java implementation?"

          Do you want to support a single language for rule matching (e.g. XPath) or leave it open-ended/extensible? I think that will help define whether or not to support Serializable. It would be cool to be able to specify a rule using EL on a POJO graph and XPath on an XML document. I guess it wouldn't really matter as long as the RHS of the rule fulfilled some contract like asserting a Destination to working memory.

          For my needs, if no rule matches the message, the message router returns an error. I could envision others wanting a default route/destination. I'm not very familiar with the action chain, but could that just be another action for those that require that functionality? I.e. an action after the CBR action would check for a defined destination and set a default if none exists. I suppose that would only work if the CBR service returns an action name instead of forwarding the message.

          I don't have an immediate need for sending one message to multiple destinations, but I will in the future. The only tricky part about that I can think of is with synchronous messages. If I send a message and am expecting a reply, who do I accept a reply from? My thought would be for synchronous messages to have one primary destination, from which the reply would come, and secondary destinations, which would be sent the message asynchronously. I haven't followed JBossESB that closely -- is there even the concept of synchronous and asynchronous messages?

          • 2. Re: Design of a Content Based Routing Service
            Kurt Stam Master

             

            Upgrading to JBossRules 3 will mean rewriting the DSL, which is fine. I haven't looked at how it's done in the new version, but it was dead simple to write a DSL for Drools. What do you mean by "plain java implementation?"

            Right, and using the JBossIDE2.0 it seems even easier. We'd like to keep our implemenation POJO based, but I realize that one would like to have only 1 CBR service running, where clients (listeners) connect to. This would make reloading the ruleset more manageable. I'll talk to Mark about this one.

            Do you want to support a single language for rule matching (e.g. XPath) or leave it open-ended/extensible?

            I think it make sense to leave it open-ended. Do you think this would be more work for us?

            I suppose that would only work if the CBR service returns an action name instead of forwarding the message.
            Well the CBR could first apply the rules, obtain one or more action/service names, then send messages to each of these destinations.

            is there even the concept of synchronous and asynchronous messages?

            Well I don't think it matter whether or not it is synchroneous, for instance in our loanbroker example we send out the loanrequest to 2 banks and receive 2 quotes. We listen for both quotes to come in, before processing the quotes. Typically the message flow is asynchroneous, and a separete listener (different thread) will listen for a response. However within an action pipeline things go synchronous (things like the transformation of a message into a different format). Hmm.. just thinking out loud here but I think the if the CBR would be at the end of an action pipeline, then mutliple desitinations would be no problem.

            Can we conclude that CBR can only happen as the last step in an action pipeline? That seems to make the most sense to me.

            Then I thought of some more interesting questions:
            - will we use a rules repository (will this be a database? And if so will it have it's own schema?) I think you use an xmlfile, read from an url somewhere right?
            - how do you update the rules? You use an mbean for that right?

            • 3. Re: Design of a Content Based Routing Service
              Dave Dunkin Newbie

               

              I think it make sense to leave it open-ended. Do you think this would be more work for us?


              No, I don't think it will be any more work. I may actually be less work. However, it may just be more abiguous for a user starting out. Good example usage will be important.

              - will we use a rules repository (will this be a database? And if so will it have it's own schema?) I think you use an xmlfile, read from an url somewhere right?
              - how do you update the rules? You use an mbean for that right?


              The way I implemented it was around deployment units. All the components in my home-grown ESB are EARs or SARs. These deployment units may contain rulesets, in an xml file, that define the routing for those components. I have a deployer that takes these routing-rule files, builds a single rule base for all deployed rulesets, and binds it to a location in JNDI. Updating rules involves redeploying the ruleset that needs to be updated.

              • 4. Re: Design of a Content Based Routing Service
                Mark Little Master

                 

                "kurt.stam@jboss.com" wrote:
                I've been looking at the donated cbr code (see http://www.jboss.com/index.html?module=bb&op=viewtopic&t=86937, thx for that btw :)) and I think that for us to use it we need to make the following changes:

                1. Technology:
                a. upgrade to the latest JBossRules (3.x)
                b. move towards the new java-like rules definition
                c. move to a plain java implementation

                2. Integration
                a. Name the CBR as an action and define a processor for it, or in the architecture to come,


                Sounds very like Hitchhiker's Guide to the Galaxy ;-)


                define the CBR as a Service and register it's ServiceBinding in the Registry


                I'm not sure if it would be a service or some logic within the dispatcher, which could use an external service I suppose.


                b. Add the CBR action as a (routing) action in the action chain.

                3. Questions
                a. Should the CBR service return a service/action name, or should it actually forward the message straight to it.


                Can you go through the scenario you have in mind?


                b. Do we support routing to one or multiple destinations for the next release?


                One would be a good start.


                c. Our messages are either of type Serializable or XML, will we support the content based routing of both types (or just XML)?


                Since the message formats can be extended arbitrarily and at runtime, I'd expect the CBR implementation to be suitable extensible too. We provide necessary plugins to support the out-of-the-box message formats we have, and use an architecture that allows others to add their own CBR-handlers at runtime.


                d. What happens when no rules match? Do we have to create a 'catch all' rule, or is it ok to return 'sorry'. What will happen next? Will the message simply go to the next action?

                Dave I'd love to hear your, or anyone elses, thoughts on any of this.

                --Kurt


                • 5. Re: Design of a Content Based Routing Service
                  Mark Little Master

                   

                  "kurt.stam@jboss.com" wrote:
                  Upgrading to JBossRules 3 will mean rewriting the DSL, which is fine. I haven't looked at how it's done in the new version, but it was dead simple to write a DSL for Drools. What do you mean by "plain java implementation?"

                  Right, and using the JBossIDE2.0 it seems even easier. We'd like to keep our implemenation POJO based, but I realize that one would like to have only 1 CBR service running, where clients (listeners) connect to. This would make reloading the ruleset more manageable. I'll talk to Mark about this one.


                  One single service is always a good deployment choice in general, since it makes management easier, but the architecture we should be looking at supporting is multiple services, potentially a CBR per client/service.


                  Do you want to support a single language for rule matching (e.g. XPath) or leave it open-ended/extensible?

                  I think it make sense to leave it open-ended. Do you think this would be more work for us?

                  I suppose that would only work if the CBR service returns an action name instead of forwarding the message.
                  Well the CBR could first apply the rules, obtain one or more action/service names, then send messages to each of these destinations.


                  Not sure if this helps, but one update Tom made recently allows the Action Chaining rules to be associated with (shipped with) a message and interpreted dynamically at the receiver.


                  is there even the concept of synchronous and asynchronous messages?

                  Well I don't think it matter whether or not it is synchroneous, for instance in our loanbroker example we send out the loanrequest to 2 banks and receive 2 quotes. We listen for both quotes to come in, before processing the quotes. Typically the message flow is asynchroneous, and a separete listener (different thread) will listen for a response. However within an action pipeline things go synchronous (things like the transformation of a message into a different format). Hmm.. just thinking out loud here but I think the if the CBR would be at the end of an action pipeline, then mutliple desitinations would be no problem.

                  Can we conclude that CBR can only happen as the last step in an action pipeline? That seems to make the most sense to me.


                  I think at present, with the current architecture based on listeners and actions and action chaining, we can impose whatever restrictions are required.



                  Then I thought of some more interesting questions:
                  - will we use a rules repository (will this be a database? And if so will it have it's own schema?) I think you use an xmlfile, read from an url somewhere right?
                  - how do you update the rules? You use an mbean for that right?


                  The Rules team are building their own rules repository. We should leverage that when it's available. In the meantime, we should provide something that is good enough, but doesn't require implementing an entire repository ;-)

                  • 6. Re: Design of a Content Based Routing Service
                    Mark Little Master

                     

                    "ddunkin" wrote:
                    Upgrading to JBossRules 3 will mean rewriting the DSL, which is fine. I haven't looked at how it's done in the new version, but it was dead simple to write a DSL for Drools.


                    This would have to be the latest GA version of JBossRules.


                    What do you mean by "plain java implementation?"

                    Do you want to support a single language for rule matching (e.g. XPath) or leave it open-ended/extensible? I think that will help define whether or not to support Serializable. It would be cool to be able to specify a rule using EL on a POJO graph and XPath on an XML document. I guess it wouldn't really matter as long as the RHS of the rule fulfilled some contract like asserting a Destination to working memory.


                    It would be good to support both. As I've said in another posting, we're letting people add their own Message formats for on-the-wire, so although we can initially say CBR only works if you use our XML format, eventually we'll need to put in place a framework to allow users extensibility at this level too.


                    For my needs, if no rule matches the message, the message router returns an error. I could envision others wanting a default route/destination. I'm not very familiar with the action chain, but could that just be another action for those that require that functionality? I.e. an action after the CBR action would check for a defined destination and set a default if none exists. I suppose that would only work if the CBR service returns an action name instead of forwarding the message.


                    Certainly for security and audit-trail purposes I'd agree that "no matching rule == fault" is a good sensible default. Providing flexibility at this level is fine, but needs to be addressed in the context of security: for example, we don't want arbitrary users intercepting messages and routing them to for their own nefarious reasons ;-)


                    I don't have an immediate need for sending one message to multiple destinations, but I will in the future. The only tricky part about that I can think of is with synchronous messages. If I send a message and am expecting a reply, who do I accept a reply from? My thought would be for synchronous messages to have one primary destination, from which the reply would come, and secondary destinations, which would be sent the message asynchronously. I haven't followed JBossESB that closely -- is there even the concept of synchronous and asynchronous messages?


                    I think the first question to ask is: why multiple destinations? The answer to that will impact the rest.

                    • 7. Re: Design of a Content Based Routing Service
                      Dave Dunkin Newbie

                       

                      I think the first question to ask is: why multiple destinations? The answer to that will impact the rest.


                      I have a ProcessPurchaseOrder document that comes into the system. That document is routed to the order management system for the supplier identified in the document. Now what I want to do is send a copy of all ProcessPurchaseOrder documents to a component that will extract certain information and store it for statistical analysis. It's a separation of concerns. The listener doesn't need to know anything about the statistical analysis. Neither does the order management system.

                      Another example is a document that contains hours worked on a case where the employee working the case is paid billable time. That document should go to both receivables, so that the client can be billed, and to payroll, so the employee can be paid.

                      • 8. Re: Design of a Content Based Routing Service
                        Mark Little Master

                         

                        "ddunkin" wrote:
                        I think the first question to ask is: why multiple destinations? The answer to that will impact the rest.


                        I have a ProcessPurchaseOrder document that comes into the system. That document is routed to the order management system for the supplier identified in the document. Now what I want to do is send a copy of all ProcessPurchaseOrder documents to a component that will extract certain information and store it for statistical analysis. It's a separation of concerns. The listener doesn't need to know anything about the statistical analysis. Neither does the order management system.


                        So this is routing sequentially though? I had assumed you were maybe talking about routing the message concurrently, e.g., like a broadcast of the message to multiple destinations.


                        Another example is a document that contains hours worked on a case where the employee working the case is paid billable time. That document should go to both receivables, so that the client can be billed, and to payroll, so the employee can be paid.


                        Is this done sequentially or concurrently? Do you wait for the receivables to ack before sending to the payroll? If the answer is yes, then there's not really a problem AFAICT. If the answer is no, then we're talking about "asynchronous" interactions and, assuming the sender wants some kind of ack from each, the underlying infrastructure will need to be able to tally responses with "rendezvous" requests from the client, in just the same way it would for any "asynchronous" interaction.

                        • 9. Re: Design of a Content Based Routing Service
                          Daniel Brum Novice

                           

                          Is this done sequentially or concurrently? Do you wait for the receivables to ack before sending to the payroll? If the answer is yes, then there's not really a problem AFAICT. If the answer is no, then we're talking about "asynchronous" interactions and, assuming the sender wants some kind of ack from each, the underlying infrastructure will need to be able to tally responses with "rendezvous" requests from the client, in just the same way it would for any "asynchronous" interaction.


                          I would say that this is actually not a part of routing, but in fact a BPM process that keeps track of all the routings required for a particular message that comes in. The routing should be fire and forget, and it should be the job of an encompassing business process definition that knows if all parties required have been notified or not. If not, maybe it takes an action to have the message routed again, or if it's an optional recipient (say a statistical analysis system) that can live without the data being there for a real-time process, then it let's the logic continue, etc.

                          I don't think we should be coupling the routing system to the systems receiving those routed data sets.

                          • 10. Re: Design of a Content Based Routing Service
                            Dave Dunkin Newbie

                             

                            "mark.little@jboss.com" wrote:
                            "ddunkin" wrote:
                            I have a ProcessPurchaseOrder document that comes into the system. That document is routed to the order management system for the supplier identified in the document. Now what I want to do is send a copy of all ProcessPurchaseOrder documents to a component that will extract certain information and store it for statistical analysis. It's a separation of concerns. The listener doesn't need to know anything about the statistical analysis. Neither does the order management system.


                            So this is routing sequentially though? I had assumed you were maybe talking about routing the message concurrently, e.g., like a broadcast of the message to multiple destinations.


                            No, it would be concurrent. The order management system shouldn't have to wait for the statisticl analysis system.

                            "mark.little@jboss.com" wrote:


                            Another example is a document that contains hours worked on a case where the employee working the case is paid billable time. That document should go to both receivables, so that the client can be billed, and to payroll, so the employee can be paid.


                            Is this done sequentially or concurrently? Do you wait for the receivables to ack before sending to the payroll? If the answer is yes, then there's not really a problem AFAICT. If the answer is no, then we're talking about "asynchronous" interactions and, assuming the sender wants some kind of ack from each, the underlying infrastructure will need to be able to tally responses with "rendezvous" requests from the client, in just the same way it would for any "asynchronous" interaction.


                            It would be concurrent. I think the assumption that the sender needs an ack from each receiver is not necessarily true. In fact, the sender doesn't even need to know where the message is going, just that the message was received by the ESB.

                            • 11. Re: Design of a Content Based Routing Service
                              Mark Little Master

                               

                              "daniel.brum@jboss.com" wrote:
                              Is this done sequentially or concurrently? Do you wait for the receivables to ack before sending to the payroll? If the answer is yes, then there's not really a problem AFAICT. If the answer is no, then we're talking about "asynchronous" interactions and, assuming the sender wants some kind of ack from each, the underlying infrastructure will need to be able to tally responses with "rendezvous" requests from the client, in just the same way it would for any "asynchronous" interaction.


                              I would say that this is actually not a part of routing, but in fact a BPM process that keeps track of all the routings required for a particular message that comes in. The routing should be fire and forget, and it should be the job of an encompassing business process definition that knows if all parties required have been notified or not. If not, maybe it takes an action to have the message routed again, or if it's an optional recipient (say a statistical analysis system) that can live without the data being there for a real-time process, then it let's the logic continue, etc.

                              I don't think we should be coupling the routing system to the systems receiving those routed data sets.


                              I think the example given certainly falls into a traditional BPM scenario. There may be some cross-over between BPM and CBR examples. Enabling users to do the same thing in different ways isn't necessarily a bad thing. I don't think there was/is the intention to tie our CBR implementation to a specific set of use cases.

                              • 12. Re: Design of a Content Based Routing Service
                                Kurt Stam Master

                                Hey Dave,

                                In the code that you contributed you are using XPATH in the rules. I really like that, but do you know if that means I need to use the xml rules?

                                thx,

                                --Kurt

                                • 13. Re: Design of a Content Based Routing Service
                                  Dave Dunkin Newbie

                                  I don't think you need to use the XML rules. I'm looking into how this could be implemented in the new DSL method.

                                  • 14. Re: Design of a Content Based Routing Service
                                    Dave Dunkin Newbie

                                    I think this could be accomplished through DSL and helper classes. Here's some untested code to show the direction:

                                    Here's what the new rules might look like:

                                    #created on: Oct 30, 2006
                                    package com.jboss.soa.esb.routing.cbr
                                    
                                    #list any import classes here.
                                    import org.jboss.soa.esb.message.Message;
                                    import org.jboss.soa.esb.message.format.MessageType;
                                    
                                    #declare any global variables here
                                    global java.util.List destinationServices;
                                    
                                    expander JBossESB.dsl
                                    
                                    rule "Routing Rule - XML based message"
                                    
                                     when
                                     xmlContentMatches "/PurchaseOrder"
                                     then
                                     System.out.println("JBoss_XML");
                                     destinationServices.add("test_category:JBOSS_XMLDestination");
                                    end
                                    


                                    The DSL:
                                    #JBossESB Content Based Routing DSL
                                    [when]xmlContentMatches "{xpath}"=msg : Message( type == MessageType.JBOSS_XML ) and eval( org.jboss.soa.esb.services.routing.cbr.DslHelper.xmlContentMatches(msg, "{xpath}") )
                                    [when]xmlContentEquals "{xpath}", "{value}"=msg : Message( type == MessageType.JBOSS_XML ) and eval( org.jboss.soa.esb.services.routing.cbr.DslHelper.xmlContentEquals(msg, "{xpath}", "{value}") )
                                    


                                    Helper class that does the actual XPath matching:
                                    package org.jboss.soa.esb.services.routing.cbr;
                                    
                                    import java.io.ByteArrayInputStream;
                                    
                                    import javax.xml.xpath.XPath;
                                    import javax.xml.xpath.XPathConstants;
                                    import javax.xml.xpath.XPathExpressionException;
                                    import javax.xml.xpath.XPathFactory;
                                    
                                    import org.jboss.soa.esb.message.Message;
                                    import org.xml.sax.InputSource;
                                    
                                    public class DslHelper {
                                     private static XPathFactory xpf = XPathFactory.newInstance();
                                    
                                     public static boolean xmlContentMatches(Message msg, String xpathExp) throws XPathExpressionException {
                                     XPath xpath = xpf.newXPath();
                                     InputSource inputSource = new InputSource(new ByteArrayInputStream(msg.getBody().getContents()));
                                     Object node = xpath.evaluate(xpathExp, inputSource, XPathConstants.NODE);
                                     return node != null;
                                     }
                                    
                                     public static boolean xmlContentEquals(Message msg, String xpathExp, String value) throws XPathExpressionException {
                                     XPath xpath = xpf.newXPath();
                                     InputSource inputSource = new InputSource(new ByteArrayInputStream(msg.getBody().getContents()));
                                     String node = (String) xpath.evaluate(xpathExp, inputSource, XPathConstants.STRING);
                                     return value.equals(node);
                                     }
                                    }
                                    


                                    1 2 Previous Next