1 2 Previous Next 18 Replies Latest reply on Apr 9, 2013 5:14 AM by Yann

    "ForEach"-Splitter based on XML content

    Bernd Ruecker Master

      Hi all (again ;-))

      I currently search for a Splitter, which basically behaves like a "for each" loop, meaning I want to split one XMl in the message (e.g. an order with 10 Items) into separate messages for parts of it (in this case 10 messages with one order item each) and route that to a destination (obviously the same destination for each new message).

      I added that requirement to the EAI-Splitter JIRA Issue as well, unfortunately in the proposed code there, it is not yet done :-/ https://jira.jboss.org/jira/browse/JBESB-2805

      Does anybody already did something like this? Or is it on the concrete roadmap for the ESB? I think, that is not an exotic requirement, or is it? For the moment we have an easy workaround (we just sending messages with order items to the ESB ;-)), but this should be removed as soon as possible...

      Thanks and cheers
      Bernd

        • 1. Re:
          Hans Wolffenbuttel Expert

          hi.

          If you have an xsd for your order with items, you can use JAXB to unmarshal your xml to javaobjects and then use normal java to split them up. JAXB can also marshal it back to XML.

          Regards,

          Hans

          • 2. Re:
            Bernd Ruecker Master

            Hi Hans, thanks for the answer!

            But this means to write a peace of Java for every use case, right? Then it does not even matter if I use JAXB, XStream or whatever to transform it to Java.

            I searched for something more like this, which just needs an XPath expression to be configured and is reusable out-of-the-box: http://servicemix.apache.org/servicemix-eip.html#servicemix-eip-XPathSplitter

            • 3. Re:
              Daniel Bevenius Master

              Hi Bernd,

              I think you can do this using Smooks. The smooks_file_splitter_router does something similar. If you have not already done so could you take a look at that quickstart and see if this fits your use case?

              Regards,

              /Daniel

              • 4. Re:
                Bernd Ruecker Master

                Hi Daniel.

                Hmm, actually it should be the right direction. By the way, the URL to the smooks docs have changed: http://docs.codehaus.org/display/MILYN/Smooks+User+Guide#SmooksUserGuide-Splitting%26Routing

                There it says:

                for example, order items in an an order message need to be split out and routed (based on content - "Content Base Routing")


                I think it don't has to be CBR. Maybe you have a target service, just taking one order item at a time...

                But anyway. Seems to be pretty complicated using smooks here. And I only find examples using Java, but actually I don't want to transform my XML into Java (why??), my vision is just to provide an XPath for splitting the XML, but that seems not to be easy with Smooks? Or did I just miss it? And in the quickstart quite some own action classes are necessary :-/

                Thanks a lot for clarification.
                Cheers
                Bernd

                • 5. Re:
                  John Pount Newbie

                  Hey Bernd,

                   

                  I have just done the following which might be similar to what you need ...

                   

                  1. Read CSV file from file system (The file contains many individual invoices:)

                  2. Transform CSV file to XML

                  3. Split the XML file into individual invoices and send each one to a queue/service to be processed

                   

                  I am doing this using smooks although I'm not sure if it's the best/recommended way to do this sort of thing:

                   

                  jboss-esb.xml

                   

                  <!-- Split the XML -->
                  <action name="split-transform" class="org.jboss.soa.esb.smooks.SmooksAction">
                       <property name="smooksConfig" value="/smooks-res.xml" />
                  </action>

                   

                  smooks-res.xml

                   

                  <!-- Splitter -->
                  <resource-config selector="Invoice">
                         <resource>org.jboss.soa.esb.visitors.DOMServiceDelegateVisitor</resource>
                         <param name="service-category-name">InvoiceProcessing</param>
                         <param name="service-name">IndividualInvoiceProcessor</param>
                         <param name="send-in-visit-after">true</param>
                  </resource-config>

                   

                  I have another service - (InvoiceProcessing/IndividualInvoiceProcessor) which receives and processes the individual Invoices.

                   

                  BTW - I am using SOA 4.3 so I think there might be better/different ways to do this using a newer version of the ESB.

                   

                  If you want the whole ESB project then let me know.

                  • 6. Re:
                    Bernd Ruecker Master

                    Hey John!

                     

                    Perfect, that sounds great and exactly for what I was searching for! If you could post some code for it, that would be really great!

                     

                    Maybe the ESB guys could take that as a basis for a new quickstart as well (I could create a JIRA for it)? Or you may want to blog about it yourself? This is exactly the example that was missing there I think.

                     

                    Cheers

                    Bernd

                    • 7. Re: "ForEach"-Splitter based on XML content
                      Tom Fennelly Master

                      Hi guys, there are a number of examples of this.

                       

                      • The easiest one to start with is probably the "split-transform-route-jms" example from the Smooks project.  There are other examples there too that might interest you.
                      • Then there's also a more complex (but ESB based) quickstart called "huge-split-enrich-transform-route".  There are some docs (and a flash demo) for this one here.  The configs in the online docs are a bit out of date, but you'll get the jist..... look at the quickstart itself.  This example shows how to use the <esbr:routeBean> configuration for JBoss ESB, which allows you to route the split fragments to JBoss ESB endpoints.

                       

                      The basic idea of splitting & routing is simple enough....

                      1. Populate data from the source message into java beans (these can actually be Maps and Lists too, if you just want to apply a template - see "Virtual Object Models").
                      2. Repeatedly apply a template to the data in the beans as the message is filtered.
                      3. Repeatedly route the templating result to the target endpoint (using the <esbr:routeBean> to route to an ESB endpoint).
                      • 8. Re: "ForEach"-Splitter based on XML content
                        Bernd Ruecker Master

                        Hi Tom.

                         

                        Thanks for your answer. Actually I think the problem with all these examples is, they are too complex! Why should I create a java bean? Or a freemarker template? I just want to split an xml easily without big hazle or loads of code nobody will understand later on.

                         

                        That's what I like about Johns post, pretty simple, even if you still need a smooks configuration file.

                         

                        But with Johns Code the splitting works, but the individual messages doesn't have any content:

                         

                        body: [ objects: {org.jboss.soa.esb.message.defaultEntry=[job: null]} ]
                        

                         

                        I think somehow I have to tell Smooks, that it should copy the XML content in the new message, any hint? Sorry, but I am still not a Smooks expert (and I am not heading for it ;-)). John, did you do anything additional to make it working?

                         

                        Thanks and cheers

                        Bernd

                         

                        P.S @Tom: I really think, this is something the JBoss ESB should keep in mind, even if the current code is really powerful and you can do salmost everything with e.g. Smooks, it is too complex! Not only for beginners, but for a lot of projects and simple use cases. Splitting an XML message into individual ones by an XPath, why the hell isn't that just a few lines of code and includes different technologies and multiple configuration files? Think about maintenance of services for example. I think there is work to do to simplify some of these things, to catch up with tools like ServiceMix or Mule... Look at the ServiceMix code for example:

                         

                        <eip:xpath-splitter service="test:xpathSplitter" endpoint="endpoint"
                                            xpath="/*/*" namespaceContext="#nsContext">
                          <eip:target>
                            <eip:exchange-target uri="service:http://test/router" />
                          </eip:target>
                        </eip:xpath-splitter>
                        

                         

                        I don't want to stir up trouble or just complain, I want to see the JBoss SOA platform evolving in the right direction. And this is my view what is the biggest problem at the moment, experienced at a couple of customers already.

                        • 9. Re: "ForEach"-Splitter based on XML content
                          Bernd Ruecker Master
                          Small addition: The FragmentRouter is not yet there in soa-p 4.3.0, right?
                          • 10. Re: "ForEach"-Splitter based on XML content
                            Tom Fennelly Master

                            Hi Bernd.

                             

                            Thank you for sharing your thoughts.  You are totally right wrt that particular splitting solution being a bit too complex for the more basic use cases i.e. splitting out single-block fragments and routing them untransformed, which is what most people want.

                             

                            We actually created something like this a while ago in a "sandbox", but for whatever reason, we forgot to get it into Smooks project properly.  See this Smooks user list thread.  You could take this and try it out, if you like.

                             

                            I've created a JIRA in Smooks to add this properly: http://jira.codehaus.org/browse/MILYN-405

                            • 11. Re: "ForEach"-Splitter based on XML content
                              Tom Fennelly Master
                              Hmmm... right... fairly sure it is not in SOA-P 4.3.  Will be in SOA-P 5.
                              • 12. Re:
                                John Pount Newbie

                                Bernd,

                                 

                                Sorry for the longish delay - thankfully Tom is on the ball.

                                 

                                1. Code - I have attached a file with the ESB code for you - hopefully it's ok, but give me a shout if you have a problem. It follows the samples layout and I have been building with Ant and deploying to SOA 4.3

                                 

                                2. I took the code from the samples that Tom mentioned and cut them down to what I needed, so pretty basic (good)

                                 

                                3. "Missing" Content - I assume you are printing out to the console and seeing "null". This is weird because I was originally doing this and thought my dodgy code wasn't working , however if I dump the messages to a queue and pull them off with a client then the messages are not null (you'll see this in my sample code).

                                 

                                4. Fragment Router - this looks nice but I doesn't appear to be in 4.3 so that's why I'm using the other method.

                                 

                                That's it - hope the code works for you.

                                 

                                Cheers,

                                John

                                • 13. Re:
                                  John Pount Newbie

                                  Bernd,

                                   

                                  In my original email last week I spoke about a CSV file reader, conversion to XML and then XML splitter into multiple fragments. I have since seperated the splitter from the CSV file reader because I have several incoming endpoints - filereader, webservice, JMS client, which all accept documents/messages containing multiple orders. Each endpoint validates the input, creates a "common" XML message (Canonical) and then sends that through the splitter for processing.

                                   

                                  The file I uploaded earlier only contains the canonical message / splitter service but if you want the CSV file reader service as well please let me know.

                                   

                                  Cheers,

                                  John

                                  • 14. Re:
                                    Bernd Ruecker Master

                                    Hi John.

                                     

                                    Perfect, thanks for the code and the additional info. Indeed, I was printing the results on screen, so maybe it is really already working :-) But I am not at this customer for a few weeks and will try to come back to it as soon as possible and will let you know the rsult.

                                     

                                    So long, thanks a lot for sharing your ideas and code!!

                                     

                                    Cheers

                                    Bernd

                                    1 2 Previous Next