1 2 Previous Next 18 Replies Latest reply on Jan 13, 2011 7:28 AM by kcbabo

    CDI - Payload transformation

    tfennelly

      In fact, this is a wider discussion than CDI, but I've been looking at it as part of the CDI integration work, so we can discuss it in that context for now.

       

      First off… ultimately, I think we'll need to support the definition of transformations in a number of ways, but I'm only exploring one option here.  The option I'm looking at here is a code based option, where a service developer can define @Transformer classes that have simple @From and @To annotations on transformation methods.  These transformation methods are registered in a very simple TransformRegistry class, which is injected into each ServiceProxyHandler instance (the ServiceProxyHandler is the ExchangeHandler impl that accepts an ESB Exchange instance on behalf of a Service bean, unwraps the Exchange and invokes the appropriate Service bean method).

       

      When the handle method of the the ServiceProxyHandler is invoked, it checks the exchange context for the name of the target operation on the Service bean, as well as message type information (currently calling this a "PayloadSpec").  Yhe PayloadSpecs are very simple opaque string values that specify the IN and OUT data types of the Exchange.  IN is the data type the invoker is sending in on the Exchange, while OUT is the data type the invoker wants/expects back.

       

      For invoking the service, the ServiceProxyHandler gets the IN PayloadSpec (the @From) and the target operation's param sig (the @To), looks up and invokes the appropriate transformer method in the TransformRegistry.  This transforms the payload @From the Exchange type, @To the Service methods param type (currently only supporting a single arg Service bean method).  After invoking the Service bean method (for IN_OUT exchanges), the ServiceProxyHandler applies the same basic principles again, using the Service bean's return value type (the @From) and the Exchange's OUT PayloadSpec (the @To) to lookup and perform a transform on the response.

       

      So as an example, we can look at the unit tests where we have the following service...

       

      @Service
      public class WithProductsOrderManagementService {
      
          @Inject @Service
          private ProductService productService;
      
          public OrderResponse createOrder(OrderRequest request) {
              OrderResponse orderResponse = new OrderResponse(request.orderId);
      
              orderResponse.product = productService.getProduct(request.productId);
      
              return orderResponse;
          }
      }
      

       

      It has a single "createOrder" operation that takes a OrderRequest as input and returns an OrderResponse instance as output.  One of the tests uses a simple SOAP message to invoke this service and expects a SOAP response back.  The SOAP request and expected response look as follows….

       

      Request:
      <soap11:Envelope xmlns="urn:createOrderRequest:v1"
                       xmlns:soap11="http://schemas.xmlsoap.org/soap/envelope/">
      
          <soap11:Body>
              <createOrder>
                  <orderId>D123</orderId>
                  <productId>ABCD</productId>
              </createOrder>
          </soap11:Body>
          
      </soap11:Envelope>
      

       

      Response:
      <soap11:Envelope xmlns="urn:createOrderResponse:v1"
                       xmlns:soap11="http://schemas.xmlsoap.org/soap/envelope/">
      
          <soap11:Body>
              <createOrderResponse>
                  <orderId>D123</orderId>
                  <productId>ABCD</productId>
                  <productName>MacBook Pro</productName>
              </createOrderResponse>
          </soap11:Body>
      
      </soap11:Envelope>
      

       

      For these, we define the PayloadSpecs to be "urn:createOrderRequest:v1:soap" and "urn:createOrderResponse:v1:soap" respectively.  For this, we need 2 transform methods.  The first to convert @From "urn:createOrderRequest:v1:soap" @To an OrderRequest instance and the second to convert @From an OrderResponse instance @To "urn:createOrderResponse:v1:soap".  For this we defined the following Transformer class and added 2 transform methods as follows:

       

      @Transformer
      public class OrderModelTransforms {
      
          private Smooks readSOAP_V1;
          private Smooks writeSOAP_V1;
      
          public OrderModelTransforms() throws IOException, SAXException {
              readSOAP_V1 = new Smooks(getClass().getResourceAsStream("createOrderRequest_v1_read.xml"));
              writeSOAP_V1 = new Smooks(getClass().getResourceAsStream("createOrderResponse_v1_write.xml"));
          }
      
          public OrderRequest readSOAP_V1(@From("urn:createOrderRequest:v1:soap") String inXML) {
      
              JavaResult javaResult = new JavaResult();
              readSOAP_V1.filterSource(new StringSource(inXML), javaResult);
              return javaResult.getBean(OrderRequest.class);
          }
      
          public void writeSOAP_V1(@From OrderResponse orderResponse, @To("urn:createOrderResponse:v1:soap") Writer outXML) {
      
              JavaSource source = new JavaSource(orderResponse);
              source.setEventStreamRequired(false);
              writeSOAP_V1.filterSource(source, new StreamResult(outXML));
          }
      }
      

       

      In this case we used Smooks to perform both transformations, but you can obvsiously use whatever you like (JAXB etc).

       

      In the first method (readSOAP_V1) you'll note we don't specify a @To annotation.  This is implicit in the OrderRequest return type of the method.

       

      Invoking the Service in this case is a matter of setting SOAP as the exchange message payload and then setting 2 Exchange Context properties specifying the target operation name + the IN and OUT PayloadSpecs:

       

      Exchange exchange = domain.createExchange(new QName("WithProductsOrderManagementService"), ExchangePattern.IN_OUT, responseConsumer);
      
      ServiceProxyHandler.setOperationName(exchange, "createOrder");
      PayloadSpec.setInPayloadSpec(exchange, "urn:createOrderRequest:v1:soap");
      PayloadSpec.setOutPayloadSpec(exchange, "urn:createOrderResponse:v1:soap");
      

       

      I didn't want to add annotations on the service itself (to point to transformation logic) as that seemed to be too tight a coupling, but also, it seems logical to me that we will need to be able to perform different transformations, depending on the source of the service invocation.

       

      Another option I'd be interested in exploring is how we might hook this into the ExchangeImpl itself, so that when we call the Exchange.getContent(Class<T> type) we can apply an appropriate transformation based on the type/PayloadSpec of the content of the Exchange Message (the @From) and the Class type specified in the getContent call (the @To).

        • 1. Re: CDI - Payload transformation
          kcbabo

          Lots of interesting stuff here.  Since I can't seem to post a reply in this forum without a bulleted list in it, here ya go:

           

          • I think there are two levels of transformation we need to consider: transformation between formats (e.g. XML -> CSV) and transformation within a format (e.g. com.example.MyObject -> com.example.YourObject).  I like the idea of identifying the format itself using a string literal, the same way you specify mime types.  It seems like each format would have a handler (or possibly multiple handler implementations), but I don't think you would have a separate handler for each transformation within a message format.  Let me know if this sounds crazy and I can whip up an example.
          • You mentioned that you currently have this implemented in the ServiceProxyHandler.  Would it make sense to build a general purpose transformation framework implemented as an exchange handler?

          • I agree that specifying the transformation details on the service contract is nasty.  I do think that we can have some type of configuration (Java interface annotation and/or switchyard configuration) that identifies which formats a service is capable of receiving and returning.  This can be useful at design-time to figure out whether a separate transformation step is required to talk to a service.
          • Weaving this into Exchange.getContent(Class<T> type) is interesting, but I wonder if this is even necessary if the transformation is carried out by an ExchangeHandler?  IOW, the transformation occurs as part of the handler processing and by the time you call getContent() the content has already been transformed to the appropriate format.
          • 2. Re: CDI - Payload transformation
            kcbabo

            Oh, forgot to mention that transforming from an entire SOAP envelope into a Java object is probably not a common a use case as going from the content of SOAP:Body to a Java object.  For example, SOAP gateway will pull off the envelope and stick the HTTP and SOAP headers in the message context, so all the transformer will only have to deal with the SOAP body.  Now, if you used a raw socket or HTTP gateway, then you would have to strip out the SOAP body yourself, as depicted in your example.

             

            I know we already talked about this, but I'm writing in the forum so that other folks can participate.

            • 3. Re: CDI - Payload transformation
              tfennelly
              It seems like each format would have a handler (or possibly multiple  handler implementations), but I don't think you would have a separate  handler for each transformation within a message format.

               

              If you mean e.g. @From("A") @To("B") you'd have just one transfom handler... yeah, that's what I'd assume.

               

              You mentioned that you currently have this implemented in the  ServiceProxyHandler.  Would it make sense to build a general purpose  transformation framework implemented as an exchange handler?

               

              I thought about that.  The reason it makes sense in the ServiceProxyHandler is because it has access to the Servcie Bean impl and so is able to resolve the @To PayloadSpec for the target operation.  Of course we could jump through some hoops to avoid this, but doing it in the SPH was the easiest for now.

               

              I do think that we can have some type of configuration (Java interface  annotation and/or switchyard configuration) that identifies which  formats a service is capable of receiving and returning.

               

              I think we have this already for CDI bean Services, through the @Service interface definition.  The @Service interface defines the operation params and return types (what the consume and produce).  Not sure I'd like the idea of addding more metadata on these.  Not sure it makes sense to me.

               

              Weaving this into Exchange.getContent(Class<T> type) is  interesting, but I wonder if this is even necessary if the  transformation is carried out by an ExchangeHandler?  IOW, the  transformation occurs as part of the handler processing and by the time  you call getContent() the content has already been transformed to the  appropriate format.

               

              So this makes sense to me from the POV of when the transformation happens.  With what we have there at the moment, both the inbound and the outbound transformation happens right at the Service.  If you were doing it as part of the Exchange.getContent(Class<T> type), you'd only do it "on-demand", which seems to have the following benefit to me...

               

              The Service doesn't need to have access to a transformer that can produce the data type required by the consumer.  With the current approach of doing it in the SPH, that transformer logic (to get it to the format required by the consumer) needs to be "in the Service" exec namespace.  You could argue that this is a good thing anda bad thing, but I think it's not a good thing.  If you do it through the call to Exchange.getContent(Class<T>), the transformation happens at the consumer side and it's the consumer that needs to know how to do the conversions.  In fact... I think you may have scenarios where both could make sense at the same time i.e. having the SPH perform a transformation of the response Java object to one of it's supported non-java (transportable) types.  On receipt, the consumer/client side may not be able to consume this type directly and so may need to perform a second transform e.g. bind it into it's own java object model, which would be different to that of the Service's object model.  All that said... you could handle that in the ClientProxyBean for CDI remote invocations, but you could also handle it more transparently in the Exchange.getContent(Class).

              • 4. Re: CDI - Payload transformation
                tfennelly

                Just back to this comment again actually Keith...

                 

                I do think that we can have some type of configuration (Java interface  annotation and/or switchyard configuration) that identifies which  formats a service is capable of receiving and returning.

                 

                I don't think we'd need to define this explicitly in code... we could do it through something like the TransformRegistry we currently have there.  If you look at the return type for a particular Service operation, you should then be able to use that info with the info in the TransformRegistry to work out what possible types the "Service side" can return back in the Exchange for that type... pick one that suits you and...

                • 5. Re: CDI - Payload transformation
                  kcbabo

                  In my mind, it makes more sense to put this in a handler vs. the bean component (what I'm now calling your CDI work :-) ) because it's a cross cutting concern that (as you mentioned) impacts more than one component.  We need a way to specify that transformation is required when connecting the consumer of a service to the provider.  For example, let's take your example where the consumer is a raw HTTP gateway that accepts an entire SOAP envelopes and forwards that on to the service provider which is expecting a Java object.  Here's what needs to happen:

                   

                  1) There needs to be a way to declare what the source format is that the consumer is attaching to the message.  In your example, it's XML.  This may be configuration that's built-in to the HTTP gateway and SwitchYard configuration.  It could be a side effect of which MessageBuilder you use to create the Message instance.  There are a number of ways to get it done.

                  2) There needs to be a way to declare the expected format in the provider.  In your example, it's Java object.  Now, we can specify this in the configuration of the service itself (not necessarily in code) or we can follow the getContent() route and dynamically convert it.  The reason why specifying in configuration is better, IMHO, is that you can catch data format mismatches at validation/deployment time instead of getting crazy errors during runtime.

                  3) Once we know the source format and the target, we can look for transformers that fit the bill.  I think this is where something like a transformer registry might come into play.  I imagine there will be general purpose transformers that use brute force to convert between formats.  There will also certainly be a way to add your own transformation logic into the mix as part of your application deployment or as a separate event (e.g. like a library).  This would most likely use a syntax similar to the @From and @To you have defined above.

                   

                  I'm going to stop talking before I feel the urge to add bullet points. ;-)

                  • 6. Re: CDI - Payload transformation
                    tfennelly

                    You didn't use bullets, but you did use numbers

                     

                    So I'm not sure how much you've looked at what I've actually done, but it's quite close to what you've outlined above and the general direction it is heading is the same.  I did some bits slightly different simply because it was the easiest way to get something working.

                     

                    So I'll break into numbers here in an effort to cross ref between what you are saying and what we have in code:

                     

                    1. Declare Source Format from the Consumer:  This is currently being set on the Exchange Context.  However In reality, I think you'd need to say more than just "XML", but that's no biggie since it's just an opaque string and as long as they are consistent and match up.  Should be one of the providers "expected formats" (#3).
                    2. Declare Source Format expected back by/to the Consumer:  I don't think you mention this in your outline above, but we're setting this on the Exchange Context (ala #1 above).  This would need to be one of the possible response formats supported by the provider (#4).
                    3. Supported Request formats of the Provider operation (aka "expected format")We're not explicitly declaring this at the moment (e.g. in config or in  the registry) simply because we've no real way of doing so yet, bar  setting all the info on the exchange, where it doesn't seem to belong  (to me anyway).  As I see it, the only mechanism for "contract  definition" we currently have is the Java interface for the Service.  I  totally agree that having it explicitly defined as part of the service  contract/config is a good idea for the reasons you outline (catching  errors etc).  I do think however that we should be able to allow the  current more informal approach too as it's more lightweight.  I think  the crazy errors you might get would be hit and fixed first time you run  a unit test.
                    4. Supported Response formats of the Provider operation:  Basically the same as #3.
                    5. Transformation Lookup:  Yes... this is what we're using the TransformRegistry for and we can get more elaborate with it over time.  In any case (brute force or elaborate), I think the @From and @To mechanism would basically be the same.
                    6. Payload Transformation Exchange Handler:  This is currently in the ServiceProxyHandler (which is an Exchange Handler), but I do agree that it makes sense in a standalone handler.  As I stated earlier, the reason it's currenly in the SPH is simply because it's easier to get at the info outlined in #3 and #4 above, since it has access to the Service bean.
                    7. Transformation Points in an Exchange:  I think this is an important topic for us to be clear/formal on.  Below is a simple schema of what I currently think makes sense.

                     

                    Screen shot 2010-12-01 at 11.27.42.png

                     

                    The transformations e.g.:

                    1. @From Consumer IN format @To "A"
                    2. @From "A" @To Java
                    3. @From Java @To "B"
                    4. @From "B" @To Consumer OUT format

                     

                    With the current impl, the only transformation points are at 2 and 3.

                    • 7. Re: CDI - Payload transformation
                      kcbabo

                      You show four transformation points in your diagram, but couldn't we actually achieve the same result with two transformation points?

                       

                      C (in)    ----------------- T1 -->  P(in)

                      C (out) <---T2--------------- P(out)

                       

                      I didn't realize that you had committed code for this in your repo, so I was just going off the example code in your post.  I will check it out and respond wth more informed comments in a bit.  Overall, I feel like we are pretty close in our view of how this will work.  Another thing I'm working on today is roughing out the configuration stuff, which I think will help in this discussion since we keep falling back to "or it can be in the config".  End of the day, everything we talk about should be simple and clean to implement with just the API (i.e. no config required), but I think a rough idea of the config helps set up the larger picture of how this will all be used.

                      • 8. Re: CDI - Payload transformation
                        tfennelly

                        You show four transformation points in your diagram, but couldn't we  actually achieve the same result with two transformation points?

                         

                        C (in)    ----------------- T1 -->  P(in)

                        C (out) <---T2--------------- P(out)

                         

                        That's right and is how it's currently implemented ... quoted from above... "With the current impl, the only transformation points are at 2 and 3" i.e. two transforms at the provider side... going in and going out.

                         

                        We're probably splitting hairs a bit and perhaps we could just make life simple for ourselves and say that the transformations at the consumer side of the equation are not the responsibility of the ESB e.g. from the example in my previous post...  it's up to the consumer to work out how to produce "A" going IN and consume "B" coming OUT.

                         

                        Maybe I'm missing something, but if we didn't provide the ability to hook in the TransformHandler (to be created ) on the consumer side of the exchange (basically, in any HandlerChain, when required), then how would we support a use case where e.g. the FileGateway is the Consumer (in our above example) and is reading files containing data format "X"?  The X data needs to be transformed to A before routing to the provider.  It's obviously not the job of the gateway impl to do this... it's the job of the exchange mechanism... right? ... and the transform needs to happen before the Exchange hits the provider (because it only understands A and not X).

                         

                        Anyway... yeah... I think we're more or less on the same page (I hope) .  I'll separate out the transform logic (that's currently in the ServiceProxyHandler) into it's own TransformHandler ExchangeHandler.  In the case of our CDI components, our CDI Extension can work out how to configure it and put into the handler chain.

                        • 9. Re: CDI - Payload transformation
                          kcbabo

                          Maybe I'm missing something, but if we didn't provide the ability to hook in the TransformHandler (to be created ) on the consumer side of the exchange (basically, in any HandlerChain, when required), then how would we support a use case where e.g. the FileGateway is the Consumer (in our above example) and is reading files containing data format "X"?  The X data needs to be transformed to A before routing to the provider.  It's obviously not the job of the gateway impl to do this... it's the job of the exchange mechanism... right? ... and the transform needs to happen before the Exchange hits the provider (because it only understands A and not X)

                           

                          The gateway impl should definitely not be burdened with the changing formats.  For that matter, the provider implementation should not be burdened either, which is why putting this logic in a handler makes sense.  I think I see where our wires are crossing here.  A service provider provides a handler when registering a service today.  This handler gets stuck into a handler chain that the core runtime controls.  So all messages that move from consumer -> provider will pass through this chain.  We can add a handler to this chain that checks to see if the source and target formats match, and if they do not, attempt to resolve a transformation/conversion that will make the match.

                           

                          In the guts of the implementation, the picture looks exactly as you drew above.  There are a set of handlers that fire every time a message is sent and every time it's delivered.  Where we anchor the actual transformation is an implementation detail.  To the user, it should just look like a wire that connects the consumer and provider.

                          • 10. Re: CDI - Payload transformation
                            tfennelly

                            OK... think we'er on the same page then.  Where we anchor transformations is an impl detail, but one we need to work out.

                            • 11. Re: CDI - Payload transformation
                              kcbabo

                              Been beating my head against the wall the last couple days trying to get this right in my mind.  I went through your prototype work a number of times and I think the key elements are represented.  I'm going to throw out an initial proposal based on your code that's very basic and focuses on the API only.  We can then talk about how configuration maps into the API and address any cases that are not handled by the basic API.

                               

                              Transformation between formats is handled by an instance of Transformer.  We will most likely provide a set of core transformers, similar to converters that are present in ESB 4.x.  On top of that, users can provide their own implementation of Transformer to handle specific transformation use cases.

                               

                              interface Transformer<F,T> {
                                  T transform(F fromType);
                                  Class<F> getFrom();
                                  Class<T> getTo();
                              }
                              

                               

                              Transformer instances are made available to the core runtime through a TransformerRegistry, which can be anchored to ServiceDomain (e.g. ServiceDomain.getTransfomerRegistry()).

                               

                              interface TransformerRegistry {
                                  void addTransformer(Transformer transformer);
                                  boolean removeTransformer(Transformer transformer);
                                  List<Transformer<F,T>> getTransformers(Class<F> fromType, Class<T> toType);
                                  List<Transformer<F,?>> getTransformersFrom(Class<F> fromType);
                                  List<Transformer<?,T>> getTransformersTo(Class<T> toType);
                              }
                              

                               

                              Transformers execute in the context of an exchange handler, TransformHandler, which is always present in the core handler chain.  TransformHandler uses configuration (as yet unspecified) to determine whether translation is required and the TransformerRegistry to locate an appropriate Transformer to execute.

                               

                              public TransformationHandler extends BaseHandler {
                                  public void handleMessage(Exchange exchange) {
                                       // 1 - determine if transformation is required
                                       // 2 - lookup required transformer
                                       // 3 - transform message content and set transformed content on message
                                  }
                              }
                              

                               

                              On the configuration front, there are two different ways to go.

                               

                              1) The handler can look at the message content type and compare that to the service interface to determine which content types are accepted by the service.  It would then perform a lookup in the transform registry to find a transformer that matches the source and target formats.

                               

                              2) A specific transformer could be identified as part of the switchyard configuration for the service reference - e.g. when an @Reference is injected into bean component, that reference can be configured to execute a transform as part of the handler chain.

                               

                              I've been going back and forth on #1 and #2.  I'm gonna rough out what each option might look like and we can take it from there.

                              • 12. Re: CDI - Payload transformation
                                tfennelly

                                So just on the API then...

                                 

                                How does a Class type represent the From/To?  How would it represent e.g. an "orderCreate" message from the "http://acme.com/OM/v1/xml" namespace?  Seems that something like a QName would be more appropriate.

                                 

                                I still think it would be nice to allow multiple related transformers to be specified on a single transformer class.  This could easily translate into multiple Transformer instances in the registry, one for each transform method (ala the earlier prototype).

                                 

                                Do we really need three getTransformer methods on the registry?

                                • 13. Re: CDI - Payload transformation
                                  kcbabo

                                  How does a Class type represent the From/To?  How would it represent e.g. an "orderCreate" message from the "http://acme.com/OM/v1/xml" namespace?  Seems that something like a QName would be more appropriate.

                                   

                                  Sorry, forgot to address this in my last post.  From my POV, the Transformer needs to *start* at the Java type level.  We might need to have another identifier to resolve a specific format for the Java type, but the type is required no matter what.  So I started with type and we can add a literal identifier (string, qname, whatever) for from/to if that's required.

                                   

                                  To your specific example of "orderCreate", I think there are multiple ways to handle this case.  One, as you suggested, is to use the @From and @To annotations to specifically identify the source and target formats.  If I understand where you are coming from, the value for @From/@To would point to the message type used in the service interface.  That's how I was reading the urn: string in your prototype.  If the idea is that we will use the namespace from the XML document itself, I don't think that will work as non-XML formats will not have a namespace defined.

                                   

                                  A contrasting approach would be to specifically identify the transformer used for a given service.  This would be part of the service consumer's configuration and the information required to make this decision would be available at design/development-time.  This approach would not require a TransformerRegistry, since the config would point to the transformer.

                                   

                                  I still think it would be nice to allow multiple related transformers to be specified on a single transformer class.  This could easily translate into multiple Transformer instances in the registry, one for each transform method (ala the earlier prototype).

                                   

                                  I agree that a single class with multiple transforms could show up as independent transformers in the registry.   My proposal above was just an example of how we could get strong-typing of the Transformer class itself as well as the types being converted.  I'm guessing that the approach we choose will depend a lot on how the @From/@To conversation pans out.  Happy to go with whichever method is clearest.

                                   

                                  Do we really need three getTransformer methods on the registry?

                                   

                                  Definitely need one to pull out the list of transforms that handle a given from/to match.  The other two are simply a convenience to pull a list of transformers that match a given source or destination - e.g. give me a list of possible conversion targets for format "xyz".  I don't have a specific use case in mind where this is required, so we can drop it for now.

                                  • 14. Re: CDI - Payload transformation
                                    tfennelly
                                    Sorry, forgot to address this in my last post.  From my POV, the  Transformer needs to *start* at the Java type level.  We might need to  have another identifier to resolve a specific format for the Java type,  but the type is required no matter what.  So I started with type and we  can add a literal identifier (string, qname, whatever) for from/to if  that's required.

                                     

                                    TBH... I think the identifier is the key to this.  The Java type/interface through which the transformer accesses the message "data" (e.g. the XML, EDI, CSV, Java, etc) is kinda incidental imo.  It just so happens that you wouldn't need an explicit identifier in situations where the message "data" is a concrete Java type e.g. com.acme.Order.   In that situation, I feel the "identifier" is inbuilt and reflectively available.

                                     

                                    When the data is presented as XML, CSV, EDI or some other non-Java data type (via a String, InputStream, Reader, byte[] etc), the Java type there is completely useless.  To me, the Java type... String, InputStream, Reader or concrete Java type (e.g. com.acme.Order) is just the data carrying mechanism.

                                     

                                    If the idea is that we will use the namespace from the XML document  itself, I don't think that will work as non-XML formats will not have a  namespace defined.

                                     

                                    No doubt namespaces are widely/synonymously used in XML, but I don't see any reason why data presented in other formats (CSV, EDI, Java, JSON etc) cannot be assigned to a namespace/urn.

                                    1 2 Previous Next