12 Replies Latest reply on Apr 9, 2009 5:31 AM by davsclaus

    camel-fix usage

    rodehav

      Hello again!

       

      Lots of camel-fix questions from me lately...

       

      I am not an experienced user of Camel but it seems perfect for what I want to do. I want to use QuickFix/J for Fix communication (in this case receiving drop-copies from a market place for settlement purposes) and then trigger a classical integration flow with transformation and routing of the received messages.

       

      With help from answers to my prior post here, I'v gotten the camel-fix component up-and-running. However I still have a couple of problems/questions:

       

      1. By specifying "from("fixserver:executor.cfg").to("log:quickfix");", I manage to start the QuickFix/J engine using the configuration file "executor.cfg". However, the received messages are not forwarded further in the flow (in this case to the log component). When I look in the source code for camel-fix I noticed that the only thing being done in the "fromApp()" callback in the QuickFix/J application is to log the message. But in the "toApp()" callback the message is forwarded to the "onMessage()" method for further processing. To me it seems it should be the other way around. If I move the call to "onMessage()" from "toApp()" to "fromApp()", things starts to happen. Can someone verify if this is a bug or if I've just misunderstood?

       

      2. It seems like camel-fix is depending an code that is not open sourced. Specifically, the FixConverter class depends heavily on biz.c24.io.fix42.NewOrderSingleElement which I don't seem to be able to download from anywhere. I get the impression that it is part of Artix Data Services that is a commercial product. Can someone clarify whether camel-fix will be an open source available Camel integration or if it will be a commercial product? I understand if advanced type conversion and data services will not be open sourced, but the basic type conversion and Camel integration I think is a good candidate for open source. I'd rather use an integration from FUSE than to roll my own.

       

      3. Assuming that camel-fix will be freely available, I still need a little more hints on how to use it. It seems like the actual camel integration is to let the infrastructure create a FixEndpoint that the QuickFix application holds on to. It can then be used to communicate with Camel. I think a good approach would be for the user (me) to subclass the CamelApplication class in order to get my specific behaviour (especially my own initializing in order to use my persistent store etc) but being able to use the provided FixEndpoint to talk to Camel. Is this a good idea or did you have another usage scenario in mind? If it's a good idea, what is the best way for me to accomplish that?

       

      /Bengt

        • 1. Re: camel-fix usage
          rodehav

          I've come one more step along the way to use camel-fix but needs some directions. I created my own CamelApplication class and, in the classpath, I made sure to put it before the CamelApplication class included in the camel-fix component. My own version was identical to the one included in camel-fix except that I moved the call to onMessage() from the toApp() method to the fromApp() method. This made it possible for me to receive messages and process them in a camel route. It seems to me that this is a bug that needs to be corrected (maybe someone can create an issue for this?).

           

          However, being new to Camel I can't figure out how to reply to a fix message. E g, simulating the Executor in the QuickFix/J example, when receiving an Order, I would like to reply with an ExecutionReport. How do I accomplish this?

          • 2. Re: camel-fix usage
            davsclaus

            Hi

             

            Please keep us updated with your findings.

             

            We are currently busy with some other areas in Camel but will get back to the camel-fix component later.

            • 3. Re: camel-fix usage
              rodehav

              Hello there!

               

              I'll take you up on the offer to keep you updated on my findings. I've been experimenting a bit more and feel I have a few points for you to consider including in camel-fix.

               

              Two bugs (I think):

               

              1. As stated earlier, I'm pretty sure that you have mistakenly switched the code in CamelApplication.toApp() with the CamelApplication.fromApp(). fromApp() is the one that is called when the application needs to process an incoming message. I've moved the call to "endpoint.onMessage(message, sessionID);" and put it in fromApp(). This works fine.

               

              2. The initiator is never started in FixClientEndpoint.login(). This means that the FIX client is never started. E g writing code like the following doesn't work in a unit test:

               

                  from("direct:start").to("fix:initiator.cfg");

               

              By changing FixClientEndpoint.login() to the following, the above will work:

               

                  protected void login(SessionSettings settings, Application application, MessageStoreFactory storeFactory, LogFactory logFactory) throws Exception {

                      initiator = new SocketInitiator(application, storeFactory, settings, logFactory, getMessageFactory());

                      initiator.start();

                  }

               

              I also have 4 cases where I need to configure QuickFix/J differently than what is being done in camel-fix:

               

              1. Support for user/password.

              Normally you have to add user/password in the callback CamelApplication.fromAdmin(). My version looks like this:

               

                  public void fromAdmin(Message message, SessionID sessionID)

                          throws FieldNotFound, IncorrectDataFormat, IncorrectTagValue,

                          RejectLogon {

                      if (LOG.isDebugEnabled()) {

                          +LOG.debug("fromAdmin() session: " + sessionID + " " + message);+

                      }

               

                      final Message.Header header = message.getHeader();

                      if (header.getField(new MsgType()).valueEquals(MsgType.LOGON)) {

                          if (endpoint.getUser() != null) {

                              if (LOG.isDebugEnabled()) {

                                  +LOG.debug("fromAdmin(), logging on " + endpoint.getUser());+

                              }

                              +LOG.info("fromAdmin(), logging on " + endpoint.getUser());+

                              message.setField(new Username(endpoint.getUser()));

                              if (endpoint.getPassword() != null) {

                                  message.setField(new Password(endpoint.getPassword()));

                              }

                          }

                      }

                  }

               

              As you can see I assume that the user and password is accessible throught the endpoint. To get this working I've changed the FixEndpoint class as follows:

                  ...

                  private String user;

                  private String password;

               

                  public String getUser() {

                      return user;

                  }

               

                  public String getPassword() {

                      return password;

                  }

                  ...

                  public void setSessionID(SessionID sessionID) {

                      final String USER_KEY = "camel.user";

                      final String PASSWORD_KEY = "camel.password";

                       

                      this.sessionID = sessionID;

               

                      if (settings.isSetting(sessionID, USER_KEY)) {

                          try {

                              user = settings.getString(sessionID, USER_KEY);

                          } catch (ConfigError e) {

                              +LOG.fatal("Could not retrieve the property " + USER_KEY, e);+

                          } catch (FieldConvertError e) {

                              +LOG.fatal("Could not retrieve the property " + USER_KEY, e);+

                          }

                          if (settings.isSetting(sessionID, PASSWORD_KEY)) {

                              try {

                                  password = settings.getString(sessionID, PASSWORD_KEY);

                              } catch (ConfigError e) {

                                  LOG.fatal(

                                          +"Could not retrieve the property " + PASSWORD_KEY, e);+

                              } catch (FieldConvertError e) {

                                  +LOG.fatal("Could not retrieve the property " + PASSWORD_KEY, e);+

                              }

                          }

                      }

                  }

               

              I assume the properties "camel.user" and "camel.password" in QuickFix/J's configuration file. I think you could use this technique (with some suitable namespace) to enable Camel specific settings that make your classes more general.

               

              2. Support for specifying what MessageStoreFactory to use.

              I currently just patched FixEndpoint.createMessageStoreFactory() as follows:

               

                  protected MessageStoreFactory createMessageStoreFactory(

                          SessionSettings settings) {

                      +LOG.debug("createMessageStoreFactory: " + settings);+

                      return new FileStoreFactory(settings);

                      // return new JdbcStoreFactory(settings);

                  }

               

              Ideally it should be possible to configure what MessageStoreFactory to use via a Camel specific property in QuickFix/J's configuration file. Similar to the user/password approach.

               

              3. Support for specifying what LogFactory to use.

              I currently just patched FixEndpoint.createMessageStoreFactory() as follows:

               

                  protected LogFactory createLogFactory(SessionSettings settings) {

                      // I use this for testing

                      return new CompositeLogFactory(new LogFactory[]{new ScreenLogFactory(settings), new SLF4JLogFactory(settings)} );

                       

                      // This is what is being done today in camel-fix

                      // return new ScreenLogFactory(settings);

                       

                      // This is what I will use in production

                      return new SLF4JLogFactory(settings);

                  }

                   

              Currently the ScreenLogFactory is hard coded in camel-fix. This is inadequate for production use. I think this must also be configurable via a Camel specific property in the QuickFix/J settings file.

               

              4. - JMX support.

              In production I think this is a must. In order to enable this in QuickFix/J I changed the login() method in FixClientEndpoint and FixServerEndpoint as follows:

               

              FixClientEndpoint:

                  protected void login(SessionSettings settings, Application application, MessageStoreFactory storeFactory, LogFactory logFactory) throws Exception {

                      initiator = new SocketInitiator(application, storeFactory, settings, logFactory, getMessageFactory());

                      initiator.start(); // I added this line as well

                       

                      JmxExporter jmxExporter = new JmxExporter();

                      jmxExporter.export(initiator);

                  }

               

              FixServerEndpoint:

                  protected void login(SessionSettings settings, Application application, MessageStoreFactory storeFactory, LogFactory logFactory) throws ConfigError, JMException {

                      acceptor = new SocketAcceptor(application, storeFactory, settings,

                              logFactory, getMessageFactory());

                       

                      JmxExporter jmxExporter = new JmxExporter();

                      jmxExporter.export(acceptor);

                       

                      acceptor.start();

                  }

               

              Originally I also had my own FixConverter class since the one provided with camel-fix required classes from Artix Data Services that were not open sourced (or at least not accessible). Since the release of FUSE Mediation Router 1.5.4.0 the classes seem to be included. However, the source code is not available for those classes which means that I cannot build camel-fix by myself. This doesn't seem right for an open source product.

               

              I realize that the 4 feature requests above are needed because  QuickFix/J doesn't enable everything to be configured in the settings file but requires a lot of configuration to be done in code. E g I cannot understand why QuickFix/J doesn't  support specifying user/password in the initiator's configuration file. But things are as they are which means that to make camel-fix usable you have to add ways to make these things configurable without requiring it to be coded. I think you probably have to make it possible for users (like me) to be able to subclass necessary classes in camel-fix in case they are not general enough. As you can see above, I've found a number of things that I must be able to configure but I'm sure I will find more when I start using camel-fix more. I will then need a better fallback then my current which is to put my own versions of your classes first in the classpath.

               

              At the risk of being even more troublesome for you, I have two more requests...

               

              a) QuickFix/J 1.4 was just released with lots of bug fixes and support for FIX 5.0. It would be great if camel-fix supported (was compiled aginst) that version of QuickFix.

               

              b) I Need to use FUSE Mediation Router with Websphere MQ and have noticed that fixes concerning this has been added to the version 2.0 branch. When is this due for release? I noticed that Apache Camel 1.6 was recently released which means that FUSE is now lagging compared to the Apache project.

               

              My dream scenario is to have:

               

              - The two bugs fixed

              - More possibilities of configuring camel-fix

              - An easy way to subclass your classes when configuration is not enough

              - Support for QuickFix/J 1.4

              - A quick release of FUSE Mediation Router 2.0 (or other version that includes all above camel-fix changes as well as the MQ improvements I've seen)

               

              I'll be gone skiing for a week (without a computer) so I won't be able to answer any replies until after that.

               

              Hope you didn't choke on this post - remember that I really like your product and it really gives value for me to be able to use it in conjunction with QuickFix/J.

               

              Best regards,

               

              /Bengt

              • 4. Re: camel-fix usage
                davsclaus

                Hi Bengt

                 

                Thanks a lot for your detailed findings.

                 

                The good news is that we are working together with Anthon Arhipov that is author of camel-quickfix component

                http://code.google.com/p/camel-quickfix/

                 

                He is doing improvements to the camel-fix component to do a camel-quickfix component that will be moved into the Apache distribution, eg. Apache Camel. And of course also standard in FUSE-MR.

                 

                This will untie us from the Artix DS and we will create a camel-fix-ads component in FUSE MR for end users relying on Artix DS.

                 

                Anthon will/has upgrade to QuickFix 1.4.0.

                 

                We have a ticket in Apache for this task:

                https://issues.apache.org/activemq/browse/CAMEL-1350

                 

                I will ping Anthon about your findings.

                 

                If the demand is high we could backport it into 1.x but we are aiming at 2.0 currently.

                 

                We plan to release a milestone release of Camel 2.0 first. Would like for end users to try it out a bit earlier. You must expect that you might have to recompile, and rename some routes etc. as we have redefine the public API a bit. This was our overall goal for 2.0 to get the public API sorted. The milestone release should be ready in a few weeks.

                 

                The next FUSE MR release will of course be based on 1.6. out next month. Sometimes FUSE MR is behind when a new release is released at Apache. But often is is ahead as we do monthly releases, as opposed to the Apache is only done 2-3 times a year.

                • 5. Re: camel-fix usage
                  davsclaus

                  Hi

                   

                  BTW: What WebsphereMQ improvements have you seen in Camel 2.0? I cant recall we have done anything only for 2.0 in this regard? But as usual we have been very busy so there are many issues done in 2.0 over 1.x

                  • 6. Re: camel-fix usage
                    arhan

                    Hello Bengt,

                     

                    I'm aiming to add to  camel-quickfix  almost everything you have listed.

                     

                    I've also noticed the two bugs you have mentioned.

                     

                    Configuring the quickfixj (logging factory/ message store/ etc) resources should be done via URIs. I will add that very soon.

                     

                    QuickfixJ 1.4 is included and also I'm developing with Camel 2.0-SNAPSHOT version.

                     

                    What I'm missing at the moment is the converter from Exchange into Message object.

                     

                    Anton

                    • 7. Re: camel-fix usage
                      cmoulliard

                      @All,

                       

                      In order to facilitate the parsing/formatting (or in camel language unmarshaling/marshaling) of FIX messages in camel, I have extended the camel-bindy component to support key value pair messages (which is the format used in FIX messages).

                       

                      So it will be possible in combination with camel-quickfix to receive the data stream of the messages from quickfix gateway and to unmarshall them in camel using the following route :

                       

                      from("fix:server")

                      .unmarshall( BindyKeyValuePairFormat )

                      .to(bean:OrderProcess)

                       

                      or

                       

                      from(bean:OrderProcess)

                      .marshall( BindyKeyValuePairFormat )

                      .to(fix:server)

                       

                      The patch has been attached to the following ticket (https://issues.apache.org/activemq/browse/CAMEL-1364) and the documentation updated on the web site (http://cwiki.apache.org/CAMEL/bindy.html). Two examples have been added in the unit tests.

                       

                      Regards,

                       

                      Charles Moulliard

                      SOA Architect

                      Xpectis

                      • 8. Re: camel-fix usage
                        rodehav

                        Hi there!

                         

                        Back from my skiing vacation now...

                         

                        Your plan seems perfect. When do you think I can expect a 2.0 release? You say the next month's release will be a 1.6 release. Will the release after that be 2.0?

                        • 9. Re: camel-fix usage
                          rodehav

                          Hi!

                           

                          Looking forward to test the new version of camel-fix.

                           

                          I don't know if it helps but at first I added the following methods to the FixConverter class. It utilizes the fact that a Message can be transformed to/from a String. I'm sure you're ahead of me at this but it seemed to work for the simple things I tested.

                           

                              @Converter

                              public static String toString(Message message) throws IOException {

                                  return message.toString();

                              }

                           

                              @Converter

                              public static Message convert(InputStream in) throws IOException,

                                      InvalidMessage {

                                  final int BUFFER_SIZE = 8192;

                                  byte buffer[] = new byte [[BUFFER_SIZE]];

                                  ByteArrayOutputStream baos = new ByteArrayOutputStream();

                                  int count;

                                  count = in.read(buffer);

                                  while(count != -1) {

                                      baos.write(buffer, 0, count);

                                      count = in.read(buffer);

                                  }

                           

                                  String str = baos.toString("ISO-8859-1");

                                  in.close(); // Didn't work without this line

                                  return new Message(str);

                              }

                          • 10. Re: camel-fix usage
                            rodehav

                            Hi!

                             

                            It's probably not an MQ specific problem. You can read about it at this URL:

                             

                            http://www.mail-archive.com/camel-user@activemq.apache.org/msg04568.html

                             

                            Here it looks like the fix is added to the 2.0 branch:

                             

                            http://issues.apache.org/activemq/browse/CAMEL-505

                             

                            I haven't tried this out yet but I do have the situation where I will write to MQ via Java (JMS) but the reader will be a Microsoft program. I anticipate problems unless I have someway of telling MQ not to include the JMS headers in the message.

                            • 11. Re: camel-fix usage
                              rodehav

                              Just curios if you have any new information as to the improvements of camel-fix and when I can expect them in a release.

                              • 12. Re: camel-fix usage
                                davsclaus

                                Hi

                                 

                                I dont think Anton had got so far with his camel-quickfix project

                                http://code.google.com/p/camel-quickfix/

                                 

                                It wont make it into Camel 2.0.

                                But we have it on the roadmap for 2.1.