7 Replies Latest reply on Jul 7, 2011 3:12 PM by sannegrinovero

    What is the best way to create a JMS-like solution on top of Infinispan?

    susanin

      Hi,

       

      I'm new to Infinispan and try to use Infinispan for distributed caching in our Java-based application. In addition to that, I was thinking to build a small messaging queue solution on top of it. My hope is that it would be much more efficient that usage of a standard RMI approach or JMS, as these two add quite some overhead based on our profiling. And we need a very fast (async) communication between our nodes/sub-systems.

       

      The question is: What is the best to implement such a queue in terms of performance (latency, throughput)?

       

      We had a previous solution that used Hazelcast and it was very easy to do such a queue there, because Hazelcast's API supports out-of-the-box distributed queues as well as distributed topics and subscriptions. But  Infinispan does not provide any distributed data-structures other than distributed maps. Everything else should be modeled on top of it.

       

      Questions:

      1) Are there any plans to add support for any further distributed data-structures, like queues, topics, lists, etc? It would be very valueable for many practical use-cases. And would make Infinispan stronger against competition, e.g. Teracotta or Hazelcast.

       

      2) I tried to implement a queue by means of a distributed map. A producer would add an entry to the distributed cache (eventually with autoexpiration after a certain period of inactivity), and a consumer would be subscribed to the entry modification events. As a result, every time a new item is added to the map, the consumer would be notified and can process it. I use something like this as a configuration:

       

      Configuration c = new Configuration();

      c.setCacheMode(Configuration.CacheMode.DIST_ASYNC);

      // Do not try to group updates. Publish changes in the cluster as soon as possible

      c.setUseLockStriping(false);

      c.setUseAsyncMarshalling(true);

      c.setUseReplQueue(false);

      c.setEvictionStrategy(EvictionStrategy.FIFO);

      c.setEvictionWakeUpInterval(5000);

       

      I set these values in the configuration after reading some Infinispan guides and doing some experimentation. Overall, it seems to work. But  performance of this implementation is not very good, e.g. compared to Hazelcast topics-based approach. It is about 3-4 times slower for some reason. I guess I'm doing something wrong either in configuration or in the overall implementation of the queuing. I'd be very interested if someone could tell me what's wrong or propose a better way to implement it.

       

      Thanks,

        Ivan

        • 1. Re: What is the best way to create a JMS-like solution on top of Infinispan?
          pmuir

          susanin wrote:

           

          Questions:

          1) Are there any plans to add support for any further distributed data-structures, like queues, topics, lists, etc? It would be very valueable for many practical use-cases. And would make Infinispan stronger against competition, e.g. Teracotta or Hazelcast.

          I don' t think that such things are on the roadmap, however I would encourage you to open an feature request in JIRA.

          • 2. Re: What is the best way to create a JMS-like solution on top of Infinispan?
            sannegrinovero

            1) Are there any plans to add support for any further distributed data-structures, like queues, topics, lists, etc? It would be very valueable for many practical use-cases. And would make Infinispan stronger against competition, e.g. Teracotta or Hazelcast.

            Initially we wanted to "be different" and focus on the cache/grid, but for some of our own needs we're going to need efficient atomic integers so we're going to create at least that, and maybe add more collections like lists, but there's no timeframe. Atomic integers might come in 5.1, 5.2/6 are more likely.

            Contributions are very welcome if you want to help on what you need, as Pete said at least ask for it on JIRA.

             

             

            2) I tried to implement a queue by means of a distributed map. A producer would add an entry to the distributed cache (eventually with autoexpiration after a certain period of inactivity), and a consumer would be subscribed to the entry modification events. As a result, every time a new item is added to the map, the consumer would be notified and can process it. I use something like this as a configuration:

            Implementations such as HornetQ are extremely fast and have specialized data structures to handle this optimally. I'd suggest you to add eventlisteners to the cache and when the event is triggered to send a message via a standard JMS implementation.

             

            Besides, your implementation might have an issue, depending on your cluster topology and needs: remember that not all nodes are going to receive the events, only the ones on which the event originated (the one on which you did the "put" operation, for example), and the nodes which are owning the key. Nodes which are not key owners and not writing directly will *not* be notified by the event.

            • 3. Re: What is the best way to create a JMS-like solution on top of Infinispan?
              susanin

              I agree that JMS could be use to send such notifications. Especially, if you want to have all the cool features of JMS like durability, deliver-only-once, etc.

               

              But as indicated in my original post, JMS-based solutions introduce quite some overhead in our scenarios. Moreover, using JMS/HornetQ in addition to Infinispan means one more additional subsystem that requires installation, configuration and maintenance. I don't see a clear benefit of doing that, if a distributed system like Infinispan could provide a similar functionality for simple use-cases, where we just need a very quick communication between nodes.

               

              BTW, I guess that Infinispan (or any other distributed cache or data grid) has a more efficient mechanism for inter-node communication rather than JMS. Otherwise, they would use JMS as a transport, wouldn't they? ;-)

              • 4. Re: What is the best way to create a JMS-like solution on top of Infinispan?
                sannegrinovero

                If you don't need durability nor transactions you can use Infinispan's own transport to send messages using custom commands; see also http://community.jboss.org/thread/168928?tstart=0

                 

                In doing so you would use Infinispan's own transport, i.e. JGroups, which is also able to "deliver-only-once", etc.. Still both JGroups and any JMS implementation would use the network, so it's only about the datastructures being used and as you say the guarantees you might need.

                • 5. Re: What is the best way to create a JMS-like solution on top of Infinispan?
                  susanin

                  Thanks for the hint regarding Infinispan's custom commands! I'll have a look at them.

                  • 6. Re: What is the best way to create a JMS-like solution on top of Infinispan?
                    ckulenk

                    Hi,

                    You could give the Delta/DeltaAware Interfaces a try - this should be faster than using the Map. A DeltaAware Object which delegates to a ConcurrentLinkedQueue works as queue. A generic object that implements Delta and adds its containing object to the ConcurrentLinkedQueue, when merge is called, acts as message container. A special ConsistentHash implementation that manages queue subscriptions with special keys and normally delegates to the DefaultConsistentHash could enable full control over message routing.

                    I don't know if this really works, I only implemented the first part (Delta/DeltaAware Queue) as part of a test for a chat system.

                    Regards,

                    Christian

                    • 7. Re: What is the best way to create a JMS-like solution on top of Infinispan?
                      sannegrinovero

                      Christian wrote:

                       

                      Hi,

                      You could give the Delta/DeltaAware Interfaces a try - this should be faster than using the Map. A DeltaAware Object which delegates to a ConcurrentLinkedQueue works as queue. A generic object that implements Delta and adds its containing object to the ConcurrentLinkedQueue, when merge is called, acts as message container. A special ConsistentHash implementation that manages queue subscriptions with special keys and normally delegates to the DefaultConsistentHash could enable full control over message routing.

                      I don't know if this really works, I only implemented the first part (Delta/DeltaAware Queue) as part of a test for a chat system.

                      Regards,

                      Christian

                      Christian, interesting ideas! If you're interested in working on some proper patches with testcases we might add such helpers. I'm not sure if Manik really wants queues in Infinispan core, but you could propose it to the developer mailing list.