10 Replies Latest reply on Nov 4, 2010 12:35 PM by galder.zamarreno

    Possible Infinispan Architectures

    shane_dev

      We've been considering a few different options for Infinispan architectures. A little background on where we are now...

       

      We are running 2 cache instances in replicated mode. Each instance resides within our application. However, we do this for failover. So while one instance is active, the other is passive.

       

      That being said, we have a couple goals in mind....

       

      • Remove the Coupling between the Application and the Cache Instance
      • Remove the Coupling between the Cache Instances (e.g. merge / state transfer)
      • Reduce the Latency for Replicated Operations

       

      The idea is that we will add 2 more cache instances. These instances will be independent of our application process. In addition, we put each application / standalone cache pair on the same machine. This should allow us to reduce our latency by replicating to a cache on the same machine.

       

       

      Basic Architecture

      basic_arch.gif

       

      The problem we have with this approach is that the two application processes are still coupled (cache 1 & cache 2) and as a result may affect each other (e.g. merge / state transfer).

       

      The benefit is that the application processes can both fail, but the cache will remain intact via the separate Infinispan instances.

       

      Hot Rod Multi Cluster Architecture

      hr_multi_cluster_arch.gif

       

      The only difference here is that we have substituted a single 4-node cluster for two 2-node clusters using Hot Rod.

       

      We still have the same problem with the two application processes being coupled and we still have the benefit that the cache will remain intact in the event that both application processes fail.

       

      To be honest, I'm not sure what the benefit of using Hot Rod is. We still have the same pros and cons, but now we have a much more complex architecture.

       

      Hot Rod Architecture

      hr_basic_arch.gif

       

      This is our preferred approach in that the application processes are no longer coupled, and the cache will remain intact in the event that both application processes fail.

       

      However, it appears that Hot Rod does not yet support the functionality we need. Right now, it won't take long for the two caches to become inconsistent since Hot Rod does NOT 'push' operations to the remote caches.

       

      It seems that the remote cache is designed to operate as an L1 cache with a very aggressive eviction policy. I assume that ISPN-374 (Async Hot Rod Events) will allow you to use the remote cache as a proper L1 cache without the need to set up an eviction policy. However, we will take a performance hit on failover since the other cache will be empty.

       

      What we would like to do is build on ISPN-374 such that rather than sending only the key with the event, we'll send the value too. This will allows us to propagate values (for inserts / updates) rather than just keys (for removes).

       

      Question: When will ISPN-374 be ready? What are you thoughts on us extending on it so that we can propagate values and not just keys (for invalidation)? The idea is that once ISPN-374 is ready, I will code the extension myself.

       

      Finally, is there anything I've missed here or are there any other suggestions?

        • 1. Re: Possible Infinispan Architectures
          manik

          ISPN-374 won't happen for a while yet, given current priorities, workload, etc.  Pushing both keys and values is certainly a possibility. 

           

          Regarding getting ISPN-374 out sooner, If it is something you are willing to contribute that may help speed things up.

           

          As a workaround in your current architecture, you could have a local listener on HotRodCache2 which, when it receives an update, pushes the update to RemoteCache2.  This would require you to have a receiving endpoint on RemoteCache2 for this purpose, which could be implemented as either (a) another Hot Rod endpoint, solely to receive these pushes or (b) your own socket to receive serialized keys and values, and place them in the cache.

          • 2. Re: Possible Infinispan Architectures
            shane_dev

            Thanks for the clarifications.

             

            Let me see if I'm following you correctly here....

             

            • Put an entry into Remote Cache 1.
            • It is sent to Hot Rod Server 1.
            • It is replicated to Hot Rod Server 2.
            • A listener/interceptor sends it to Remote Cache 2.

             

            I'm wondering if it wouldn't be better to use an interceptor in this case. I'm a little concerned about consistency using a listener.

             

            Regarding the receiver, we'll be using the Java client. Is it possible to set up a separate JGroups channel? I'll have to start taking a look at the Hot Rod client / server code to see where a good place would be to start work on the receiver and a channel to connect to it. If you can point me in the right direction or have any thoughts on how you guys would go about this, it would be greatly appreciated.

             

            It sounds like the key is being able to register the remote caches so that when we receive a replicated command, we knowing where to send it next since we don't want to send it back to the source. Any thoughts on that?

            • 3. Re: Possible Infinispan Architectures
              mircea.markus

              a) won't work straight away because hotrod keeps the entries in its own proprietary format. You would need to unwrap from that format into your native one, by using a custom interceptor on the client.

              1 of 1 people found this helpful
              • 4. Re: Possible Infinispan Architectures
                savin7

                Dear Shane,

                 

                How you are forming the HotRot server cluster.

                 

                Is it two Hot Rod Servers running in same machines as different instances forming a Cluster ?

                or

                Two Hot Rod Servers running in different machines forming a cluster ?

                If Two Hot Rod Servers running in different machines, then
                Can you please let me know the configuration to set up a cluster between two Hot Rod servers (Server1 & Server2) thats running on different machines ?

                because I have a scenario where I have to form a cluster between hot rod servers that are running in different machines. I couldnt able to find any docs or materials to achieve this scenario.

                 

                Please help me out, if you achieved this scenario earlier..

                 

                Thanks & Regards,

                Savin

                • 5. Re: Possible Infinispan Architectures
                  shane_dev

                  Hi Guys,

                   

                  We have decided to take another look at this again using your recommendations. I wanted to verify that we understand the potential solution correctly. We'll be using a custom listener and interceptor in order to synchronize the remote caches. Here is the diagram I have put together along with my understanding of the flow.

                   

                  arch.gif

                  1. Application writes object to application cache on A (Remote Cache Store).
                  2. Cache on A (RCS) forwards (in HR format) object to C (Hot Rod Server).
                  3. C (HR Server) writes object to its cache.
                  4. Cache on C (HR Server) syncs with Cache on D (HR Server) via distributed/replication configuration.
                  5. Listener on Cache D (HR Server) writes object to a separate cache (RCS)?
                  6. Cache (RCS) forwards object to an embedded HR Server on B.
                  7. Interceptor on embedded HR Server on B unwraps object (from HR format).
                  8. Embedded HR Server on B writes object to application cache (RCS) with flag set to not persist.

                   

                  I am wondering if step 5 is accurate. Is this how we are propagating commands from the HR servers back to the remote cache stores on the applications?

                   

                  My interpretation was that since HR client/server communication is basically one-way (cache to server) that we are using pairs of HR clients/servers in order to create two, one-way channels. I realize that ISPN-374 essentially makes this channel two-way. The tricky part for me is how to acquire a connection back to a separate remote cache store. If remember correctly that is what we are using the listener for. Or, was the intentially that we would manually create a connection (socket, etc)?

                   

                  Thanks,

                  Shane

                  • 6. Re: Possible Infinispan Architectures
                    shane_dev

                    I thought I would clarify just a little bit. Obviously we need to progate a commands from our main HR server to the embedded HR server within our application. I had assumed the easiest way to do that was to use an RCS as a HR client. However, perhaps it is possible to use a smaller subset of the HR client API to directly propagate the commands from the Listener to the embedded HR server?

                     

                    Also, I assumed we were talking about placing a listener on the actual cache itself. Does ISPN-509 come in to play at all here?


                    Thanks,

                    Shane

                    • 7. Re: Possible Infinispan Architectures
                      mircea.markus

                      5. -> what is the point of using another cache for writing from C to B? Can't you use HR client directly?

                      The tricky part for me is how to acquire a connection back to a separate remote cache store. If remember correctly that is what we are using the listener for. Or, was the intentially that we would manually create a connection (socket, etc)?

                      Is this needed e.g. for writing from the listener on C to HR server on B? If so, the listener would need to be statically configured: HR server host and port. 

                      • 8. Re: Possible Infinispan Architectures
                        mircea.markus
                        However, perhaps it is possible to use a smaller subset of the HR client API to directly propagate the commands from the Listener to the embedded HR server

                        You can use the java HR client API directly.

                        Also, I assumed we were talking about placing a listener on the actual cache itself. Does ISPN-509 come in to play at all here?

                        Listeners are attached to the cache itself indeed. This functionality is fully implemented and not related to ISPN-509.

                        • 9. Re: Possible Infinispan Architectures
                          shane_dev

                          Mircea,

                           

                          I wanted to update the diagram after your feedback. Here is my updated interpretation.

                           

                          1. Application writes object to cache on A (Remote Cache).
                          2. Cache on A forwards object (via HR protocol) to Hot Rod server on C.
                          3. Hot Rod server on C writes object to the cache.
                          4. Listener on cache uses the Java HR client to forward the object to the embedded HR server on B.
                          5. Interceptor on the embedded HR Server unwraps the object (from HR format) and writes it to the cache.

                           

                          The two HR servers are synchronized via replication (in our case). The black lines are simply the inverse where the application writes to the cache on B instead.

                           

                          arch.gif

                          • 10. Re: Possible Infinispan Architectures
                            galder.zamarreno

                            Wrt https://jira.jboss.org/browse/ISPN-509, you'd need it if Hot Rod server was a separate process. However, you can create your own runnable main that starts Hot Rod and hooks up listeners programmatically to the corresponding CacheManager/Cache. You can find plenty of examples on how to start a Hot Rod server programatically in the Java Hot Rod client code (http://anonsvn.jboss.org/repos/infinispan/trunk/client/hotrod-client/)