13 Replies Latest reply on Dec 4, 2013 6:09 AM by nadirx

    Remote Store of Infinispan

    ranga033

      I am just trying to understand the Remote Store concept and trying the following Use Case :-

      1.) The embedded Infinispan of AS7 (say cache container name - cluster) has a remote store configured..  Can this remote store be a Remote Server having clusters of Infinispan nodes (An Independent Cache Server )?

       

      Another question I have on this topic is :-

      2.) If case 1 is possible ( or atleast with a Remote Store as JDBC) , suppose 2 instances of AS7 are using this configuration and instance 1 goes down or has updated its info, will only Invalidate cache make instance 2 to discard its cache (in memory) and fetch it from Remote store ?

       

      Thanks !

        • 1. Re: Remote Store of Infinispan
          nadirx

          1.) The embedded Infinispan of AS7 (say cache container name - cluster) has a remote store configured..  Can this remote store be a Remote Server having clusters of Infinispan nodes (An Independent Cache Server )?

          The remote store must be a (cluster of) Infinispan HotRod server(s).

          2.) If case 1 is possible ( or atleast with a Remote Store as JDBC) , suppose 2 instances of AS7 are using this configuration and instance 1 goes down or has updated its info, will only Invalidate cache make instance 2 to discard its cache (in memory) and fetch it from Remote store ?

          Could you rephrase the question please ?

           

          Tristan

          • 2. Re: Remote Store of Infinispan
            ranga033

            Thanks Tristan!

             

            Regarding 1.) You mentioned that Remote store must be a Infinispan HotRod server(s). So when the AS7 servers come up (each server is part of a cluster) , how do I ensure they connect to this remote store, by default I see that in domain.xml it points to the local cache containers, there is not configuration to connect to this Remote store ? Is it possible?

             

            Rephrasing 2

            2.) Assuming I have 3 nodes in a AS7 cluster. If all 3 nodes always connect to the Remote store (which is a cluster or Infinispan HotRod Servers) with no local cache , then fine. The cluster cache is available in a remote store and each of nodes can access it. If this is supported, I get the answer what I am looking for.

             

            In case the above does not work, then I have local cache in each of the 3 AS7 server instances. The local cache has remote store configured(say JDBC or Infinispan HotRod servers). When AS7 server instance1 updates info in its local cache and subsequently stores this in the Remote Store, how do the other AS7 server instances update their local cache. Is this achieved only when the local cache is configured as an Invalidate cache.

             

            Hope it is clearer now.

             

            Thank you !

            • 3. Re: Remote Store of Infinispan
              nadirx

              1.) You mentioned that Remote store must be a Infinispan HotRod server(s). So when the AS7 servers come up (each server is part of a cluster) , how do I ensure they connect to this remote store, by default I see that in domain.xml it points to the local cache containers, there is not configuration to connect to this Remote store ? Is it possible?

               

              This configuration snippet shows how to connect to a remote HotRod store:

               

              <distributed-cache name="default" mode="SYNC" owners="2" remote-timeout="30000" start="EAGER">

                  <remote-store cache="default" socket-timeout="60000" tcp-no-delay="true" fetch-state="false" preload="true" purge="false" shared="true">

                      <remote-server outbound-socket-binding="remote-store-hotrod-server"/>

                  </remote-store>

              </distributed-cache>

               

              You also need to add an outbound socket binding to the <socket-binding-group> element:

               

              <outbound-socket-binding name="remote-store-hotrod-server">

                  <remote-destination host="192.168.1.10" port="11222"/>

              </outbound-socket-binding>

               

              2.) Assuming I have 3 nodes in a AS7 cluster. If all 3 nodes always connect to the Remote store (which is a cluster or Infinispan HotRod Servers) with no local cache , then fine. The cluster cache is available in a remote store and each of nodes can access it. If this is supported, I get the answer what I am looking for.

              If you configure everything as in the example above you should have what you need:

              1. a backend cluster of hotrod servers using whatever type of clustering (repl or dist)
              2. a frontend cluster of AS7 instances with a distributed cache backed by the HotRod cluster (note the shared="true" option on the <remote-server /> element)
              3. if one of your frontend nodes updates an item it is propagated to the other frontend nodes via distribution and also sent to the HotRod cluster (for "persistence")
              4. if one of your frontend nodes dies, you get HA

               

              I hope this answers your question.

               

              Tristan

               

              Tristan

              • 4. Re: Remote Store of Infinispan
                ranga033

                Thanks Tristan for the detailed reply, I will test this config and get back !

                • 5. Re: Remote Store of Infinispan
                  ranga033

                  Tristan, sorry, I got held up on something else. I just checked this today. Does this configuration need adding of any hotrod client jars in AS7 ? (if so where do I need to add them)

                   

                  Getting few errors :-

                   

                  [Server:server-one] 17:00:00,889 ERROR [org.jboss.msc.service.fail] (pool-13-thread-1) MSC00001: Failed to start service jboss.infinispan.web.repl: org.jboss.msc.servic

                  e.StartException in service jboss.infinispan.web.repl: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.loaders.CacheLoaderManagerImpl.

                  start() on object

                   

                  [Server:server-one]     at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:62) [jboss-as-clustering-common-7.1.1.Final.jar:7.1.1.Final]

                   

                  [Server:server-one] Caused by: org.infinispan.CacheConfigurationException: Unable to instantiate class org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFacto

                  ry

                   

                   

                  [Server:server-one] Caused by: java.lang.ClassNotFoundException: org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory

                  • 6. Re: Remote Store of Infinispan
                    ranga033

                    I could resolve this problem by adding the following :-

                            <module name="org.infinispan.client.hotrod"/>

                     

                    in %JBOSS_HOME%/modules/org/infinispan/main/module.xml

                     

                    So this is a issue with JBoss (AS7.1.1.FINAL) /Infinispan , since it did not work out of the box ?

                    • 7. Re: Remote Store of Infinispan
                      ranga033

                      I have few questions on this ..

                       

                      Whats the purpose of remote store  when the inbuilt caches (within AS7) are already in cluster ?

                       

                      Can I use a remote-store with two local caches (which are inbuilt in AS7). Will the local caches always lookup to the Remote store?

                      • 8. Re: Remote Store of Infinispan
                        ranga033

                        Tristan, this is the scenario I am trying :-

                         

                        1.) Have a JBoss cluster with 2 nodes (each using local web cache ) connected to a remote-store.

                        2.)Deployed a web application (replication-test war app)

                        3.) Request 1 was processed by Node1. ( Node 1 has a local web cache and a remote store)

                        4.) I brough down Node 1 and Request 2 is processed by Node 2 now. I was expecting this to be in a same session (My assumption was the local cache in Node 2 would get the session details from Remote-store.) But this was not happening. The request is treated as a new session.


                        Hence my question if local caches with a common remote store would help in failover as mentioned above ? If yes could you give me some pointers on how to debug this issue?

                        • 9. Re: Remote Store of Infinispan
                          nadirx

                          Ah, no. You need to use replicated and not local.

                          • 10. Re: Remote Store of Infinispan
                            sonalm13

                            Hi Tristan,

                             

                            I tried the same configuration that you have mentioned. But when trying to start the server with configuration i am getting following error:

                             

                            ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC00001: Failed to start service jboss.infinispan.clustered.test-disc-cache: org.jboss.msc.service.StartException in service jboss.infinispan.clustered.test-disc-cache: Failed to start service

                              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1767) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

                              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_40]

                              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_40]

                              at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_40]

                            Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object of type PersistenceManagerImpl

                              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:185)

                              at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)

                              at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)

                              at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)

                              at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)

                              at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)

                              at org.infinispan.CacheImpl.start(CacheImpl.java:666)

                              at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:551)

                              at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:514)

                              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:396)

                              at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:410)

                              at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:89)

                              at org.jboss.as.clustering.infinispan.DefaultEmbeddedCacheManager.getCache(DefaultEmbeddedCacheManager.java:80)

                              at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)

                              at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

                              at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA.jar:1.0.4.GA]

                              ... 3 more

                            Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders

                              at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:155)

                              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_40]

                              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_40]

                              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_40]

                              at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_40]

                              at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:183)

                              ... 18 more

                            Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport

                              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:291)

                              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:165)

                              at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:25)

                              at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:45)

                              at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:433)

                              at org.infinispan.client.hotrod.RemoteCacheManager.ping(RemoteCacheManager.java:632)

                              at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:613)

                              at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:524)

                              at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:520)

                              at org.infinispan.persistence.remote.RemoteStore.start(RemoteStore.java:89)

                              at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:122)

                              ... 23 more

                            Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /w.x.y.z:11222

                              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:74)

                              at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:35)

                              at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)

                              at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)

                              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:287)

                              ... 33 more

                            Caused by: java.net.ConnectException: Connection refused

                              at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [rt.jar:1.7.0_40]

                              at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735) [rt.jar:1.7.0_40]

                              at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:117) [rt.jar:1.7.0_40]

                              at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:65)

                              ... 37 more

                             

                            Can you suggest what could be the problem here?

                             

                            Thanks,

                            Sonal

                            • 11. Re: Remote Store of Infinispan
                              nadirx

                              Well, looking at that log,

                               

                              is a hotrod server actually listening on w.x.y.z:11222 ?


                              Tristan

                              • 12. Re: Remote Store of Infinispan
                                sonalm13

                                Ya, i got this in the log before getting the above mentioned error:

                                 

                                [org.infinispan.server.endpoint] (MSC service thread 1-15) JDGS010001: HotRodServer listening on w.x.y.z:11222

                                 

                                Sonal

                                • 13. Re: Remote Store of Infinispan
                                  nadirx

                                  Sonal can you post the configuration of both servers on pastebin or similar please ?