Can multiple Infinispan servers manage the same cached data?

[mmr11408]

We have internal apps that make millions of REST requests daily to our app-server to retrieve some data. We want the app-server to be very fast and always available. The full data is around 1 GB, stored in an Oracle DB and does not change often. The app-server never changes the data. We will have several instances of the app-server running on separate hosts.

I am thinking of using Infinispan in a replication clustering mode to keep all (or part of) the data in the cache for each app-server, and create a cache-server that reads the data from the DB into the cache (which causes the app-servers’ cache to be updated by Infinispan). The cache-server is notified when the data in the DB is changed, at which time it refreshes itself from the DB and causes the app-servers’ cache to get updated too.

Can multiple cache-servers be brought up behind a load balancer to manage the same data? This would mean that to the app-servers there is a single IP/host that identifies the cache-server (jdg.host=load-balanced-address-of-cache-server, jdg.hotrod.port=cache-server-port); and the cache-servers will operate independently of each other but have the same configuration (e.g. same hotrod port and same named cache).

[dinoop.p1]

Yes Mehdi, Infinispan is intended for this purpose.

[mmr11408]

Thanks. Is there anything that I have to worry about, for example when multiple cache servers broadcast updates to the same data which is held in the same named cache but by independent cache servers?

[dinoop.p1]

Are you bundling Infinispan in your app-server? If yes, you don't want to use a load balancer between app-server and cache server. Use same cluster name for all your cache-server & app-server. so that changes in one cache-server will be automatically updated in rest of the nodes.

Also you don't want load same data multiple times using different cache-servers when any changes in db occurs. Because Infinispan will update rest of the nodes automatically if any changes occurs in a distributed cache.

There is no problem if you update the data simultaneously from different cache-servers. All write operations of a particular key are atomic. 

[mmr11408]

Thank you. Yes, I plan on bundling Infinispan with the app-server.

“Use same cluster name for all your cache-server & app-server. so that changes in one cache-server will be automatically updated in rest of the nodes.” If one cache-server updates its cache, does the second cache-server’s cache get automatically updated by Infinispan or only the app-servers (cluster members) get updated?

Regarding “you don't want to use a load balancer between app-server and cache server” Do you mean that instead of using a load balancer I should use the ConfigurationBuilder.addServers() API to add multiple cache-server definitions to each app-server?

I tried that with the sample remote-query program and got unexpected results.

Single cache-server steps:

• Copied standalone.xml into standaloneRemoteQuery.xml
• Added address book definitions to standaloneRemoteQuery.xml
• Started standalone.bat using standaloneRemoteQuery.xml and remote-query example, all seems to work fine. I was able to add entries to the address book and display them.

Multiple cache-server steps:

• Made a full copy of the cache-server directory (infinispan-server-7.1.0.Final to infinispan-server-copy)
• Changed the ports in standaloneRemoteQuery.xml of the copied cache-server

  •     <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">

  •         <socket-binding name="management-http" interface="management" port="${jboss.management.http.port:19990}"/>
  •         <socket-binding name="management-https" interface="management" port="${jboss.management.https.port:19993}"/>
  •         <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
  •         <socket-binding name="hotrod" port="21222"/>
  •         <socket-binding name="http" port="${jboss.http.port:18080}"/>
  •         <socket-binding name="https" port="${jboss.https.port:8443}"/>
  •         <socket-binding name="memcached" port="21211"/>
  •         <socket-binding name="txn-recovery-environment" port="4712"/>
  •         <socket-binding name="txn-status-manager" port="4713"/>
  •         <socket-binding name="websocket" port="18181"/>
  •     </socket-binding-group>
• Changed the remote-query sample to use multiple servers:
  1. builder.addServers( "localhost:11222;localhost:21222" ).marshaller(new ProtoStreamMarshaller());

Test scenario #1: One cache-server at a time

• Started one cache-server
• Started the remote-query example
• Everything worked as expected. The same test was performed with each cache-server separately

Test scenario #2: Two cache-servers running simultaneously

• Cache-server 11222 was started first, then cache-server 21222
• Started the remote-query example and added 1 entry. Everything worked.
• Added a second entry and remote-query example and 11222 cache-server got exceptions:

  • Remote-query:

    • > 1

    • Enter person id (int): 1
    • Enter person name (string): 1
    • Enter person email (string): 1
    • > 1
    • Enter person id (int): 2
    • Enter person name (string): 2
    • Enter person email (string): 2
    • Feb 16, 2015 8:17:25 AM org.infinispan.client.hotrod.impl.protocol.Codec20 checkForErrorsInResponseStatus
    • WARN: ISPN004005: Error received from the server: org.hibernate.search.bridge.BridgeException: Exception while calling bridge#set
    •         class: org.infinispan.query.remote.indexing.ProtobufValueWrapper
    •         path:
    • org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[8] returned server error (status=0x85): org.hibernate.search.bridge.BridgeException: Exception while calling bridge#set
    •         class: org.infinispan.query.remote.indexing.ProtobufValueWrapper
    •         path:
    •         at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:321)
  • Cache-server 11222:

    • 08:17:25,042 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-7-1) ISPN000136: Execution error: org.hibernate.search.bridge.

    • BridgeException: Exception while calling bridge#set
    •         class: org.infinispan.query.remote.indexing.ProtobufValueWrapper
    •         path:
    •         at org.hibernate.search.bridge.util.impl.ContextualExceptionBridgeHelper.buildBridgeException(ContextualExceptionBridgeHelper.java:84)
    • …
    • Caused by: java.lang.IllegalArgumentException: Message descriptor not found : quickstart.Person
    •         at org.infinispan.protostream.impl.SerializationContextImpl.getMessageDescriptor(SerializationContextImpl.java:172)
• Looking at ClusterCacheStats with jconsole, each cache-server shows numberOfEntries=1

Test scenario #3: Two cache-servers running simultaneously

• Cache-server 21222 was started first, then cache-server 11222
• Started the remote-query example and added 1 entry. Listing entries shows nothing.

  • > 1

  • Enter person id (int): 1
  • Enter person name (string): 1
  • Enter person email (string): 1
  • > 9
  • >
• Added a second entry. No errors but listing entries shows nothing:

  • > 1

  • Enter person id (int): 2
  • Enter person name (string): 2
  • Enter person email (string): 2
  • > 9
  • >
• Looking at ClusterCacheStats with jconsole, the cache-server that was started first (21222) shows numberOfEntries=2, the one that was started second (11222) shows numberOfEntries=0
  

[dinoop.p1]

Hi Mehdi,

If one cache-server updates its cache, does the second cache-server’s cache get automatically updated by Infinispan or only the app-servers (cluster members) get updated?

Only cluster members will be updated.

Do you mean that instead of using a load balancer I should use the ConfigurationBuilder.addServers() API to add multiple cache-server definitions to each app-server?

If you are bundling the Infinispan inside your application you can access the cache as like a local cache. For this the app-server cache name & cluster name should be same as that of cache server. Also the cache should be replicated/distributed.

Regarding the exceptions

I am not much familiar with Hotrod setup, because I have used it by bundling inside application.

[mmr11408]

Thank you.

Note that I am bundling Infinispan inside my application but the app-server and cache-manager will be run in different JVMs (different tomcat instances) on different hosts. This is because I want only the cache-managers to interact with the DB and handle updates to the cache while the app-servers respond to the REST call for the data requests. In this setup the app-server assumes that it has the whole data in the cache and if a request comes in for data that is not in the cache it concludes that the data is not in the DB either.

If when one cache-manager updates its cache only cluster members' cache is updated, then if both cache-managers do not monitor the DB and update their cache accordingly they will get out of sync. And if they both monitor the DB and update their cache then both of their updates will be sent to the cluster members; in this scenario are there any issues (e.g. timing issues, race conditions, network interruptions, ...) that I have to worry about?

[wdfink]

If you try to separate cache and application you should use the Hot Rod client in your application and not a direct cache. The Infinispan can be installed in client-server mode and here you can configure distributed caches for all instances. The cached data will be stored somewhere in one instance, for failover you can set a number of owners which should have the same key.

The HotRod client know which node to contact for a specific key.

If you store a key you don't need to care about where it is stored. If you read a key you can use the advancedCache with flags to prevent from reading the database if the key does not exists.

In this case the cache is distributed and you don't have to worry about the database and whether the different instances are in-sync. The cluster will take care of this in case of failure or topology changes.

[mmr11408]

The distributed clustering mode where different cache entries reside on different cluster members is not beneficial to my application. I want the replicated clustering mode where all the data is resident in every copy of the cache and there is no passivation or expiration.

From Infinispan’s point of view, the app-servers and cache-managers are cluster members sharing the same named cache.

The cache-managers would come up first, load their cache from the DB, monitor the DB for updates and update the cache which would get propagated to the other cluster members by Infinispan.

The app-servers load their cache at startup (by setting fetchInMemoryState="true" in the configuration file) and then access the cache for data query only and never update the cache.

The cache-managers and app-servers will be running in different tomcat instances on different hosts.

The idea is that if the cache-managers are down or cannot be accessed due to network issues, the app-servers continue to function properly, with the only exposure being that the data in the cache might be stale. Similarly, if the connection to the DB is interrupted during normal processing, only cache-managers will be impacted but the app-servers will continue to function properly.

In this model:

- Do I need to use HotRod or using JGroup with Infinispan would suffice?  

- Are there any issues that may prevent this from working properly (e.g. timing)?  

[dinoop.p1]

wdfink Is it possible to update the data from Hot rod server itself?. As per my understanding we can update cache by connecting to hot rod from another client. The client which is connected will update data.

I understand using Hot road server configuration is more standardized way of doing the above use case. But according to Mehdi, he has to segregate the cache loading logic from app server. So he cannot use Hot road server as cache - server.. The cache - server should be an embedded app with Infinispan.

Again Mehdi, your requirement can be satisfied by the above approach.

[wdfink]

From what I understand you change the values in the database with other processes.

I see different options.

- Set eviction to reload the data from the DB sometimes, this will not ensure that the data is up-to-date.

- Use an external process to monitor the database and update the cache, depend on the approach how long it takes for the cache to be updated

If you need to be independent from the database you might create a custom cache store which skip the DB access if not available without an error.

If your cache is down the application will not have access to any cached data if you use separate hot-rod servers, but it is designed as a HA cluster and this should mostly not happen.

If you use library mode the cache is inside of the applications JVM, adding a custom cache store  for the DB access might help for load new values, but not for updating it if you don't set eviction.

I think it will be complicated to keep the cache up-to-date if you access the database from other processes and update the values