5 Replies Latest reply on Nov 28, 2012 11:26 AM by dan.berindei

    Two servernodes,hotrod and a big database


      I try to use infinispan for a distributed data storage management.

      Here's the situation:

      We have two server machines delivering a big big database.

      My task is to "publish"  that data from both servers as a cloud service, so that a client can ask for a key and gets its data from one of that nodes.

      The data in the database is already distributed, meaning that the db contains different key/value pairs.


      It's my first time with infinispan.

      I have a client that can connect to two servernodes by hotrod protocol.

      Because the data is too big to fit into memory, I played with the CacheLoaders (File & BDB) but realized that it just takes too much time to read all data from db into memory (and its a waste of hdd space).

      So I decided to make an own CacheLoader, so that I can lazy load the data (when the load(Object key) method is called)

      This worked fine with one server. But when I tried to work with two servers, I realized that the client doesn't ask the second servernode, if he gets no data from the first (return null in load method).

      I tried the Interceptor pattern with the same result (Tried with visitGetKeyValueCommand()).


      Now please help me: I need a mechanism to make the client ask the second node as well so get valid data.


      Thanks in advance

        • 1. Re: Two servernodes,hotrod and a big database

          Here's my configuration


          <?xml version="1.0" encoding="UTF-8"?>



                xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"




            <transport clusterName="myCloud" distributedSyncTimeout="50000"  transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport">


                <property name="configurationFile" value="jgroups-tcp.xml"/>




          <namedCache name="test">

            <clustering mode="distribution">





















              <interceptor position="FIRST" class="MyInterceptor">





          • 2. Re: Two servernodes,hotrod and a big database

            I created a really small project so that anybody can try this on his own.

            When you build the project (mvn clean package) you'll get a zip containing directories for one client and two servers.

            In each server there's a storage.properties file containting a tiny amount of different key/value pairs


            Start two servers in different machines (VirtualMachines maybe):

            java -jar infinispan_test- server <ip_address>


            Then start the client

            java -jar infinispan_test- client <server_ip_address_1> <server_ip_address_2>


            The client retrieves a Remotecache and queries 3 keys (from different storage)

            It looks like an error to me, that the client doesn't ask the second node, if the return value is null.

            Or do I have to return a special value in my ChacheLoader?

            Maybe its just a misconfiguration?


            Please help me with this.

            • 3. Re: Two servernodes,hotrod and a big database

              Hi Astaldo


              The HotRod client assumes that it receives the correct data from the server, and null is considered a valid value. It's the server's job to get the data from whatever source it has to.


              You can configure a ClusteredCacheLoader on the server, which will force a remote lookup on all the other nodes before returning a null to the client. However, since the distribution of entries in Infinispan won't match the distribution of entries in your database, I'd recommend switching from DIST_SYNC to INVALIDATION_SYNC.

              • 4. Re: Two servernodes,hotrod and a big database

                Thanks for your hint, dan.


                I configured the ClusterCacheLoader in my xml and added a CustomInterceptor, to lazyload the data from the db.

                It works now, although I had problem with the return value type.



                Client connects to Node1 and retrieves multiple keys


                Case 1:

                Key is found on node1. Return value has to be of type CacheValue.

                If not I get a ClassCastException


                2012-11-26 16:34:49,398 DEBUG (New I/O  worker #1) [org.infinispan.server.hotrod.HotRodDecoder] Exception caught org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: org.infinispan.container.entries.MortalCacheEntry cannot be cast to org.infinispan.server.core.CacheValue

                      at org.infinispan.server.hotrod.HotRodDecoder.createServerException(HotRodDecoder.scala:214)

                      at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:77)


                Case 2

                Key is not found on node1. The interceptor of node2 is triggered now. The return value has to be CacheEntry now.

                Otherwise: ClassCastException


                2012-11-26 15:54:18,857 ERROR [RpcManagerImpl] (New I/O  worker #1) ISPN000073: Unexpected error while replicating java.lang.ClassCastException: org.infinispan.server.core.CacheValue cannot be cast to  org.infinispan.container.entries.CacheEntry                                                    

                     at org.infinispan.commands.remote.ClusteredGetCommand.perform(ClusteredGetCommand.java:131)        

                     at org.infinispan.commands.remote.ClusteredGetCommand.perform(ClusteredGetCommand.java:59)       

                      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)


                I dont know why there's a difference in the return value type.

                My workaround is to check ctx.getOrigin().

                Seems that it's  null if it's local.


                But that looks like a dirty workaround. Any other hints?


                Thanks, Astaldo

                • 5. Re: Two servernodes,hotrod and a big database

                  Astaldo, it seems that we have two get commands: GetKeyValueCommand and GetCacheEntryCommand. As implied by their names, one of them returns a CacheValue and the other returns a CacheEntry.


                  GetCacheEntryCommand seems to be called only from ClusteredGetCommand, so your ctx.getOrigin() check will work, but it's better to check the command type.

                  1 of 1 people found this helpful