5 Replies Latest reply on Nov 28, 2012 11:26 AM by dan.berindei

    Two servernodes,hotrod and a big database

      Hello,

      I try to use infinispan for a distributed data storage management.

      Here's the situation:

      We have two server machines delivering a big big database.

      My task is to "publish"  that data from both servers as a cloud service, so that a client can ask for a key and gets its data from one of that nodes.

      The data in the database is already distributed, meaning that the db contains different key/value pairs.

       

      It's my first time with infinispan.

      I have a client that can connect to two servernodes by hotrod protocol.

      Because the data is too big to fit into memory, I played with the CacheLoaders (File & BDB) but realized that it just takes too much time to read all data from db into memory (and its a waste of hdd space).

      So I decided to make an own CacheLoader, so that I can lazy load the data (when the load(Object key) method is called)

      This worked fine with one server. But when I tried to work with two servers, I realized that the client doesn't ask the second servernode, if he gets no data from the first (return null in load method).

      I tried the Interceptor pattern with the same result (Tried with visitGetKeyValueCommand()).

       

      Now please help me: I need a mechanism to make the client ask the second node as well so get valid data.

       

      Thanks in advance

        • 1. Re: Two servernodes,hotrod and a big database

          Here's my configuration

           

          <?xml version="1.0" encoding="UTF-8"?>

          <infinispan

                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

                xmlns="urn:infinispan:config:5.1">

            

          <global>

            <transport clusterName="myCloud" distributedSyncTimeout="50000"  transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport">

              <properties>

                <property name="configurationFile" value="jgroups-tcp.xml"/>

              </properties>

            </transport>

          </global>

          <namedCache name="test">

            <clustering mode="distribution">

            </clustering>

            <eviction

              maxEntries="10000"

              strategy="LRU"/>

            <expiration

              wakeUpInterval="500"

              lifespan="60000"

              maxIdle="10000"/>

            <loaders

              preload="false"

              passivation="false">

              <loader

                class="MyCacheLoader"

                ignoreModifications="true"

                fetchPersistentState="false"

                purgeOnStartup="false">

              </loader>   

            </loaders>

          <!--

            <customInterceptors>

              <interceptor position="FIRST" class="MyInterceptor">

              </interceptor>

            </customInterceptors>

          -->

          </namedCache>

          • 2. Re: Two servernodes,hotrod and a big database

            I created a really small project so that anybody can try this on his own.

            When you build the project (mvn clean package) you'll get a zip containing directories for one client and two servers.

            In each server there's a storage.properties file containting a tiny amount of different key/value pairs

             

            Start two servers in different machines (VirtualMachines maybe):

            java -jar infinispan_test-0.0.0.1-SNAPSHOT.jar server <ip_address>

             

            Then start the client

            java -jar infinispan_test-0.0.0.1-SNAPSHOT.jar client <server_ip_address_1> <server_ip_address_2>

             

            The client retrieves a Remotecache and queries 3 keys (from different storage)

            It looks like an error to me, that the client doesn't ask the second node, if the return value is null.

            Or do I have to return a special value in my ChacheLoader?

            Maybe its just a misconfiguration?

             

            Please help me with this.

            • 3. Re: Two servernodes,hotrod and a big database
              dan.berindei

              Hi Astaldo

               

              The HotRod client assumes that it receives the correct data from the server, and null is considered a valid value. It's the server's job to get the data from whatever source it has to.

               

              You can configure a ClusteredCacheLoader on the server, which will force a remote lookup on all the other nodes before returning a null to the client. However, since the distribution of entries in Infinispan won't match the distribution of entries in your database, I'd recommend switching from DIST_SYNC to INVALIDATION_SYNC.

              • 4. Re: Two servernodes,hotrod and a big database

                Thanks for your hint, dan.

                 

                I configured the ClusterCacheLoader in my xml and added a CustomInterceptor, to lazyload the data from the db.

                It works now, although I had problem with the return value type.

                 

                Situation:

                Client connects to Node1 and retrieves multiple keys

                 

                Case 1:

                Key is found on node1. Return value has to be of type CacheValue.

                If not I get a ClassCastException

                 

                2012-11-26 16:34:49,398 DEBUG (New I/O  worker #1) [org.infinispan.server.hotrod.HotRodDecoder] Exception caught org.infinispan.server.hotrod.HotRodException: java.lang.ClassCastException: org.infinispan.container.entries.MortalCacheEntry cannot be cast to org.infinispan.server.core.CacheValue

                      at org.infinispan.server.hotrod.HotRodDecoder.createServerException(HotRodDecoder.scala:214)

                      at org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:77)

                 

                Case 2

                Key is not found on node1. The interceptor of node2 is triggered now. The return value has to be CacheEntry now.

                Otherwise: ClassCastException

                 

                2012-11-26 15:54:18,857 ERROR [RpcManagerImpl] (New I/O  worker #1) ISPN000073: Unexpected error while replicating java.lang.ClassCastException: org.infinispan.server.core.CacheValue cannot be cast to  org.infinispan.container.entries.CacheEntry                                                    

                     at org.infinispan.commands.remote.ClusteredGetCommand.perform(ClusteredGetCommand.java:131)        

                     at org.infinispan.commands.remote.ClusteredGetCommand.perform(ClusteredGetCommand.java:59)       

                      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:101)

                 

                I dont know why there's a difference in the return value type.

                My workaround is to check ctx.getOrigin().

                Seems that it's  null if it's local.

                 

                But that looks like a dirty workaround. Any other hints?

                 

                Thanks, Astaldo

                • 5. Re: Two servernodes,hotrod and a big database
                  dan.berindei

                  Astaldo, it seems that we have two get commands: GetKeyValueCommand and GetCacheEntryCommand. As implied by their names, one of them returns a CacheValue and the other returns a CacheEntry.

                   

                  GetCacheEntryCommand seems to be called only from ClusteredGetCommand, so your ctx.getOrigin() check will work, but it's better to check the command type.

                  1 of 1 people found this helpful