3 Replies Latest reply on Feb 21, 2011 7:48 AM by Mircea Markus

    Failure in HotRod server under load

    stevenll Newbie

      I'm trying to stress test a cluster of HotRod servers, and I find that under a load the servers are failing requests from the clients with a TimeoutException:

       

      2011-02-16 13:54:21,927 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (OOB-1,foo,node526-61879) Execution error:

      org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [10 seconds] on key [ByteArrayKey{data=ByteArray{size=7, hashCode=6bf7dab1, array=[2, 62, 4, 52, 55, 51, 54, ..]}}] for requestor [Thread[OOB-1,foo,node526-61879,5,Thread Pools]]! Lock held by [(another thread)]

       

      (see the full exception stack trace in the attached infinispan.log)

       

      My test clients contain a loop of put/get operations that looks like this:

       

            for (int c = 0 ; c < numOps ; ++c)

            {

              int key = random.nextInt(numKeys) ;

              String keyStr = Integer.toString(key) ;

              String value = cache.get(keyStr) ;

              if (value == null)

              {

                cache.put(keyStr, data) ;

              }

            }

       

      To paraphrase this: "For numOps, select a random key out of numKeys and attempt to get it's value from the cache. If it's not there, put a value for this key."

       

      I have two servers  configured with the default cache in distribution mode (see the attached infinispan.xml and my-jgroups.xml) In my test, I run just two clients, each with numOps = 1000, numKeys = 10,000. The size of the cached data string is always 1 byte.

       

      Any suggestions to resolve this issue would be appreciated.

       

      Message was edited by: Steven Levitt Followup: I recreated the problem while running all servers and clients locally on my development machine. So, it would appear that the problem lies either in my configuration or in the Infinispan code.

        • 1. Re: Failure in HotRod server under load
          Galder Zamarreño Master

          Amongst other things, you should try suggestions in http://community.jboss.org/docs/DOC-14881, such as, try disabling lock stripping and increasing lock timeout. You should also try to adjust the concurrency level to avoid collisions. How many concurrent threads are interacting against the server? Default is 32, you should probably increase it, since you probably have more than 32 threads concurrently hitting the cache.

           

          i.e.

           

          <default>
            <locking
               lockAcquisitionTimeout="20000"
               useLockStriping="false"
               concurrencyLevel="1000"
             />
          ...
          
          1 of 1 people found this helpful
          • 2. Re: Failure in HotRod server under load
            stevenll Newbie

            Thank you. I will try your suggestions, but I'm still a little puzzled. My test doesn't attempt explicitly to lock any keys, so I'm not sure why the 'locking' element is necessary, unless the HotRod server is locking keys under-the-hood. Also, I have only two HotRod clients running a single thread each, so I'm not sure how the concurrency level can be an issue.

             

            The exception specifies a timeout of 10 seconds, but my test runs less than 1 second before the exception occurs. How can this really be a timeout?

             

            Message was edited by: Steven Levitt Correction: after double-checking my test logs, I see that the failing request really is taking 10 seconds to return, so it seems that the timeout error is genuine. My apologies. However, my other questions still apply.

            • 3. Re: Failure in HotRod server under load
              Mircea Markus Master
              Thank you. I will try your suggestions, but I'm still a little puzzled. My test doesn't attempt explicitly to lock any keys, so I'm not sure why the 'locking' element is necessary, unless the HotRod server is locking keys under-the-hood

              Infinispan does lock keys for you in order to assure operation atomicity.

              Also, I have only two HotRod clients running a single thread each, so I'm not sure how the concurrency level can be an issue.    

              It still can be an issue if you have lock striping enabled (by default it is). If the threads operate on keys that hash to the same lock, then one thread would wait for the other.


              1 of 1 people found this helpful