Performance of Creating New Sessions

[adam.mccormick]

We are using ModeShape 4.6.1-Final...

Our application is high read, low write. We actually only write to the repository at most once per day (we use the JCR export/import to move content from a staging environment to production). Client's need not be authenticated so all sessions are anonymous.

I recently discovered that our application was not logging out the session after the http request had been completed. I changed this to create a new session for each request and log it out after rendering has completed (I took the same approach as in the modeshape-web-jcr-rest project of using a thread local to create and remove the request bound session).

After doing this we noticed a marked degradation in our performance tests. I think the problem lies in the workspace cache having to read from the persistence for each new session where as before each request may have been reusing a session which already has a lot of the content in the workspace cache.

Is there a way to help with this? Is some ModeShape or Infinispan configuration that can help with high read scenarios? The only thing I can think of upping the minimum binary size to try and avoid going to the binary store so often (i will give this a go tomorrow).  In the mean time... 

Any idea's would be great!

For reference here's my current infinispan.xml:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="urn:infinispan:config:7.2 http://www.infinispan.org/schemas/infinispan-config-7.2.xsd"
  xmlns="urn:infinispan:config:7.2">
  <cache-container default-cache="unused" statistics="false" >
       <jmx duplicate-domains="true"/>
       <!-- avoids duplicate persistent directories because of cache configuration inheritance -->
       <local-cache name="unused" statistics="false" />
       <local-cache name="repository" statistics="false">
            <locking acquire-timeout="2000" />
            <transaction mode="NON_XA" locking="OPTIMISTIC"/>
            <persistence passivation="false">
                 <file-store fetch-state="false" shared="false" preload="true" purge="false" path="${home}/jcr/data"/>
            </persistence>
       </local-cache>
       <!-- Local in-memory caches with no persistence.  Used instead of the default one in modeshape to avoid the LIRS bug in infinispan -->
       <local-cache name="repository/system" statistics="false">
            <!-- No more that 10K entries per cache, with LRU eviction. -->
            <eviction max-entries="10000" strategy="LRU"/>
            <!-- Expire entries after 120 seconds or after 60 seconds of not being used (whichever comes first). -->
            <expiration lifespan="120000" max-idle="6000"/>
       </local-cache>
       <!-- Local in-memory caches with no persistence. Used instead of the default one in modeshape to avoid the LIRS bug in infinispan -->
       <local-cache name="repository/default" statistics="false">
            <!-- No more that 10K entries per cache, with LRU eviction. -->
            <eviction max-entries="10000" strategy="LRU"/>
            <!-- Expire entries after 120 seconds or after 60 seconds of not being used (whichever comes first). -->
            <expiration lifespan="120000" max-idle="6000"/>
       </local-cache>
       <!-- persistent cache used by a connector -->
       <local-cache name="connector" statistics="false">
            <locking acquire-timeout="2000" />
            <transaction mode="NON_XA" locking="OPTIMISTIC"/>
            <persistence passivation="false">
                 <file-store fetch-state="false" shared="false" preload="false" purge="false" path="${home}/jcr/data"/>
            </persistence>
       </local-cache>
  </cache-container>
</infinispan>

And repository configuration:

{
  "name" : "repository",
  "transactionMode":"none",
  "workspaces":
  {
    "default":"default",
    "cacheConfiguration":"infinispan.xml"
  },
  "storage" :
  {
      "cacheConfiguration" : "infinispan.xml",
      "cacheName" : "repository",
      "binaryStorage" :
      {
          "directory":"${home}/jcr/binary",
          "minimumBinarySizeInBytes" : 100, 
          "minimumStringSize" : 999999999999, // avoids a problem we had in the connector with strings coming back as binary types
          "type" : "file"
      }
  },
  "garbageCollection":
  {
    "initialTime":"12:18",
    "intervalInHours":1
  },
  "security":
  {
     "anonymous":
     {
          "roles" : ["readonly","readwrite","admin"],
          "username" : "<anonymous>",
          "useOnFailedLogin" : false
      }
  },
  "journaling":{
    "enabled":false
  }
}

[hchiorean]

First, there's no reason to use expiration on any of the ws caches as that can only increase churn operating alongside eviction.

I think the problem lies in the workspace cache having to read from the persistence for each new session where as before each request may have been reusing a session which already has a lot of the content in the workspace cache.

That should not be the case: a workspace cache is shared by all the sessions for that particular workspace. Changing your usage pattern from one long-lived session to multiple short lived sessions (which is the preferred approach) should not impact in any way the workspace cache. This is mainly cleared on each session.save() but only the nodes (and subnodes) that were affected by the session.save() are evicted

If on the other hand you have only reader sessions (no writes are taking place), the workspace cache is only cleared when the eviction policy kicks in (again, don't use expiration) - this is something you can entirely control via the ISPN configuration and in this case it make no difference if you're using one session or multiple sessions.

To conclude:

1. the ws cache evicts all the nodes involved in a session.save operation upon successful completion of that operation or
2. when the eviction policy kicks in, when it will evict *some* of the nodes based on the eviction policy
3. if your sessions only read and never change data, changing the eviction policy and ws cache size (via a default ws cache configuration file) can help.

I don't believe your change is impacting the ws cache. One thing that may be happening is that with 1 session you were essentially "synchronizing" requests on that session which acted as a semaphore. With multiple writing sessions, especially changes to the same portions of data, the ws cache churn is a lot higher causing more hits to the persistent store. If you never write and only read data, you should not see any performance degradation.

I would recommend you profile the 2 approaches and compare the results, to see exactly where the problems are coming from.

[adam.mccormick]

Thanks Horia,

Just curious though; I had the expiration in the workspace cache configurations because the default-workspace-cache-config.xml has it. I copied that one and just changed the eviction policy to work around the LIRS problem under high load.  I'll remove it if it is not necessary but curious why it's in the default...

To be honest after some investigation today I found that our application is being put under almost 20x the load it was previously after the introduction of new consumer apps so, as you said, having multiple short lived sessions probably has nothing to do with it

Thanks for your help.

[hchiorean]

Just curious though; I had the expiration in the workspace cache configurations because the default-workspace-cache-config.xml has it. I copied that one and just changed the eviction policy to work around the LIRS problem under high load. I'll remove it if it is not necessary but curious why it's in the default...

fair question, I forgot about that.

I actually don't know why we have it there (I suspect it was added originally in the first 3.x releases when we thought it was a good idea and never revisited it). Thinking about it now though, it doesn't make sense for non-clustered uses cases at all since we're always clearing the WS cache on writes, to prevent stale reads from other sessions. On top of that we're using the eviction policy to make sure we don't run OOM.  It may make a little sense for clusters with a high write-rate where clearing the WS cache from a node more often than not can help with the stale reads problem (which can be a real issue in a cluster).

So unless you're clustering with heavy writes, I'd say don't use it (using it won't cause any problems, expect as mentioned previously I can see it increasing the cache churn rate for no good reason)