1 2 Previous Next 16 Replies Latest reply on Feb 26, 2013 7:16 PM by rhauch

    Webdav internal caches - how to disable them ?

    pjakub

      Hi,

       

      I'm using modeshape in cluster and have problem with webdav synchronization.

      If I request file like 3-4 seconds after it's uploaded into other cluster node, it get served and everything is ok.

      If I request webdav to serve file when it's not in storage yet, then it returns 404 not found as it should. But then after I upload my file it keeps anserwing with 404 not found. it somehow caches that answer.

      If I wait for 2-3 minutes, then it refresh some internal cache and return the file properly.

      I can't find any configuration option to disable that caching.

       

      Can that be somehow configured ? Or can I override that cache to lookup file and don't cache answer if it was not found?

       

       

      my modeshape configruation looks like that:

       

      {

          "name" : "tlk_repo",

          "jndiName" : "",

          "storage" : {

              "cacheName" : "tlk_infinispan_repo",

              "cacheConfiguration" : "ispn-configuration.xml",

          "binaryStorage" : {

               "type" : "cache",

                  "dataCacheName" : "dataCache",

                  "metadataCacheName" : "metadataCache"

          }

          },

       

          "query" : {

              "enabled" : false,

              "enableFullTextSearch" : false

          },

       

          "workspaces" : {

              "predefined" : ["default","global"],

              "default" : "default",

              "allowCreation" : true

           },

          "security" : {

              "anonymous" : {

                  "roles" : ["readonly", "connect","admin","readwrite"],

                  "useOnFailedLogin" : false

              }

          },   

       

          "node-types" : ["tlk-custom-node-types.cnd"],

       

           "clustering" : {

              "clusterName" : "ms-cluster",

          "channelConfiguration" : "jgroups-ms.xml"

          }

      }

       

       

      and important part of ispn configuration:

       

      <namedCache name="tlk_infinispan_repo">

              <clustering mode="distribution">

                  <sync />

                  <hash numOwners="2" rehashWait="120000" rehashRpcTimeout="600000" />

                  <l1 enabled="true" lifespan="600000" />

              </clustering>

              <transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.DummyTransactionManagerLookup" transactionMode="TRANSACTIONAL" lockingMode="OPTIMISTIC" />

          </namedCache>

       

       

          <namedCache name="dataCache">

              <clustering mode="distribution">

                  <sync />

                  <hash numOwners="2" rehashWait="120000" rehashRpcTimeout="600000" />

                  <l1 enabled="true" lifespan="600000" />

              </clustering>

          </namedCache>

       

       

          <namedCache name="metadataCache">

              <clustering mode="distribution">

                  <sync />

                  <hash numOwners="2" rehashWait="120000" rehashRpcTimeout="600000" />

                  <l1 enabled="true" lifespan="600000" />

              </clustering>

          </namedCache>

       

      modifying l1 cache don't bring any change to behaviour

        • 1. Re: Webdav internal caches - how to disable them ?
          vasilievip

          Do you have apache server between client and modeshape webdav servlet? Can you try w/o apache web server, like push file directly to one node and query it from second?

          • 2. Re: Webdav internal caches - how to disable them ?
            pjakub

            Thanks for your answer.

             

            yes, I'm using apache.

            I've got two instances of jboss app server + webdav.

            Jboss instances see changes between each other immediatley. Webdav don't.

            I'm also using index, but webdav doesn't use queries, so index on webdav is disabled.

             

            Is there any way I can tune logging to see what's happening inside modeshape in webdav ? Wether It's anserwing with clena data, or from some cache?

            • 3. Re: Webdav internal caches - how to disable them ?
              vasilievip

              Well, first I would eliminate web server to avoid caching there. Then I would take a look at code on github to see where caching may occur:

              https://github.com/ModeShape/modeshape/blob/master/web/modeshape-web-jcr-webdav/src/main/java/org/modeshape/web/jcr/webdav/ModeShapeWebdavServlet.java

              1 of 1 people found this helpful
              • 4. Re: Webdav internal caches - how to disable them ?
                pjakub

                Ok,

                InfinispanBinaryStore have blobCache, I think it might be the cause, I'll investigate that tomorrow. Thanks for helpful tip!

                • 5. Re: Webdav internal caches - how to disable them ?
                  rhauch

                  InfinispanBinaryStore's blob cache is not a "cache" in the sense you're thinking. It's the ISPN cache that is a store used to persist the binary content; there's also a metadata cache (again, a store) in which all metadata about the binaries is stored. Any BinaryStore implementation you pick will persist the binary content immediately.

                   

                  The WebDAV service caches nothing. I've mounted the WebDAV service to an OS, uploaded files, and they become immediately visible to another mount as well as through the RESTful interface.

                   

                  My guess is that Apache is caching requests.

                  • 6. Re: Webdav internal caches - how to disable them ?
                    pjakub

                    Well,

                    All guesses were wrong, and I was wrong too. Tomcat never caches 404 not found. I was also debugging tomcat and I'm sure that JcrSession getItem() method is called.

                    I'm using infinispan as storage, and Infinispan doesn't always return fresh data, here's test case:

                     

                    Download Infinispan 5.1.6.Final "Binaries, server and demos" distribution from http://www.jboss.org/infinispan/downloads.html

                    go to bin directory and run:

                    ./runGuiDemo.sh

                    twice. Start both caches. When they're both running add single item at "manipulate data" tab. do it two or three times when looking at both windows, keep Data view open at second window. You'll see that second window is not seeing all new data, or seing it with one operation delay.

                    You have to use Refresh button, or add data in second window by yourself to see fresh data in both windows.

                    And that's my webdav problem. If you use webdav to perform read/write operations, it's working ok, but if you only perform reads, it's getting out of sync, there is no magic "Refresh" button.

                     

                    Magic of "Refresh" button is that it runs :

                    cache.getAdvancedCache().getDataContainer()

                    and that refreshes data.

                     

                    I've got intuition, that there must be some option in infinispan configuration that woul force him to always return fresh data, but can't find one.

                    Quick and dirty hack would be to modify webdav to call same funcation as refresh does  when node is not found, but that would still return old data when ie. listing root node content.

                     

                    That's seems like a bug or misconfiguration. Am I right?

                     

                    Message was edited by: Jakub Pawłowski  I don't understand well what must happen to make JcrSession refresh it's SessionCache in modeshape. I fought that when I add anything at one node, information is send throug infinispan and jgroups to second node and it knows about changes, and refresh internal cache, but that doesn't happen like that. Could someone please explain what needs to be done in webdav instance to fetch fresh data into JcrSession SessionCache ?

                    • 7. Re: Webdav internal caches - how to disable them ?
                      rhauch

                      Can you give us the rest of the ISPN configuration? Also, you said that changing "l1" does nothing ... does that include removing it altogether?

                       

                      Also, does it behave differently in replicated mode?

                      • 8. Re: Webdav internal caches - how to disable them ?
                        hchiorean

                        Jakub, I also notice you're using Infinispan in distributed mode with a DummyTransactionManagerLookup. Why is that ?

                        Have you tried running in local mode to check if the problem still occurs ?

                        • 9. Re: Webdav internal caches - how to disable them ?
                          pjakub

                          @Horia, what kind of transaction manager do you propose? I've copied that from some sample configuration. I don't think local mode would resolve any of those issues.

                           

                           

                          I'm not at work, I've simplified my infinispan configuration, but it does work same way as before:

                           

                          <?xml version="1.0" encoding="UTF-8"?>

                          <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

                              xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"

                              xmlns="urn:infinispan:config:5.1">

                           

                              <global>

                                  <transport clusterName="modeshape-grid">

                                      <properties>

                                          <property name="configurationFile" value="jgroups-ispn.xml" />

                                      </properties>

                                  </transport>

                              </global>

                             

                               <default>

                                <loaders passivation="false" shared="true" preload="false">

                                      <loader class="org.infinispan.loaders.remote.RemoteCacheStore" fetchPersistentState="false"

                                             ignoreModifications="false" purgeOnStartup="false" >

                                            <properties>

                                           <property name="hotRodClientPropertiesFile" value="hotrod-client.properties"/>

                                           <property name="remoteCacheName" value="tlk_content"/>

                                            </properties>

                                       </loader>

                                  </loaders>

                                    <clustering mode="distribution">

                                      <sync />

                                      <hash numOwners="2" rehashWait="120000" rehashRpcTimeout="600000" />

                                      <l1 enabled="true" lifespan="600000" />

                                  </clustering>

                               <default>

                              

                                   <namedCache name="tlk_infinispan_repo">

                             <transaction transactionManagerLookupClass="org.infinispan.transaction.lookup.DummyTransactionManagerLookup" transactionMode="TRANSACTIONAL" lockingMode="OPTIMISTIC" />

                              </namedCache>

                           

                              <namedCache name="dataCache"></namedCache>

                           

                              <namedCache name="metadataCache">  </namedCache>

                          </infinispan>

                           

                           

                          Disabling l1 is not that easy, I've tried setting it's lifespan to 1, it doesn't help.

                          Infinispan is enabled by default.

                          If you set <l1 enabled="false"> in debug output I can see something like : detected L1 element, enabling it (even if I set enabled="false"), and later that causes exception saying "L1 is not enabled".

                          I can't give you detailed exceptions right now, I'll post them at monday because I'm not at work tomorrow. Im using ISPN 5.1.2.Final, there were some bug like that resolved in 5.1.0.CR3

                          I haven't tried replication mode, will try it at monday, but I must have disturbed mode in production, so using replication is not a fix.

                          • 10. Re: Webdav internal caches - how to disable them ?
                            hchiorean

                            Normally you would use org.infinispan.transaction.lookup.GenericTransactionManagerLookup which can auto-detect at runtime the existing JTA provider.

                             

                            Are you able to test with a local cache ? (not a distributed or remote one, just a plain simple local cache using for example a file-store). We haven't really tested Infinispan 5.1.x in distributed mode, but we know 5.1.x is pretty buggy around both replication & distribution, so that would help to narrow it down and tell us whether it's an Infinispan issue or something else.

                             

                            Thanks

                            • 11. Re: Webdav internal caches - how to disable them ?
                              pjakub

                              Ok, I've tried with modeshape file store, and with infinispan as local cache and infinispan witch FileSystemStore,

                              basically if you only read, modeshape doesn't see changes, and that's not fun.

                              Could you please check my configuration?

                               

                              I'm attaching my project source.

                              It's remake of one of examples from github.

                              If you'll import it in eclipse run it with

                              -Djava.net.PrefferIPv4Stack=true -Dlog4j.configFile=log4j.properties

                               

                               

                              Start two instances

                              pres 'l' in both consoles, you should see that there are only two inintial nodes.

                              add some nodes in  window 1, by pressing ie. 'x'

                              switch to window 2, pres l for listing, and you still see only 2 nodes.

                              press 'x' in window 2, and it suddenly sees changes from window 1 + new node, but window 1 doesn't see changes made in window 2.

                              • 12. Re: Webdav internal caches - how to disable them ?
                                rhauch

                                What class are you running? The test case or the main?

                                 

                                It appears that your Infinispan configuration is hard coding the directory where it will store the data, and multiple processes will share this same directory. Each process should have it's own storage (of content and indexes), so none of the Infinispan caches used by ModeShape should be shared (unless you're storing data in a relational database or some place that can handle transactions and concurrent writes; the FileCacheStore is does not fit the bill). I think this is the main cause of your issue.

                                 

                                You're also using OPTIMISTIC locking. That's fine for the very limited cases where there will be no concurrent updates to the same content. Some applications may provide their own infrastructure for centralizing writes, but most won't want to bother and thus should use PESSIMISTIC locking. Perhaps this particular test doesn't really do concurrent writes (unless you are really lucky at trying to create nodes on both clients at exactly the same time), but if you're building up to a practical real-world configuration you should probably start using PESSIMISTIC.

                                 

                                Also, the code is still using the DummyTransactionManager, which is not a full-featured JTA implementation. In fact, I'm not even sure it works except in a local mode. I have no idea whether this will impact the behavior, but I'm pretty sure it will. ModeShape and Infinispan need a full-featured, distributed JTA implementation. Consider using the JBoss Transaction Manager, Atomikos, or some other real transaction manager.

                                • 13. Re: Webdav internal caches - how to disable them ?
                                  pjakub

                                  I'm running main, from org.modeshape.example.filesystem.ModeShapeExample

                                  I'll try changing transaction manager and locking strategy in a minute.

                                  • 14. Re: Webdav internal caches - how to disable them ?
                                    rhauch

                                    And file storage location, right?

                                    1 2 Previous Next