6 Replies Latest reply on Apr 20, 2012 11:43 AM by dex80526

    No such file or directory exception in FileCacheStore

    dex80526

      I ran into the following exceptions when I am testing the FileCacheStore in my code.  It randomly happens.

      Here is my environment:

      ISPN 5.1.3 FINAL, CentOS 5.x, and Java 7

       

      I configured the CacheStore as the following:

      ...

      <namedCache name="ispn-test">

              <clustering mode="replication">

                  <stateRetrieval

                          timeout="240000"

                          fetchInMemoryState="true"

                          alwaysProvideInMemoryState="false"

                          />

                  <async useReplQueue="true" replQueueInterval="5000" replQueueMaxElements="500" asyncMarshalling="false" />

              </clustering>

              <loaders passivation="false" shared="false" preload="true">

                  <!-- We can have multiple cache loaders, which get chained -->

                  <loader class="org.infinispan.loaders.file.FileCacheStore"

                          fetchPersistentState="true"

                          purgerThreads="3"

                          purgeSynchronously="true"

                          ignoreModifications="false"

                          purgeOnStartup="false">

                      <!-- See the documentation for more configuration examples and flags. -->

                      <properties>

                          <property name="location" value="/tmp/test/cacheData/cipher"/>

                      </properties>

                      <async enabled="true" flushLockTimeout="15000" threadPoolSize="5"/>

                  </loader>

              </loaders>

              <transaction transactionMode="NON_TRANSACTIONAL"/>

              <eviction

                      maxEntries="50"

                      strategy="LRU"

                      />

              <!--  time units below are millseconds -->

              <expiration

                      wakeUpInterval="-1"

                      lifespan="-1"

                      maxIdle="-1"

                      />

       

          </namedCache>

      ...

       

      I noticed that the cahce store dir is not created on the filesystem.

       

      What cause the failures? Do I miss something in the config?

       

      ======

      2012-04-16/13:08:41.661/MDT [CoalescedAsyncStore-5] ERROR org.infinispan.loaders.file.FileCacheStore[338] - ISPN000063: Exception while saving bucket Bucket{entries={TestAds_userSearch=ImmortalCacheEntry{key=TestAds_userSearch, value=ImmortalCacheValue {value=[B@6a5baf1c}}}, bucketId='-150279168'}

      java.io.IOException: No such file or directory

              at java.io.UnixFileSystem.createFileExclusively(Native Method)

              at java.io.File.createNewFile(File.java:947)

              at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.write(FileCacheStore.java:530)

              at org.infinispan.loaders.file.FileCacheStore.updateBucket(FileCacheStore.java:336)

              at org.infinispan.loaders.bucket.BucketBasedCacheStore.insertBucket(BucketBasedCacheStore.java:137)

              at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:94)

              at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)

              at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:207)

              at org.infinispan.loaders.decorators.AbstractDelegatingStore.store(AbstractDelegatingStore.java:68)

              at org.infinispan.loaders.decorators.AsyncStore.applyModificationsSync(AsyncStore.java:245)

              at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.put(AsyncStore.java:405)

              at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.innerRun(AsyncStore.java:389)

              at org.infinispan.loaders.decorators.AsyncStore$AsyncProcessor.run(AsyncStore.java:312)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

              at java.lang.Thread.run(Thread.java:722)

        • 1. Re: No such file or directory exception in FileCacheStore
          dex80526

          This problem is very reproducable in my environment.  In the following stack trace, you can see the FileCacheStore does detectect the file exists on the disk.  Howerver, when the channel tries to flush, it throws the exception.

          Note: this is a 2 node cluster in replication.

          Here is more trace:

          ...

          2012-04-17/13:23:34.660/MDT [Incoming-1,Dex,dexportal2.net-63566] TRACE org.infinispan.loaders.file.FileCacheStore[294] - Found bucket file: '/test/ldapcookie/ispn-ldapcookie/-1285177344'

          2012-04-17/13:23:34.661/MDT [Incoming-1,Dex,dexportal2.net-63566] ERROR org.infinispan.loaders.file.FileCacheStore[310] - ISPN000062: Error while reading from file: /test/ldapcookie/ispn-ldapcookie/-1285177344

          java.io.IOException: No such device

                  at sun.nio.ch.FileDispatcherImpl.force0(Native Method)

                  at sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:75)

                  at sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:363)

                  at org.infinispan.loaders.file.FileCacheStore$BufferedFileSync.flush(FileCacheStore.java:563)

                  at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:300)

                  at org.infinispan.loaders.file.FileCacheStore.loadBucket(FileCacheStore.java:280)

                  at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:86)

                  at org.infinispan.loaders.bucket.BucketBasedCacheStore.storeLockSafe(BucketBasedCacheStore.java:49)

                  at org.infinispan.loaders.LockSupportCacheStore.store(LockSupportCacheStore.java:207)

                  at org.infinispan.interceptors.CacheStoreInterceptor.visitPutKeyValueCommand(CacheStoreInterceptor.java:214)

                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  at org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:89)

                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  at org.infinispan.interceptors.EntryWrappingInterceptor.invokeNextAndApplyChanges(EntryWrappingInterceptor.java:199)

                  at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:134)

                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  at org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor.visitPutKeyValueCommand(NonTransactionalLockingInterceptor.java:68)

                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61)

                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  at org.infinispan.interceptors.StateTransferLockInterceptor.handleWithRetries(StateTransferLockInterceptor.java:208)

                  at org.infinispan.interceptors.StateTransferLockInterceptor.handleWriteCommand(StateTransferLockInterceptor.java:181)

                  at org.infinispan.interceptors.StateTransferLockInterceptor.visitPutKeyValueCommand(StateTransferLockInterceptor.java:152)

                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:124)

                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

                  at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:130)

                  at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:61)

                  at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:108)

                  at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:76)

                  at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:116)

          • 2. Re: No such file or directory exception in FileCacheStore
            dex80526

            It seems none has any insight to this.  Any hint is helpful.

             

            I tried with JdbmCacheStore in the place of FileCacheStore, I got the simialr errors.  

            • 3. Re: No such file or directory exception in FileCacheStore
              nadirx

              What kind of filesystem are you using ?

              • 4. Re: No such file or directory exception in FileCacheStore
                dex80526

                The filesystem is Linux ext3. I saw the same issues with both  NFS and CIFS monuts. I was looking at the BufferedFileSync code where the Exception occured.

                 

                Do you think that we get the obsoleted streams (the mapping between path and channel) entries?  In other words, we get the channle back from the strams map, which is already invalid.  I think this could happen if we un-mount and (re) mount the dir/filesystem, but we did not update this map.

                • 5. Re: No such file or directory exception in FileCacheStore
                  nadirx

                  That is what it looks like. I'll investigate the matter further

                  • 6. Re: No such file or directory exception in FileCacheStore
                    dex80526

                    Just FYI. I confirmed the IOException can be reproduced by just umount and re-mount the CIFS/NFS share which used by the cache store.