5 Replies Latest reply on Oct 28, 2015 4:32 AM by hchiorean

    ~60MB mp4 files uploaded through WEBDAV interface not surviving WildFly restarts

    mashama

      I am able to replicate this on both Windows and OSX.  I am uploading a ~60MB mp4 via WEBDAV only to find the file inaccessible after restarting WildFly.  The issue seems to impact all releases after the 4.1 release.  Here are the replication steps:

       

      1. Download and extract WildFly 9.0.1
      2. Download and extract into WildFly the 4.4 ModeShape WildFly subsystem
      3. Configure 'max-post-size' in undertow subsystem in '100000000' in standalone-modeshape.xml
      4. Start WildFly with standalone-modeshape.xml configuration file
      5. Upload mp4 file into sample/default via WEBDAV interface
      6. Download mp4 file via WEBDAV to confirm successful upload
      7. Restart WildFly
      8. Download mp4 file via WEBDAV to see 0 B transfered

       

      I used CyberDuck on Windows and "connect to server" on OSX. 

       

      sample.mp4 - Google Drive

        • 1. Re: ~60MB mp4 files uploaded through WEBDAV interface not surviving WildFly restarts
          hchiorean

          Is this related to the repository storage of binary data or WebDAV or Wildfly ? Did you try uploading & reading between restarts the same file using the REST service for example ? Or directly using the JCR API ?

          Also, did try setting the logging level to DEBUG and check if there are any messages which could explain why this is happening ?

           

          Update: I don't know how Wildfly works internally, but 100MB may not be enough. I tried locally with max-post-size="212400000" max-header-size="212400000" and the upload/download worked fine.

          • 2. Re: ~60MB mp4 files uploaded through WEBDAV interface not surviving WildFly restarts
            mashama

            I just tried to retrieve the file via the REST service and got the following error/stacktrace:

             

            2015-10-21 09:57:20,243 ERROR [org.modeshape.web.jcr.rest.ModeShapeExceptionMapper] (default task-17) Server error: org.modeshape.jcr.value.binary.BinaryStoreException: Unable to find binary value with key "c9ca2cca218d886cec4727c3351fae3f715df2f4" within binary store at "C:\Users\MMCFAR~1\AppData\Local\Temp\modeshape-binary-store"

              at org.modeshape.jcr.value.binary.FileSystemBinaryStore.getStoredMimeType(FileSystemBinaryStore.java:592)

              at org.modeshape.jcr.value.binary.AbstractBinaryStore.getMimeType(AbstractBinaryStore.java:160)

              at org.modeshape.jcr.value.binary.StoredBinaryValue.getMimeType(StoredBinaryValue.java:55)

              at org.modeshape.web.jcr.rest.handler.RestBinaryHandler.getDefaultMimeType(RestBinaryHandler.java:96)

              at org.modeshape.web.jcr.rest.ModeShapeRestService.getBinary(ModeShapeRestService.java:306)

              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

              at java.lang.reflect.Method.invoke(Method.java:483)

              at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)

              at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)

              at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)

              at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:237)

              at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)

              at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:179)

              at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)

              at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)

              at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)

              at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

              at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:86)

              at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)

              at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)

              at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

              at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)

              at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)

              at io.undertow.server.handlers.DisableCacheHandler.handleRequest(DisableCacheHandler.java:33)

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

              at io.undertow.security.handlers.AuthenticationConstraintHandler.handleRequest(AuthenticationConstraintHandler.java:51)

              at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)

              at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)

              at io.undertow.servlet.handlers.security.ServletSecurityConstraintHandler.handleRequest(ServletSecurityConstraintHandler.java:56)

              at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:58)

              at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:72)

              at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)

              at io.undertow.security.handlers.SecurityInitialHandler.handleRequest(SecurityInitialHandler.java:76)

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

              at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

              at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)

              at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:282)

              at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:261)

              at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:80)

              at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:172)

              at io.undertow.server.Connectors.executeRootHandler(Connectors.java:199)

              at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:774)

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

              at java.lang.Thread.run(Thread.java:745)

             

            Before doing so I updated the max-post-size and max-header-size configuration.  Looking at the stacktrace above perhaps it is related to the storage of binary data.  I have always known the out of box configuration to store the binary data within the data directory in the WildFly installation directory.  I will try setting the logging level to DEBUG and see if it reveals any other tidbits.  Just for clarification I initially encountered the issue when upgrading from 4.1 to 4.2 on WildFly 8.2.

            • 3. Re: ~60MB mp4 files uploaded through WEBDAV interface not surviving WildFly restarts
              hchiorean

              Using the settings from my previous post I was able to upload & download the file after a restart using the latest master code and Rei's Shed - WebDAV Client CarotDAV - as a WebDAV client storing the binary data in a FS binary store on disk.

               

              I suggest you write a simple test using the JCR API of uploading the file without going through any web client. That should tell you if the problem has to do with the binary storage or not (I doubt that's the case since there are no significant changes between 4.4.0.Final and the lastest master in that area). Also, you should make sure that when you start you upload the file from scratch (using the updated max-post settings)

              • 4. Re: ~60MB mp4 files uploaded through WEBDAV interface not surviving WildFly restarts
                mashama

                So I was able to replicate the issue with the latest off of master running in WildFly 9.0.  It turns out that this seemingly has nothing to do with the WEBDAV interface as I am, per your suggestion, storing the file with the JCR API and then retrieving it via the REST interface.  Working backwards with logging set to DEBUG it seems that in 4.1 the binary store defaults to type = "file" and points to a directory within the  WildFly installation directory, while 4.2 defaults to type = "transient".

                 

                4.1:

                 

                00:39:23,886 DEBUG [org.modeshape.jcr.JcrRepository] (default task-2) Starting 'sample' repository with configuration:

                { "name" : "sample" , "jndiName" : "" , "monitoring" : { "enabled" : true } , "workspaces" : { "allowCreation" : true , "default" : "default" } , "storage" : { "cacheName" : "sample" , "cacheConfiguration" : "content" , "binaryStorage" : { "type" : "file" , "directory" : "/Users/mashama/Downloads/wildfly-8.2.0.Final 5/standalone/data/modeshape/sample/sample/binaries" } } , "security" : { "jaas" : { "policyName" : "modeshape-security" } , "anonymous" : { "username" : "<anonymous>" , "useOnFailedLogin" : false , "roles" : [ "admin" ] } , "providers" : [ { "classname" : "servlet" } ] } , "garbageCollection" : { "threadPool" : "modeshape-gc" , "initialTime" : "00:00" , "intervalInHours" : 24 } }

                 

                4.2:

                 

                00:53:57,026 DEBUG [org.modeshape.jcr.JcrRepository] (default task-3) Starting 'sample' repository with configuration:

                { "name" : "sample" , "jndiName" : "" , "monitoring" : { "enabled" : true } , "workspaces" : { "allowCreation" : true , "default" : "default" } , "storage" : { "cacheName" : "sample" , "cacheConfiguration" : "content" , "binaryStorage" : { "type" : "transient" } } , "security" : { "anonymous" : { "username" : "<anonymous>" , "useOnFailedLogin" : false , "roles" : [ "admin" ] } , "providers" : [ { "classname" : "org.modeshape.jboss.security.JBossDomainAuthenticationProvider" , "domainName" : "modeshape-security" } , { "classname" : "servlet" } ] } , "garbageCollection" : { "threadPool" : "modeshape-gc" , "initialTime" : "00:00" , "intervalInHours" : 24 } }

                 

                Looking at the standalone-modeshape.xml files across both these versions it is apparent that the following was added in 4.2:

                 

                                <!--Store binary artifacts on the FS under ${jboss.data.dir}/modeshape/artifacts/binaries -->

                                <file-binary-storage/>

                 

                This is the root cause of my issue.  Looking forward to ModeShape 4.x on WildFly 10.x!

                • 5. Re: ~60MB mp4 files uploaded through WEBDAV interface not surviving WildFly restarts
                  hchiorean

                  Glad you found the issue. You're correct: starting with 4.2.0.Final ([MODE-2402] WF kit should not persist binaries on the FS is no such store is configured - JBoss Issue Tracker) when using the WF kit, ModeShape will not persist binaries on disk by default. This will only happen when a <file-binary-store> is explicitly configured.