1 2 3 Previous Next 34 Replies Latest reply on Mar 17, 2011 1:29 PM by rshan.bob.shanahan.alascorp.com

    Large file download

    djn

      Hi.


      I've been searching high and low for a solution to a buffering issue I've been having with a large file download in Seam.


      The data is generated by a seam component, and at first I tried something along the lines of:


      In my JSF:


      <s:link value="Fetch file" action="#{streaming.getFile()}"/>
      



      My bean:


      @Name("streaming")
      @Stateless
      public class StreamingTest implements StreamingTestLocal {
          @EJB
          MyStreamerLocal streamer;
      
          @In
          private FacesContext facesContext;
      
          @Override
          public void sendFile() {
              HttpServletResponse response = (HttpServletResponse) facesContext.getExternalContext().getResponse();
              response.setContentType("application/zip");
              response.addHeader("Content-disposition", "attachment; filename=\"data.zip\"");
              response.addHeader("Cache-Control", "no-cache");
              response.setStatus(200);
              try {
                  // committing status and headers
                  response.flushBuffer();
                  ServletOutputStream out = response.getOutputStream();
                  ZipOutputStream zip = new ZipOutputStream(out);
                  zip.putNextEntry(new ZipEntry("data.txt"));
                  PrintWriter writer = new PrintWriter(zip);
                  for (StreamData te : esl) {
                      writer.println(te.getTextualData());
                  }
                  zip.closeEntry();
                  writer.close();
      
              } catch (Exception e) {
                  // ignore for test
              }
              facesContext.responseComplete();
          }
      }
      



      Which actually works and the file is transmitted. However, the output is buffered by something and not sent directly to the browser. That I would very much like to happen.


      I've been playing around with creating a servlet which generates the data. However, I need access to Seam components (e.g. security and data generation) So I added <web:context-filter url-pattern="/streamtestservlet"/> to components.xml and got going. Again, it worked, but something is buffering the data and not transmitting it directly to the client.


      I modified my servlet to transmit random data, disabled the seamfilter for my servlet (in web.xml) and presto! I could stream data to the browser without buffering.


      Can I disable the seamfilter buffering? I'd like it to be for my Seam component (JSF page?), but for the servlet will also do fine.


        • 1. Re: Large file download
          djn

          Anyone?


          I'd really appreciate any input.


          • 2. Re: Large file download
            mail.micke

            Can't really help much but...


            Do you know which filter it is that does the buffering?


            I asked on the old jsf ri mailing list about JSF-RI and buffering and got some useful replies. You should be able to find it in nabble, subject 'Is rendered output buffered or streamed?'


            One reply from Ryan Lubke was extra interesting:




            If you're using 1.2.04 or later, the response won't be buffered until a state marker has been written.  At that point buffering
            begins in order to support writing the state after the the rendering of the component tree has completed.


            The state marker was related to forms in the page (if I remeber correctly) so would guess that doesn't help in your case.


            Also are you using the tomahawk extension filter, I believe that buffers the data as well.


            Good luck and please let us know what you discover,
            micke

            • 3. Re: Large file download

              I hope I understood the problem correctly, did you try to writer.flush() in the loop?


              Anyway this is my pseudo code that worked (the file isn't that big though, just a few hundred kB):



              HttpServletResponse response = (HttpServletResponse)extCtx.getResponse();
              
              response.setContentType("application/octet-stream");
              
              response.addHeader("Content-Disposition","attachment;filename=" + FILE_NAME);
              
              PrintWriter writer = response.getWriter();
              
              ...
              
              while (your condition) {
                  writer.write("do your thing");
              }
              ...
              
              writer.flush();
              
              facesContext.responseComplete();
              
              writer.close();


              • 4. Re: Large file download
                djn

                Hi, and thanks for replying.


                Yes I did. I tried flushing at several levels, nothing changed the observed behaviour.


                With a file of a few 100kb, memory is not an issue. But sometimes the downloads I generate will be quite large. Upto several hundred Mb.



                • 5. Re: Large file download
                  djn

                  Hi, and I really appreciate the input.


                  I'm not using tomahawk.


                  For a seam request it makes no sense disabling the SeamFilter, defined in web.xml:



                  <filter>
                    <filter-name>Seam Filter</filter-name>
                    <filter-class>org.jboss.seam.servlet.SeamFilter</filter-class>
                  </filter>
                  
                  <filter-mapping>
                    <filter-name>Seam Filter</filter-name>
                    <url-pattern>/*</url-pattern>
                  </filter-mapping>
                  



                  If I change the url-pattern to *.seam instead, the servlet I have created streams the data correctly to the client.


                  As I understand it, SeamFilter initializes a lot of different filters. Is there any way to control which ones are enabled for a specific url?


                  • 6. Re: Large file download
                    mail.micke

                    FYI:


                    Became a bit interested and implemented a small test app that sends a large csv file and found the following:


                    OutOfMemoryException happens in the constructor for org.ajax4jsf.io.ByteBuffer(int cacheSize), when the exception occur the value of cacheSize is 268435456 (running in Eclipse with lots of memory).


                    What happens is that the append method tries to double the size of the ByteBuffer :(


                    Wonder if it is possible to bypass the richfaces/ajax4jsf framework for certain urls?


                    The first 3 classes below belong to ajax4jsf.


                    Daemon Thread [http-8080-Processor4] (Suspended (exception OutOfMemoryError))   
                            ByteBuffer.<init>(int) line: 54   
                            ByteBuffer.append(byte[], int, int) line: 112   
                            FastBufferOutputStream.write(byte[]) line: 92   
                            FilterServletResponseWrapper$ByteArrayServletOutputStream.write(byte[]) line: 276       
                            DownloadBacking.fileDownload() line: 44 
                            NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
                            NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39      
                            DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25  
                            Method.invoke(Object, Object...) line: 597      
                            Reflections.invoke(Method, Object, Object...) line: 21  
                            RootInvocationContext.proceed() line: 31        
                            SeamInvocationContext.proceed() line: 56        
                            RollbackInterceptor.aroundInvoke(InvocationContext) line: 31    
                            SeamInvocationContext.proceed() line: 68        
                            BusinessProcessInterceptor.aroundInvoke(InvocationContext) line: 49     
                            SeamInvocationContext.proceed() line: 68        
                            MethodContextInterceptor.aroundInvoke(InvocationContext) line: 42       
                            SeamInvocationContext.proceed() line: 68        
                            JavaBeanInterceptor(RootInterceptor).invoke(InvocationContext, EventType) line: 107     
                            JavaBeanInterceptor.interceptInvocation(Method, Object[]) line: 166     
                            JavaBeanInterceptor.invoke(Object, Method, Method, Object[]) line: 102  
                            DownloadBacking_$$_javassist_3.fileDownload() line: not available       
                            NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
                            NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39      
                            DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25  
                            Method.invoke(Object, Object...) line: 597      
                            ReflectionUtil.invokeMethod(Object, Method, Object[]) line: 329 
                            ReflectionUtil.invokeMethod(Object, Object, Class[], Object[]) line: 342        
                            AstPropertySuffix.invoke(Object, EvaluationContext, Class[], Object[]) line: 58 
                            AstValue.invoke(EvaluationContext, Class[], Object[]) line: 96  
                            MethodExpressionImpl.invoke(ELContext, Object[]) line: 276      
                            TagMethodExpression.invoke(ELContext, Object[]) line: 68        
                            MethodBindingMethodExpressionAdapter.invoke(FacesContext, Object[]) line: 88    
                            ActionListenerImpl.processAction(ActionEvent) line: 102 
                            HtmlCommandButton(UICommand).broadcast(FacesEvent) line: 387    
                            AjaxViewRoot.processEvents(FacesContext, EventsQueue, boolean) line: 317        
                            AjaxViewRoot.broadcastEvents(FacesContext, PhaseId) line: 292   
                            AjaxViewRoot.processPhase(FacesContext, PhaseId, InvokerCallback) line: 249     
                            AjaxViewRoot.processApplication(FacesContext) line: 462 
                            InvokeApplicationPhase.execute(FacesContext) line: 82   
                            InvokeApplicationPhase(Phase).doPhase(FacesContext, Lifecycle, ListIterator<PhaseListener>) line: 100     
                            LifecycleImpl.execute(FacesContext) line: 118   
                            FacesServlet.service(ServletRequest, ServletResponse) line: 265 
                            ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 252      
                            ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 173      
                            SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 83   
                            ExceptionFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 64 
                            SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
                            RedirectFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 45  
                            SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
                            ConfigurableXMLFilter(BaseXMLFilter).doXmlFilter(FilterChain, HttpServletRequest, HttpServletResponse) line: 154        
                            Filter(BaseFilter).handleRequest(HttpServletRequest, HttpServletResponse, FilterChain) line: 260        
                            Filter(BaseFilter).processUploadsAndHandleRequest(HttpServletRequest, HttpServletResponse, FilterChain) line: 366       
                            Filter(BaseFilter).doFilter(ServletRequest, ServletResponse, FilterChain) line: 493     
                            Ajax4jsfFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 60  
                            SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
                            LoggingFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 58   
                            SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
                            HotDeployFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 68 
                            SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
                            SeamFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 158     
                            ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 202      
                            ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 173      
                            StandardWrapperValve.invoke(Request, Response) line: 213        
                            StandardContextValve.invoke(Request, Response) line: 178        
                            NonLoginAuthenticator(AuthenticatorBase).invoke(Request, Response) line: 432    
                            StandardHostValve.invoke(Request, Response) line: 126   
                            ErrorReportValve.invoke(Request, Response) line: 105    
                            StandardEngineValve.invoke(Request, Response) line: 107 
                            CoyoteAdapter.service(Request, Response) line: 148      
                            Http11Processor.process(InputStream, OutputStream) line: 869    
                            Http11Protocol$JmxHttp11ConnectionHandler(Http11BaseProtocol$Http11ConnectionHandler).processConnection(TcpConnection, Object[]) line: 667      
                            PoolTcpEndpoint.processSocket(Socket, TcpConnection, Object[]) line: 527        
                            LeaderFollowerWorkerThread.runIt(Object[]) line: 80     
                            ThreadPool$ControlRunnable.run() line: 684      
                            ThreadWithAttributes(Thread).run() line: 619    
                    
                    


                    • 7. Re: Large file download
                      djn

                      Hi.


                      Interesting.


                      I just double checked my setup and it is also an OutOfMemory Exception within ajax4jsf that stops my application.


                      Caused by: java.lang.OutOfMemoryError: Java heap space
                           at org.ajax4jsf.io.ByteBuffer.<init>(ByteBuffer.java:54)
                           at org.ajax4jsf.io.ByteBuffer.append(ByteBuffer.java:112)
                           at org.ajax4jsf.io.FastBufferOutputStream.write(FastBufferOutputStream.java:107)
                           at org.ajax4jsf.webapp.FilterServletResponseWrapper$ByteArrayServletOutputStream.write(FilterServletResponseWrapper.java:296)
                           at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
                           at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
                           at java.util.zip.ZipOutputStream.write(ZipOutputStream.java:272)
                           at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
                           at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263)
                           at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
                           at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190)
                           at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:111)
                           at java.io.BufferedWriter.write(BufferedWriter.java:212)
                           at java.io.PrintWriter.write(PrintWriter.java:412)
                           at java.io.PrintWriter.write(PrintWriter.java:429)
                           at java.io.PrintWriter.print(PrintWriter.java:559)
                           at java.io.PrintWriter.println(PrintWriter.java:695)
                           at com.blipsystems.blipzones.web.test.StreamingServlet.doGet(StreamingServlet.java:48)
                           at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
                           at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
                           at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
                           at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
                           at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                           at org.jboss.seam.web.ContextFilter$1.process(ContextFilter.java:42)
                           at org.jboss.seam.servlet.ContextualHttpServletRequest.run(ContextualHttpServletRequest.java:53)
                           at org.jboss.seam.web.ContextFilter.doFilter(ContextFilter.java:37)
                           at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
                      



                      I've tried disabling ajax4jsf xmlparser as described here, to no avail.

                      • 8. Re: Large file download
                        mail.micke

                        FYI:
                        I use the NEKO filter, which sped up the render response phase a lot for me when rendering larg:ish tables.


                        You might want to take this discussion to the RichFaces forum and hope that one of their developers can shed some light on the (remote) possibility of avioding the buffering.


                        Cheers,
                        micke

                        • 9. Re: Large file download

                          In the early days you had to specify a separate filter mapping for ajax4jsf filters seam apps.


                          I am wondering whether it's still possible to override the ajax4jsf filter mapping for your app...

                          • 10. Re: Large file download
                            mirco

                            Daniel,


                            I have exactly the same problem. I am trying to stream files from amazon s3 to my users, but the file starts downloading to user only after it is completely downloaded to my server. Some servlet is buffering incoming stream and only allows reading it after it is closed. :-(


                            I hope to find a solution soon.

                            • 11. Re: Large file download
                              djn

                              Hi,


                              It's nice to know that I'm not the only one with this issue :-)


                              However, I'm currently busy with other projects but will be looking into the buffering problems again soon.

                              • 12. Re: Large file download
                                gena

                                Hello Daniel and all,



                                It's nice to know that I'm not the only one with this issue :-)

                                I fill so, as i run already into all possible Seam/Richfaces issues ;-). Back to the thema:




                                Ajax4Jsf filter (via FilterServletResponseWrapper) uses the byte array buffer (FastBufferOutputStream) and fills it before transfering data to the native response.outputStream. Naturally it can't work with large files. The only solution i found, is to disable the ajax4jsf filter for affected servlet path. <url-pattern>*.seam</url-pattern> repairs the download servlet but all scripts and css aren't served at all!



                                I didn't find how to negate the url pattern and i think it could be very helpful to introduce something like regex-url-pattern-exclude and url-pattern-exclude into the web:*filter components. My solution is to override the doFilter(...) method of Ajax4jsfFilter in my own component to something like:




                                @Scope(APPLICATION)
                                @Name("org.jboss.seam.web.ajax4jsfFilter")
                                @Install(dependencies=["org.jboss.seam.web.ajax4jsfFilterInstantiator"])
                                @BypassInterceptors
                                @Filter
                                public class Ajax4jsfFilter extends Ajax4jsfFilter{
                                      def log = Logging.getLog(Ajax4jsfFilter.class)
                                      public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain chain) throws IOException, ServletException
                                        { 
                                          log.trace(servletRequest.requestURI)
                                           if(servletRequest.requestURI =~ '^.+/file/.+$')
                                                // bypass a4jsf filter for download servlet
                                                chain.doFilter(servletRequest, servletResponse)
                                           else
                                                super.doFilter(servletRequest,servletResponse,chain)
                                                
                                        }   
                                     
                                }




                                I also got problem with the url rewriting: because the ajax4jsf acts as the front filter, the url is not rewriten back to the RESOURCE_PATH of my FileDownloadResource. So my regex is built using the rewriten url part (ugly!).



                                Hope, it can help


                                Gena

                                • 13. Re: Large file download
                                  rituraj_tiwari

                                  Gena,
                                  Came across your post while looking for something similar. Any chance you came up with a cleaner solution?


                                  Thanks.


                                  -Raj

                                  • 14. Re: Large file download
                                    stephen

                                    I think it should be enough to add this line to components.xml to exclude the ajax4jsf filter for any URL that contains /file/:



                                    <web:ajax4jsf-filter regex-url-pattern="^(?:[^/]|/(?!file/))*$"/>



                                    1 2 3 Previous Next