Large file download

[djn]

Hi.

I've been searching high and low for a solution to a buffering issue I've been having with a large file download in Seam.

The data is generated by a seam component, and at first I tried something along the lines of:

In my JSF:

<s:link value="Fetch file" action="#{streaming.getFile()}"/>

My bean:

@Name("streaming")
@Stateless
public class StreamingTest implements StreamingTestLocal {
    @EJB
    MyStreamerLocal streamer;

    @In
    private FacesContext facesContext;

    @Override
    public void sendFile() {
        HttpServletResponse response = (HttpServletResponse) facesContext.getExternalContext().getResponse();
        response.setContentType("application/zip");
        response.addHeader("Content-disposition", "attachment; filename=\"data.zip\"");
        response.addHeader("Cache-Control", "no-cache");
        response.setStatus(200);
        try {
            // committing status and headers
            response.flushBuffer();
            ServletOutputStream out = response.getOutputStream();
            ZipOutputStream zip = new ZipOutputStream(out);
            zip.putNextEntry(new ZipEntry("data.txt"));
            PrintWriter writer = new PrintWriter(zip);
            for (StreamData te : esl) {
                writer.println(te.getTextualData());
            }
            zip.closeEntry();
            writer.close();

        } catch (Exception e) {
            // ignore for test
        }
        facesContext.responseComplete();
    }
}

Which actually works and the file is transmitted. However, the output is buffered by something and not sent directly to the browser. That I would very much like to happen.

I've been playing around with creating a servlet which generates the data. However, I need access to Seam components (e.g. security and data generation) So I added <web:context-filter url-pattern="/streamtestservlet"/> to components.xml and got going. Again, it worked, but something is buffering the data and not transmitting it directly to the client.

I modified my servlet to transmit random data, disabled the seamfilter for my servlet (in web.xml) and presto! I could stream data to the browser without buffering.

Can I disable the seamfilter buffering? I'd like it to be for my Seam component (JSF page?), but for the servlet will also do fine.

[djn]

Anyone?

I'd really appreciate any input.

[mail.micke]

Can't really help much but...

Do you know which filter it is that does the buffering?

I asked on the old jsf ri mailing list about JSF-RI and buffering and got some useful replies. You should be able to find it in nabble, subject 'Is rendered output buffered or streamed?'

One reply from Ryan Lubke was extra interesting:

If you're using 1.2.04 or later, the response won't be buffered until a state marker has been written.  At that point buffering
begins in order to support writing the state after the the rendering of the component tree has completed.

The state marker was related to forms in the page (if I remeber correctly) so would guess that doesn't help in your case.

Also are you using the tomahawk extension filter, I believe that buffers the data as well.

Good luck and please let us know what you discover,
micke

[svadu]

I hope I understood the problem correctly, did you try to writer.flush() in the loop?

Anyway this is my pseudo code that worked (the file isn't that big though, just a few hundred kB):

HttpServletResponse response = (HttpServletResponse)extCtx.getResponse();

response.setContentType("application/octet-stream");

response.addHeader("Content-Disposition","attachment;filename=" + FILE_NAME);

PrintWriter writer = response.getWriter();

...

while (your condition) {
    writer.write("do your thing");
}
...

writer.flush();

facesContext.responseComplete();

writer.close();

[djn]

Hi, and thanks for replying.

Yes I did. I tried flushing at several levels, nothing changed the observed behaviour.

With a file of a few 100kb, memory is not an issue. But sometimes the downloads I generate will be quite large. Upto several hundred Mb.

[djn]

Hi, and I really appreciate the input.

I'm not using tomahawk.

For a seam request it makes no sense disabling the SeamFilter, defined in web.xml:

<filter>
  <filter-name>Seam Filter</filter-name>
  <filter-class>org.jboss.seam.servlet.SeamFilter</filter-class>
</filter>

<filter-mapping>
  <filter-name>Seam Filter</filter-name>
  <url-pattern>/*</url-pattern>
</filter-mapping>

If I change the url-pattern to *.seam instead, the servlet I have created streams the data correctly to the client.

As I understand it, SeamFilter initializes a lot of different filters. Is there any way to control which ones are enabled for a specific url?

[mail.micke]

FYI:

Became a bit interested and implemented a small test app that sends a large csv file and found the following:

OutOfMemoryException happens in the constructor for org.ajax4jsf.io.ByteBuffer(int cacheSize), when the exception occur the value of cacheSize is 268435456 (running in Eclipse with lots of memory).

What happens is that the append method tries to double the size of the ByteBuffer :(

Wonder if it is possible to bypass the richfaces/ajax4jsf framework for certain urls?

The first 3 classes below belong to ajax4jsf.

Daemon Thread [http-8080-Processor4] (Suspended (exception OutOfMemoryError))   
        ByteBuffer.<init>(int) line: 54   
        ByteBuffer.append(byte[], int, int) line: 112   
        FastBufferOutputStream.write(byte[]) line: 92   
        FilterServletResponseWrapper$ByteArrayServletOutputStream.write(byte[]) line: 276       
        DownloadBacking.fileDownload() line: 44 
        NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
        NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39      
        DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25  
        Method.invoke(Object, Object...) line: 597      
        Reflections.invoke(Method, Object, Object...) line: 21  
        RootInvocationContext.proceed() line: 31        
        SeamInvocationContext.proceed() line: 56        
        RollbackInterceptor.aroundInvoke(InvocationContext) line: 31    
        SeamInvocationContext.proceed() line: 68        
        BusinessProcessInterceptor.aroundInvoke(InvocationContext) line: 49     
        SeamInvocationContext.proceed() line: 68        
        MethodContextInterceptor.aroundInvoke(InvocationContext) line: 42       
        SeamInvocationContext.proceed() line: 68        
        JavaBeanInterceptor(RootInterceptor).invoke(InvocationContext, EventType) line: 107     
        JavaBeanInterceptor.interceptInvocation(Method, Object[]) line: 166     
        JavaBeanInterceptor.invoke(Object, Method, Method, Object[]) line: 102  
        DownloadBacking_$$_javassist_3.fileDownload() line: not available       
        NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method]  
        NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39      
        DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25  
        Method.invoke(Object, Object...) line: 597      
        ReflectionUtil.invokeMethod(Object, Method, Object[]) line: 329 
        ReflectionUtil.invokeMethod(Object, Object, Class[], Object[]) line: 342        
        AstPropertySuffix.invoke(Object, EvaluationContext, Class[], Object[]) line: 58 
        AstValue.invoke(EvaluationContext, Class[], Object[]) line: 96  
        MethodExpressionImpl.invoke(ELContext, Object[]) line: 276      
        TagMethodExpression.invoke(ELContext, Object[]) line: 68        
        MethodBindingMethodExpressionAdapter.invoke(FacesContext, Object[]) line: 88    
        ActionListenerImpl.processAction(ActionEvent) line: 102 
        HtmlCommandButton(UICommand).broadcast(FacesEvent) line: 387    
        AjaxViewRoot.processEvents(FacesContext, EventsQueue, boolean) line: 317        
        AjaxViewRoot.broadcastEvents(FacesContext, PhaseId) line: 292   
        AjaxViewRoot.processPhase(FacesContext, PhaseId, InvokerCallback) line: 249     
        AjaxViewRoot.processApplication(FacesContext) line: 462 
        InvokeApplicationPhase.execute(FacesContext) line: 82   
        InvokeApplicationPhase(Phase).doPhase(FacesContext, Lifecycle, ListIterator<PhaseListener>) line: 100     
        LifecycleImpl.execute(FacesContext) line: 118   
        FacesServlet.service(ServletRequest, ServletResponse) line: 265 
        ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 252      
        ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 173      
        SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 83   
        ExceptionFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 64 
        SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
        RedirectFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 45  
        SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
        ConfigurableXMLFilter(BaseXMLFilter).doXmlFilter(FilterChain, HttpServletRequest, HttpServletResponse) line: 154        
        Filter(BaseFilter).handleRequest(HttpServletRequest, HttpServletResponse, FilterChain) line: 260        
        Filter(BaseFilter).processUploadsAndHandleRequest(HttpServletRequest, HttpServletResponse, FilterChain) line: 366       
        Filter(BaseFilter).doFilter(ServletRequest, ServletResponse, FilterChain) line: 493     
        Ajax4jsfFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 60  
        SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
        LoggingFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 58   
        SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
        HotDeployFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 68 
        SeamFilter$FilterChainImpl.doFilter(ServletRequest, ServletResponse) line: 69   
        SeamFilter.doFilter(ServletRequest, ServletResponse, FilterChain) line: 158     
        ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 202      
        ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 173      
        StandardWrapperValve.invoke(Request, Response) line: 213        
        StandardContextValve.invoke(Request, Response) line: 178        
        NonLoginAuthenticator(AuthenticatorBase).invoke(Request, Response) line: 432    
        StandardHostValve.invoke(Request, Response) line: 126   
        ErrorReportValve.invoke(Request, Response) line: 105    
        StandardEngineValve.invoke(Request, Response) line: 107 
        CoyoteAdapter.service(Request, Response) line: 148      
        Http11Processor.process(InputStream, OutputStream) line: 869    
        Http11Protocol$JmxHttp11ConnectionHandler(Http11BaseProtocol$Http11ConnectionHandler).processConnection(TcpConnection, Object[]) line: 667      
        PoolTcpEndpoint.processSocket(Socket, TcpConnection, Object[]) line: 527        
        LeaderFollowerWorkerThread.runIt(Object[]) line: 80     
        ThreadPool$ControlRunnable.run() line: 684      
        ThreadWithAttributes(Thread).run() line: 619    

[djn]

Hi.

Interesting.

I just double checked my setup and it is also an OutOfMemory Exception within ajax4jsf that stops my application.

Caused by: java.lang.OutOfMemoryError: Java heap space
     at org.ajax4jsf.io.ByteBuffer.<init>(ByteBuffer.java:54)
     at org.ajax4jsf.io.ByteBuffer.append(ByteBuffer.java:112)
     at org.ajax4jsf.io.FastBufferOutputStream.write(FastBufferOutputStream.java:107)
     at org.ajax4jsf.webapp.FilterServletResponseWrapper$ByteArrayServletOutputStream.write(FilterServletResponseWrapper.java:296)
     at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:161)
     at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
     at java.util.zip.ZipOutputStream.write(ZipOutputStream.java:272)
     at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
     at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263)
     at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
     at java.io.OutputStreamWriter.write(OutputStreamWriter.java:190)
     at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:111)
     at java.io.BufferedWriter.write(BufferedWriter.java:212)
     at java.io.PrintWriter.write(PrintWriter.java:412)
     at java.io.PrintWriter.write(PrintWriter.java:429)
     at java.io.PrintWriter.print(PrintWriter.java:559)
     at java.io.PrintWriter.println(PrintWriter.java:695)
     at com.blipsystems.blipzones.web.test.StreamingServlet.doGet(StreamingServlet.java:48)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83)
     at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73)
     at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64)
     at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
     at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45)
     at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)
     at org.jboss.seam.web.ContextFilter$1.process(ContextFilter.java:42)
     at org.jboss.seam.servlet.ContextualHttpServletRequest.run(ContextualHttpServletRequest.java:53)
     at org.jboss.seam.web.ContextFilter.doFilter(ContextFilter.java:37)
     at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69)

I've tried disabling ajax4jsf xmlparser as described here, to no avail.

[mail.micke]

FYI:
I use the NEKO filter, which sped up the render response phase a lot for me when rendering larg:ish tables.

You might want to take this discussion to the RichFaces forum and hope that one of their developers can shed some light on the (remote) possibility of avioding the buffering.

Cheers,
micke

[svadu]

In the early days you had to specify a separate filter mapping for ajax4jsf filters seam apps.

I am wondering whether it's still possible to override the ajax4jsf filter mapping for your app...

[mirco]

Daniel,

I have exactly the same problem. I am trying to stream files from amazon s3 to my users, but the file starts downloading to user only after it is completely downloaded to my server. Some servlet is buffering incoming stream and only allows reading it after it is closed. :-(

I hope to find a solution soon.

[djn]

Hi,

It's nice to know that I'm not the only one with this issue :-)

However, I'm currently busy with other projects but will be looking into the buffering problems again soon.

[gena]

Hello Daniel and all,

It's nice to know that I'm not the only one with this issue :-)

I fill so, as i run already into all possible Seam/Richfaces issues ;-). Back to the thema:

Ajax4Jsf filter (via FilterServletResponseWrapper) uses the byte array buffer (FastBufferOutputStream) and fills it before transfering data to the native response.outputStream. Naturally it can't work with large files. The only solution i found, is to disable the ajax4jsf filter for affected servlet path. <url-pattern>*.seam</url-pattern> repairs the download servlet but all scripts and css aren't served at all!

I didn't find how to negate the url pattern and i think it could be very helpful to introduce something like regex-url-pattern-exclude and url-pattern-exclude into the web:*filter components. My solution is to override the doFilter(...) method of Ajax4jsfFilter in my own component to something like:

@Scope(APPLICATION)
@Name("org.jboss.seam.web.ajax4jsfFilter")
@Install(dependencies=["org.jboss.seam.web.ajax4jsfFilterInstantiator"])
@BypassInterceptors
@Filter
public class Ajax4jsfFilter extends Ajax4jsfFilter{
      def log = Logging.getLog(Ajax4jsfFilter.class)
      public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain chain) throws IOException, ServletException
        { 
          log.trace(servletRequest.requestURI)
           if(servletRequest.requestURI =~ '^.+/file/.+$')
                // bypass a4jsf filter for download servlet
                chain.doFilter(servletRequest, servletResponse)
           else
                super.doFilter(servletRequest,servletResponse,chain)
                
        }   
     
}

I also got problem with the url rewriting: because the ajax4jsf acts as the front filter, the url is not rewriten back to the RESOURCE_PATH of my FileDownloadResource. So my regex is built using the rewriten url part (ugly!).

Hope, it can help

Gena

[rituraj_tiwari]

Gena,
Came across your post while looking for something similar. Any chance you came up with a cleaner solution?

Thanks.

-Raj

[stephen]

I think it should be enough to add this line to components.xml to exclude the ajax4jsf filter for any URL that contains /file/:

<web:ajax4jsf-filter regex-url-pattern="^(?:[^/]|/(?!file/))*$"/>