Dandelion not flushing response buffer until complete

classic Classic list List threaded Threaded
7 messages Options
tom
Reply | Threaded
Open this post in threaded view
|

Dandelion not flushing response buffer until complete

tom
Hi guys,

I'm having issues that seem like they might be coming from the way DandelionFilter wraps a HttpServletResponse.  We have a method in our web application that zips a number of large files with a ZipOutputStream and returns them via the HttpServletResponse.  Prior to using Dandelion this method was able to stream the zipping and download to the user without much impact on the server's memory.  After Dandelion started being used it appears the HttpServletResponse doesn't respect its buffer size and holds on to the entire download until the response is complete which will often cause the machine to run out of heap space.

Here's an example of a method where this would occur:

public void downloadFiles(HttpServletResponse response) throws IOException {
        // Add the appropriate headers
        response.setContentType("application/zip");
        response.setHeader("Content-Disposition", "attachment; filename=\"download.zip\"");
        response.setHeader("Transfer-Encoding", "chunked");

        List<Path> files = null; // get files from somewhere

        // open zip stream
        try (ZipOutputStream outputStream = new ZipOutputStream(response.getOutputStream())) {
                for (Path path : files) {
                        String name = path.getFileName().toString();

                        outputStream.putNextEntry(new ZipEntry(name.toString()));

                        Files.copy(path, outputStream);

                        outputStream.closeEntry();
                }
                outputStream.finish();
        }
}

It looks like DandelionFilter is holding the response from being flushed until its able to perform its magic, but this causes issues with large responses.

Is there any way to disable dandelion on a given method or to otherwise deal with this problem?
Reply | Threaded
Open this post in threaded view
|

Re: Dandelion not flushing response buffer until complete

Thibault Duchateau
Administrator
Hi Tom,

Prior to the v1.0.0, you can disable the DandelionFilter using request attributes or parameters. See https://github.com/dandelion/dandelion/blob/dandelion-0.10.1/dandelion-core/src/main/java/com/github/dandelion/core/web/DandelionFilter.java#L187-235. So in order to see if it solves the issue, and if you can quickly modify request URIs, try to append ?dandelionAssetFilterState=false.

Starting from the v1.0.0 (which is currently being released), the DandelionFilter has been fully refactored, including the way the HttpServletResponse is processed . It would be great if you could give it a try. If the problem persists, it seems to be the priority issue for the 1.0.1 :-)

Regards,
Thibault.
tom
Reply | Threaded
Open this post in threaded view
|

Re: Dandelion not flushing response buffer until complete

tom
Hi Thibault, thanks for the quick reply.

I looked at the solution you provided with adding the query param dandelionAssetFilterState=false to the request.  Unfortunately this doesn't seem to solve our problem.  While it seems to successfully mark the request as not relevant in DandelionFilter#isRelevant, that operation doesn't occur until after the response has been loaded into memory.  

Assuming I'm reading this right I think the offending section would be at https://github.com/dandelion/dandelion/blob/dandelion-0.10.1/dandelion-core/src/main/java/com/github/dandelion/core/web/DandelionFilter.java#L136-L147.  After this the response is checked for "relevancy", and if it isn't relevant the response byte array is dumped to the response at https://github.com/dandelion/dandelion/blob/dandelion-0.10.1/dandelion-core/src/main/java/com/github/dandelion/core/web/DandelionFilter.java#L181-L184.

I've tested a solution which checks the request flag before wrapping the response, and if the flag is set will immediately continue with the filter chain.  If you would like I could create a pull request for you to take a look.
Reply | Threaded
Open this post in threaded view
|

Re: Dandelion not flushing response buffer until complete

Thibault Duchateau
Administrator
Hi Tom,

Thanks for your feedback. The issue has been added: https://github.com/dandelion/dandelion/issues/60
Already fixed but not commited yet. The issue was coming from the ByteArrayResponseWrapper, I'll let you look at the fix if you're interesed.

However, I'm also figuring out how the entire filter can be properly disabled in such a use-case. Using the internal request and response handlers seems interesting but require some adaptations..

Anyway, thanks!

Regards,
Thibault
Reply | Threaded
Open this post in threaded view
|

Re: Dandelion not flushing response buffer until complete

Thibault Duchateau
Administrator
This post was updated on .
In reply to this post by tom
Hi Tom,

Could you please give a try to the v1.0.1-SNAPSHOT?

Regards,
Thibault
tom
Reply | Threaded
Open this post in threaded view
|

Re: Dandelion not flushing response buffer until complete

tom
Hi again,

With a quick test it looks like this is working now as expected.  Thanks again for the quick response!

Tom
Reply | Threaded
Open this post in threaded view
|

Re: Dandelion not flushing response buffer until complete

Thibault Duchateau
Administrator
Awesome. Thanks!