REST webservice returning lot of data crushing

[gioppoluca]

I'm trying to implement a REST service (through TEIID Designer) from a virtual procedure that return 60.000 rows.

Apart from paginating the results the problem is that I get this error:

09:51:20,254 WARN  [org.jboss.resteasy.core.SynchronousDispatcher] (http--0.0.0.0-8080-1)  Failed executing GET /AlberataView/Alberate: org.jboss.rest

easy.spi.WriterException: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error

        at org.jboss.resteasy.core.ServerResponse.writeTo(ServerResponse.java:262) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.core.SynchronousDispatcher.writeJaxrsResponse(SynchronousDispatcher.java:585) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:506) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Fin

al.jar:]

        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]

        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.

1.1.Final]

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_35]

Caused by: ClientAbortException:  java.net.SocketException: Connection reset by peer: socket write error

        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:403) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:354) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:426) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:415) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:83) [jbossweb-7.0.13.Final.jar:]

        at org.jboss.resteasy.util.CommitHeaderOutputStream.write(CommitHeaderOutputStream.java:45) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.plugins.providers.StringTextStar.writeTo(StringTextStar.java:60) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.plugins.providers.StringTextStar.writeTo(StringTextStar.java:20) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proceed(MessageBodyWriterContextImpl.java:117) [resteasy-jaxrs-2.3.2.Fina

l.jar:]

        at org.jboss.resteasy.plugins.interceptors.encoding.GZIPEncodingInterceptor.write(GZIPEncodingInterceptor.java:100) [resteasy-jaxrs-2.3.2.Fina

l.jar:]

        at org.jboss.resteasy.core.interception.MessageBodyWriterContextImpl.proceed(MessageBodyWriterContextImpl.java:123) [resteasy-jaxrs-2.3.2.Fina

l.jar:]

        at org.jboss.resteasy.core.ServerResponse.writeTo(ServerResponse.java:250) [resteasy-jaxrs-2.3.2.Final.jar:]

        ... 20 more

Caused by: java.net.SocketException: Connection reset by peer: socket write error

        at java.net.SocketOutputStream.socketWrite0(Native Method) [rt.jar:1.6.0_35]

        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) [rt.jar:1.6.0_35]

        at java.net.SocketOutputStream.write(SocketOutputStream.java:136) [rt.jar:1.6.0_35]

        at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:559) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.Response.doWrite(Response.java:594) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:398) [jbossweb-7.0.13.Final.jar:]

        ... 32 more

09:51:20,424 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/AlbertataHackVDB].[RestDataservice]] (http--0.0.0.0-8080-1)  S

ervlet.service() for servlet RestDataservice threw exception: java.lang.IllegalStateException

        at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:408) [jbossweb-7.0.13.Final.jar:]

        at org.jboss.resteasy.plugins.server.servlet.HttpServletResponseWrapper.sendError(HttpServletResponseWrapper.java:71) [resteasy-jaxrs-2.3.2.Fi

nal.jar:]

        at org.jboss.resteasy.core.SynchronousDispatcher.handleFailure(SynchronousDispatcher.java:265) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.core.SynchronousDispatcher.handleWriterException(SynchronousDispatcher.java:393) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.core.SynchronousDispatcher.handleException(SynchronousDispatcher.java:219) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.core.SynchronousDispatcher.handleWriteResponseException(SynchronousDispatcher.java:203) [resteasy-jaxrs-2.3.2.Final.jar:

]

        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:510) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Fin

al.jar:]

        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]

        at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]

        at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.

1.1.Final]

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

        at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_35]

Mozilla has a timeout of 115 seconds, I do not know where to set the keepalive in jboss, but I do not know if that could be the problem.

If I access the VDB from squirrel I get the whole dataset in 3 seconds so there should not be a big delay in the query time, it could be an out of memory in the REST component, but I do not know where to look for the error.

Any hints?

Luca

[gioppoluca]

The partial response that arrives to the browser is 17 MB in size so a big part of the data arrives arrives more or less the same amount of data all the time so there has to be something somewhere that goes out of memory, but where?

[gioppoluca]

If I create a page with the link to the API and invoke with "save as" I manage to get the complete XML file so it has to do with some problem on the browser.

Maybe some limitation on the max page size, never occurred to me.

Luca

[rareddy]

Luca,

I do not think there is any paging technique used in the REST framework, so you may be right that the engine is putting all the resultset at one time into the response and running into OOM. May be you can turn on the RESTEasy logging up and see any more logs.

OData based REST framework does have paging built in, where you can specify the page sizes.

Ramesh..

[gioppoluca]

Ramesh,

indeed the Rest framework does not implement the pagination since is a feature that should be implemented by the API design (that is why I was trying to implement it using a procedure paramenter in the LIMIT construct https://community.jboss.org/message/829243#829243 ) there should be the possibility to define your own URI parameter to use as LIMIT and offset and the API designer should decide how to implement it.

So I consider correct that the framework does not offer such a feature (even if additional features are always welcome )

In any case since if I save the result of the call as a file and not use the simple GET in the browser I manage to get the correct answer we can conclude that browsers XML parser are not able to correctly parse large XML and break at some time either for some OOM or parser exception that closes the HTTP connection so in JBoss I get the HttpServletResponseWrapper.sendError.

This time no bug on the JBoss part, but on the browser (both IE and Firefox).

Luca

[markaddleman]

Hi Luca -

You may have already considered this but, in case you haven't: 

You can create a materialized view that is backed by your REST service.  The pagination would work against the materialized view without making extraneous calls through REST.  If materialized views don't offer the scoping that you'd like (perhaps due to security concerns), you can use result set caching and achieve a similar effect. 

[gioppoluca]

Thanks Mark,

the view is in fact already amaterialized one.

Otherwise performances would be terrible

Luca