10 Replies Latest reply on Nov 6, 2014 9:53 AM by raghav prakasam Branched to a new discussion.

    Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,

    David Rodriguez Newbie

      We are having problems with our Wildfly 8.1.0 installations.  In the server.log file we are seeing these errors quite frequently:

       

      2014-08-06 12:04:54,349 ERROR [io.undertow.request] (default task-81) Blocking request failed HttpServerExchange{ GET /Finance/WEB-INF/jsp/report/viewReport.jsp}: java.nio.BufferOverflowException

        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357) [rt.jar:1.7.0_60]

        at java.nio.ByteBuffer.put(ByteBuffer.java:832) [rt.jar:1.7.0_60]

        at io.undertow.util.HttpString.appendTo(HttpString.java:204)

        at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)

        at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)

        at io.undertow.conduits.ChunkedStreamSinkConduit.flush(ChunkedStreamSinkConduit.java:273)

        at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

        at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)

        at org.xnio.channels.Channels.flushBlocking(Channels.java:63) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

        at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:502)

        at io.undertow.servlet.spec.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:484)

        at io.undertow.servlet.spec.ServletPrintWriter.flush(ServletPrintWriter.java:54)

        at io.undertow.servlet.spec.ServletPrintWriterDelegate.flush(ServletPrintWriterDelegate.java:76)

        at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:443)

        at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:525)

        at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:287)

        at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)

        at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73)

        at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146)

        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177)

        at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_60]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_60]

        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_60]

       

      Followed by:

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) Exception in thread "default task-81" java.nio.BufferOverflowException

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:357)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at java.nio.ByteBuffer.put(ByteBuffer.java:832)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.util.HttpString.appendTo(HttpString.java:204)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.server.protocol.http.HttpResponseConduit.processWrite(HttpResponseConduit.java:150)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.server.protocol.http.HttpResponseConduit.flush(HttpResponseConduit.java:629)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.conduits.ChunkedStreamSinkConduit.flush(ChunkedStreamSinkConduit.java:273)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at org.xnio.conduits.ConduitStreamSinkChannel.flush(ConduitStreamSinkChannel.java:162)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.channels.DetachableStreamSinkChannel.flush(DetachableStreamSinkChannel.java:100)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at org.xnio.channels.Channels.flushBlocking(Channels.java:63)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:502)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.servlet.spec.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:484)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.servlet.spec.ServletPrintWriter.flush(ServletPrintWriter.java:54)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.servlet.spec.ServletPrintWriterDelegate.flush(ServletPrintWriterDelegate.java:76)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:443)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.servlet.core.ServletBlockingHttpExchange.close(ServletBlockingHttpExchange.java:72)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.server.HttpServerExchange.endExchange(HttpServerExchange.java:1404)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.server.Connectors.executeRootHandler(Connectors.java:201)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

      2014-08-06 12:04:54,349 ERROR [stderr] (default task-81) at java.lang.Thread.run(Thread.java:745)

       

      We are using the default IO settings:

            <subsystem xmlns="urn:jboss:domain:io:1.1">

                  <worker name="default"/>

                  <buffer-pool name="default"/>

              </subsystem>

       

      Do we need to change our setup somewhere?  We are having lots of server high CPU usage, but not sure if they are related.

        • 2. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
          Frank Langelage Master

          Latest build on WildFly-latest-master [Jenkins] is from August 6th.

          And this not contain the latest undertow upgrade you probably were referring to.

          • 3. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
            Tomaz Cerar Master

            Damn it looks like that recent upgrade of internal jenkins instance caused problems with publishing new builds to ci.jboss.org...

             

            We will have new CI soon(HW just arrived), that will completely overcome this problem, but for time being i pinged our Jenkins guys to take a look.

            • 4. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
              David Rodriguez Newbie

              Thanks for the response, we will look for the new release to install when it becomes available.

               

              However, we have clients using Wildfly 8.1.0 now, and if there is some workaround or configuration change that could help, we could use that right away.

              • 5. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
                Charles Ventura Newbie

                I have the EXACT same problem.  I also have a problem with my database throwing ASYNC_NETWORK_IO errors which is causing all my other queries to back up and be blocked.  From everything I've read it's pointing to my application not being able to process the result set fast enough.  Further research leads me to believe it has to do with the bufferpools & since my server log is littered with those nio.bufferoverflow errors I think it's related.  I'm using wildfly 8.1.0.Final and when I try to set the Core -> IO -> Bufferpools -> Use Direct Bufferpools - it won't accept "true" or "yes" as values.  ON Wildfly 8.0.0 it's just a checkbox and not a text box. From my 8.1.0.Final server if I click on Documentation it takes me to 8.0 documentation - which is different. 8.0 uses undertow 1.0.  Wildfly 8.1.0.Final uses undertow 1.1.  Every piece of documentation I find always has 1.0 documentation.  I don't understand this - but some of the things I've read tell me that the bufferoverflow error has to do with the browser navigating away from a page to another page before the first one comes up.  I have been able to reproduce it that way - but I think it's more of an effect that a cause.
                Anyway - this is absolutely killing me - this problem never came up in testing and now we're in production and the only way to fix it to reboot wildfly to clear all the connections.

                 

                • 6. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
                  Tomaz Cerar Master

                  Only thing that changed in 1.0 --> 1.1 undertow schema used in 8.1 is how default values are configured.

                  We needed to bump the schema version so we can do proper upgrade of configuration.

                  everything else is still the same as it was in 1.0 schema.

                   

                  attribute direct-buffers is a boolean so as such it takes true or false values.

                   

                  In any case if you system has more 64mb of system ram (not jvm assigned) direct buffers will be enabled by default in 8.1

                  see wildfly/BufferPoolResourceDefinition.java at 8.x · wildfly/wildfly · GitHub for more details on how defaults for buffer pools as calculated.

                   

                  Said that, any BufferOverflow exception is probably a bug in server impl and not a config error.

                  That is why I suggested if you guys can try with newer version of wildfly, or at least 8.1 with newer undertow.

                  • 7. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
                    Tomaz Cerar Master

                    Guys,

                     

                    I prepared zip with updated modules for 8.1.0 that includes new undertow and new xnio + few fixes in undertow subsystem.

                     

                    you can grab it here https://dl.dropboxusercontent.com/u/16104533/8.1.0.Final-undertow-1.1.0.Beta6.zip

                    all you need to do it to extract it in WILDFLY_HOME/modules/system/layers/base

                    and you will get new undertow 1.1.0.Beta6 and xnio 3.3.0.Beta1.

                    This is built from my undertow-8.x branch ctomc/wildfly at undertow-8.x · GitHub where I am working on back porting fixes from master to 8.x branch.

                     

                    It would be great If anyone of you have a chance to test latest undertow to see if problems you have still persist.

                    1 of 1 people found this helpful
                    • 8. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
                      David Rodriguez Newbie

                      Thanks Tomaz, sorry I did not see reply until today.   Did not get notified.

                       

                      We tried your fix and it seems to improve things on our Windows 2008 Server.  We had issues getting it installed on Windows 2012 server.  I will attach the errors soon.  I am waiting on our testers to get that to me.

                       

                      David R

                      • 9. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
                        David Rodriguez Newbie

                        This is the error we get on Windows 2012 Server, Wildfly 8.1.0 Final ( With your changes), Java 1.7.60

                        2014-08-26 16:16:05,109 ERROR [io.undertow.request] (default task-1) UT005023: Exception handling request to /CAS/WEB-INF/view/jsp/txeis/ui/casLoginView.jsp: org.apache.jasper.JasperException: java.lang.IndexOutOfBoundsException

                          at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:410) [jastow-1.0.0.Final.jar:1.0.0.Final]

                          at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:326) [jastow-1.0.0.Final.jar:1.0.0.Final]

                          at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:259) [jastow-1.0.0.Final.jar:1.0.0.Final]

                          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]

                          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85) [undertow-servlet-1.1.0.Beta6.jar:1.1.0.Beta6]

                        Caused by: java.lang.IndexOutOfBoundsException

                          at java.nio.CharBuffer.wrap(CharBuffer.java:371) [rt.jar:1.7.0_60]

                          at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:231) [undertow-servlet-1.1.0.Beta6.jar:1.1.0.Beta6]

                          at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:114) [undertow-servlet-1.1.0.Beta6.jar:1.1.0.Beta6]

                          at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:120) [jastow-1.0.0.Final.jar:1.0.0.Final]

                          at org.apache.jasper.runtime.PageContextImpl.release(PageContextImpl.java:191) [jastow-1.0.0.Final.jar:1.0.0.Final]

                          at org.apache.jasper.runtime.JspFactoryImpl.internalReleasePageContext(JspFactoryImpl.java:115) [jastow-1.0.0.Final.jar:1.0.0.Final]

                          at org.apache.jasper.runtime.JspFactoryImpl.releasePageContext(JspFactoryImpl.java:74) [jastow-1.0.0.Final.jar:1.0.0.Final]

                          at org.apache.jsp.WEB_002dINF.view.jsp.txeis.ui.casLoginView_jsp._jspService(casLoginView_jsp.java:326)

                          at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69) [jastow-1.0.0.Final.jar:1.0.0.Final]

                          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [jboss-servlet-api_3.1_spec-1.0.0.Final.jar:1.0.0.Final]

                          at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:366) [jastow-1.0.0.Final.jar:1.0.0.Final]

                         

                        Also, we are seeing users getting kicked out.  We are using CAS 3.1 and I see references to cas server 3.3. jar files, do we need to upgrade to higher version?

                         

                        David R.

                        • 10. Re: Problem with Wildfly 8.1.0, Blocking request failed, BufferOverflowException,
                          raghav prakasam Newbie

                          Hi Tomaz Cerar,

                           

                          I have tried your fix but still am facing the same error while giving concurrent hits of 5 or more.

                          Please suggest me the fix as am using wildfly 8.1.0 final version.

                           

                          12:04:52,874 ERROR [io.undertow.request] (default task-27) Undertow request failed HttpServerExchange{ POST /**************************************}: java.lang.RuntimeException: java.io.IOException: Broken pipe

                                  at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:562)

                                  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:307)

                                  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:246)

                                  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:75)

                                  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:165)

                                  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197)

                                  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:737)

                                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_65]

                                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_65]

                                  at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_65]

                          Caused by: java.io.IOException: Broken pipe

                                  at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) [rt.jar:1.7.0_65]

                                  at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) [rt.jar:1.7.0_65]

                                  at sun.nio.ch.IOUtil.write(IOUtil.java:148) [rt.jar:1.7.0_65]

                                  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:524) [rt.jar:1.7.0_65]

                                  at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:159) [xnio-nio-3.2.2.Final.jar:3.2.2.Final]

                                  at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:561)

                                  at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:171)

                                  at io.undertow.conduits.ChunkedStreamSinkConduit.writeFinal(ChunkedStreamSinkConduit.java:233)

                                  at org.xnio.conduits.ConduitStreamSinkChannel.writeFinal(ConduitStreamSinkChannel.java:104) [xnio-api-3.2.2.Final.jar:3.2.2.Final]

                                  at io.undertow.channels.DetachableStreamSinkChannel.writeFinal(DetachableStreamSinkChannel.java:190)

                                  at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:564)

                                  at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:598)

                                  at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:477)

                                  at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:560)

                                  ... 9 more