[Solved] Wildfly 8 Final - 100% cpu usage, no l...| JBoss.org Content Archive (Read Only)

1 2 3 Previous Next 37 Replies Latest reply on May 15, 2014 5:54 AM by ctomc Go to original post

15. Re: Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

lbonco Apr 15, 2014 3:21 PM (in response to sfcoy)

Hi to all,

I applied the following patch to undertow 1.0.0:

commit b07eab7b2cead3580be513aaa405b0357ac7ec05
Author: Stuart Douglas <stuart.w.douglas@gmail.com>
Date: Tue Mar 11 11:17:36 2014 +1100

    UNDERTOW-200 Fix issue with charset encoder in ServletPrintWriter

After some period without problem, today I get this never endign loop:

"default task-14" prio=10 tid=0xc4574000 nid=0x7d43 runnable [0xca35a000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.cs.UTF_8$Encoder.encodeBufferLoop(UTF_8.java:577)
        at sun.nio.cs.UTF_8$Encoder.encodeLoop(UTF_8.java:621)
        at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:561)
        at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:84)
        at io.undertow.servlet.spec.ServletPrintWriter.write(ServletPrintWriter.java:111)
        at io.undertow.servlet.spec.ServletPrintWriterDelegate.write(ServletPrintWriterDelegate.java:96)
        at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:120)
        at org.apache.jasper.runtime.JspWriterImpl.flush(JspWriterImpl.java:159)
        at java.io.PrintWriter.flush(PrintWriter.java:320)
        - locked <0xde762690> (a org.apache.jasper.runtime.JspWriterImpl)

I think that the patch does not solve the problem.

Thanks,
Luca
Actions
16. Re: Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

ctomc Apr 15, 2014 6:04 PM (in response to lbonco)

Did you try with WildFly 8.1.0.CR1?
Actions
17. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

lbonco Apr 16, 2014 2:31 PM (in response to ctomc)

Hi,

I checked the commit log on io.undertow.servlet.spec.ServletPrintWriter and I see no more commit from b07eab7b on undertow. Do you think that the test on WildFly 8.1.0.CR1 is usefull?

Luca
Actions
18. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

premudriy Apr 17, 2014 5:03 PM (in response to premudriy)
The issue is still happening with the very latest Wildfly snapshot. I've also read the thread about "Idle Remover Thread", but in my data source config I'm not using <use-strict-min>true</use-strict-min>, so it should be set to 'false', which is default.

Attached are thread dumps from two different machines. Both dumps are a sequence of 3, few seconds apart. One set is from machine where 1 of 2 cores is at 100% and one set of dumps is from a machine where both cores are loaded 100%.

1of2cores100percent.zip 16.9 KB

bothCores100percent.zip 17.1 KB
Actions
19. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

premudriy Apr 18, 2014 4:11 PM (in response to premudriy)

You know, guys, I agree with Luca now. Seems like it's the same problem with ServletPrintWriter and the problem is still not fixed. Today I used a different tool, jconsole with "Top Threads" plugin, and here's what I get:

As you can see those 2 threads load CPU completely and they are both from ServletPrintWriter running the encodeLoop.
Actions
20. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

swd847 Apr 21, 2014 7:09 PM (in response to premudriy)

I have applied a fix to Undertow upstream that should resolve this (I hope). It would be great if you could test it out.

Also do you know what sort of input and charset actually causes the problem? It would be great if you could provide a reproducer so we can add more extensive tests.
Actions
21. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

premudriy Apr 22, 2014 7:00 PM (in response to swd847)

Thank you, Stuart! Do you think today's daily snapshot build of Wildfly will have the fixes you have made? If so I'll try it out.

As for for the input and charset: it's hard to say. Servlets return things like json and xml mostly. There are some Cyrillic characters/text as well, but we try to use UTF-8 everywhere in our project. It would be difficult to isolate a use case for replicating this issue because there are well over 300 servlets.
Actions
22. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

lbonco Apr 23, 2014 5:12 AM (in response to swd847)

Hi,

I cherry-pick c853587 on undertow branch 1.0.0. I installed the update jar and now it seems OK.

Luca
Actions
23. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

premudriy Apr 23, 2014 4:20 PM (in response to lbonco)

Somehow I could not compile the undertow to test. I checked out the code from here: undertow-io/undertow at 1.0.x · GitHub then tried to compile with "mvn clean package" command and it failed some tests as it compiled. What am I doing wrong? Luca, how did you compile it?
Actions
24. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

ctomc Apr 23, 2014 4:54 PM (in response to premudriy)

if you just need binaries, do mvn clean install -DskipTests
1 of 1 people found this helpful
Actions
25. Re: Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

lbonco Apr 23, 2014 5:16 PM (in response to premudriy)

Hi,
I used something like:

on undertow git source:

git branch test1 1.0.0.Final
git checkout test1
git cherry-pick 393d627f05a7f5d2948b5cb33a73ed35b85424ce
git cherry-pick e3ad9bccd78043f1ecc0f65dc6084d9b01681808
mvn clean install -DskipTests

on wildfly git source

git checkout 8.0.0.Final
mvn clean install -DskipTests
Actions
26. Re: Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

swd847 Apr 23, 2014 6:53 PM (in response to lbonco)

Are you using JDK8? There was a build problem with JDK8 that I fixed a few hours ago.
Actions
27. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

premudriy Apr 24, 2014 3:52 PM (in response to swd847)

I'm using the latest JKD 7.

Also I see that unfortunately CPU is still 100%. Here's what I did for this last test:
-checked out the code from this location: https://github.com/undertow-io/undertow/tree/master
-built it with jdk7 (latest) via "mvn clean install -DskipTests"
-copied and replaced
          <jboss>\modules\system\layers\base\io\undertow\core\main\undertow-core-1.0.0.Final.jar
      with
          .m2\repository\io\undertow\undertow-core\1.1.0.Beta1-SNAPSHOT\undertow-core-1.1.0.Beta1-SNAPSHOT.jar
     on each jboss machine.

Tell me if this is not how it was supposed to be done.

Also I just compiled whatever was in master for this test. I could not complete the cherry-pick like Luca did. Git says:
"fatal: bad object 393d627f05a7f5d2948b5cb33a73ed35b85424ce"
and
"fatal: bad object e3ad9bccd78043f1ecc0f65dc6084d9b01681808"
when I run:
git cherry-pick 393d627f05a7f5d2948b5cb33a73ed35b85424ce
git cherry-pick e3ad9bccd78043f1ecc0f65dc6084d9b01681808

Am I checking out from wrong location?
Actions
28. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

ctomc Apr 24, 2014 4:09 PM (in response to premudriy)

Just take WildFly 8.1.0.CR1 which has newer xnio & newer undertow, then go from there.

also you should be building 1.0.x branch of undertow not master https://github.com/undertow-io/undertow/tree/1.0.x

1.0.x is bugfix branch for 8.1.x WildFly
Actions
29. Re: Wildfly 8 Final - 100% cpu usage, no load, either due to Undertow's ServletPriterWriter or NIO CharsetEncoder

lbonco Apr 24, 2014 6:32 PM (in response to premudriy)

Hi,
you should switch to branch 1.0.0.Final:

git branch test1 1.0.0.Final
git checkout test1

and then

git cherry-pick 393d627f05a7f5d2948b5cb33a73ed35b85424ce
git cherry-pick e3ad9bccd78043f1ecc0f65dc6084d9b01681808

after compiling this branch you will get a undertow-core-1.0.0.Final.jar pathced jar.
Actions

1 2 3 Previous Next

Go to original post