why Wildfly is using high memory when no one uses it

[hchenkwi]

Hi,

I'm using Wildfly 10.0.0.Final on Solaris.  I'm using G1 garbage collector now.

By monitoring the memory usage, I found that when nobody is using the system, the memory usage can go up to 5G and then come down to 1G, it is a cycle about every few minutes. It seems that something is running on the system, make memory high, then when GC cleaned, memory comes back. at this time no one is using Wildfly.

I did threaddump to compare when memory usage low and memory usage high, the threads are pretty much the same.

I did heapdump to compare memory usage low and memory usage high, it seems that the top biggest objects are pretty much the same, but when memory usage is low, the object count is much less than when memory usage is high. I don't know how to find out what are these extra objects, and what process is creating so many objects? why create these objects? How I can find out these process?

Is this normal? I feel Memory should not go this high when no one is using it.  Is there anything process I should stop? Or is there any garbage collector tuning I need to do?

Thanks, Helen

[ctomc]

What memory are you talking about? system memory? jvm memory? heap size? something else?

Did you check with profiler what is happening.

What are you xms/xmx settings or other GC related ones?

[hchenkwi]

it is JVM memory going high and low. We are using java 8, I used Java Mission Control to monitor the JVM memory.

this is the JVM option I set:

-Xms8192m -Xmx8192m -XX:+UseG1GC -XX:MaxMetaspaceSize=256m -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true

-Djava.util.logging.manager=org.jboss.logmanager.LogManager

-Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.3.Final.jar

I'll try to do profiling today.

[hchenkwi]

I suspect it is because I turned scanner on, because from heapdump, I see a lot of objects are I/O related. I think Wildfly keeps scanning the file system.

I have the following setting in standalone.xml:

<subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">

        <deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-enabled="true" scan-interval="5000" auto-deploy-zipped="false" auto-deploy-exploded="false" auto-deploy-xml="false" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>

        </subsystem>

If I set scan-enabled="false", then applications won't be deployed when I start Wildfly.

Is there any work around like I turn off scan, but application can be deployed when Wildfly is restarted?

Thanks,

Helen

[hchenkwi]

I turned off scanner, and added deployment from admin console. memory behaves the same as before. I'll try to do profile.

[ctomc]

-Xms8192m -Xmx8192m -XX:+UseG1GC -XX:MaxMetaspaceSize=256m

What are you saying with this setting to JVM is that it should always utilize 8GB of RAM (+ metaspace + some overhead)

That is why you always have high memory.

you could consider setting Xms to 4G for example, which would than free memory when not needed to use only 4gb.

if you want to see "real" memory usage of WildFly and your application in it, you should look with jmx or via some memory profiler like jprofiler or YourKit, or even some jvm tools that do that.

[hchenkwi]

I get impression that to improve performance, set Xms and Xmx to be the same will reduce overhead of heap memory allocation time.

So if I set Xms to be 4G, and memory is used to 7G, maybe at this time heapsize is 8G already.  Then come to quite time, when no one is using it and objects are garbage collected, the heap size will be reduced to 4G? or heap size will remain as 8G?  If heap size does not reduce to 4G, I will still see memory high when nobody uses it. is it right?

Thanks, Helen

[ctomc]

Yes it is in general recommended to have xms and xmx set to same value, especially for high performance applications.

but it is all up to application itself, there is no real silver bullet on what works best for given scenario.

yes if you have Xms smaller, it would eventually also compact the heap and use less memory.

but that doesn't happen instantly, memory usage must be low for some time, exact details are up to your jvm impl and GC used.

so you should read docs of said jvm/gc to see what fine tuning you can use.

[hchenkwi]

I used yourkit did profiling. One very strange thing is: before I collect snapshot, the memory was up and down bwetween 1G to 5G. after I collected snapshot, memory usage stayed around 1G, up and down a little bit, it didn't go up to 5G any more.  But after I remove yourkit, restart Wildfly, memory behaves the old way up and down again.

Anyway I did profiling. I found there are a lot of small objects allocated (the object allocation rate can be 300/sec) which makes memory high; and they get garbage collected and memory went low. when memory is low, the object count can be around 8M; when memory is high, the object count can be 96M.  The class are all low level java classes, I couldn't figure out what they really mean, I suspect that Wildfly has some process constantly checking something, maybe checking file system to see each file under project? that's the reason there were so many little objects allocated and then recycled.   We do have one project which is in exploded directory structure and has a lot of file there.

If anyone can give me some help to figure out what is running, and how I can disable this process, it will be greatly appreciate.  I listed details below. I can provide more info if needed.

I compared the snapshot between high memory and low memory, the following lists the each class object count difference and size different, "-" means count reduced when memory is low.

class              object-cnt-diff   object-size-diff

-----------------------------------------------------------------------------------------------

byte[]               -21427442(-22%)   -1672115304 -33%

sun.nio.fs.UnixPath           -20,711,710(-22%) -662774,720 -13%

int[]                -6,921,436(-7%)   -467,545,160 -9%

java.nio.file.attribute.FileTime         -6,903,903(-7%)   -220,924,896 -4%

sun.nio.fs.UnixFileAttributes          -6,903,903(-7%)   -883699,584 -17%

sun.nio.fs.UnixFileAttributes$UnixAsBasicFileAttributes   -6,903,903(-7%)   -110,462,448 -2%

sun.nio.fs.UnixFileAttributeViews$Basic       -6,903,903(-7%)   -165,693,672 -3%

java.nio.file.LinkOption[]          -6,903,902(-7%)   -165,693,648 -3%

java.util.HashMap$EntryIterator         -346,560(-0%)    -13,862,400 -0%

java.util.concurrent.locks.ReentrantReadWriteLock     -339,531(-0%)  -8,148,744 -0%

java.util.concurrent.locks.ReentrantReadWriteLock$FairSync  -339,531(-0%)  -16,297,488 -0%

java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock  -339,531(-0%)  -5,432,496 -0%

java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter  -339,531(-0%) -5,432,496 -0%

java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock -339,531(-0%)  -5,432,496 -0%

sun.nio.fs.UnixDirectoryStream         -339,531(-0%)  -13,581,240 -0%

sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator   -339,531(-0%)  -10,864,992 -0%

sun.nio.fs.UnixSecureDirectoryStream       -339,531(-0%)  -8,148,744 -0%

java.lang.Object[]            -190,815(-0%)  -6,209,880 -0%

char[]               -116,044(-0%)  -10,226,680 -0%

and the following is the allocation stack for byte[]:

---------------------------------------------------------------------

<All threads>  1,350,016 100 % 105,243,048 100 %

java.lang.Thread.run()  1,350,009 99 % 105,242,664 99 %

  ThreadPoolExecutor.java:1142  io.undertow.server.HttpServerExchange$1.run() 696 0% 80,544 0%

    HttpServerExchange.java:793  io.undertow.server.Connectors.executeRootHandler(HttpHandler, HttpServerExchange)  696 0% 80,544 0%

     Connectors.java:202  io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(HttpServerExchange)

696 0% 80,544 0%

      AuthenticationCallHandler.java:50  io.undertow.security.impl.SecurityContextImpl.authenticate()  656 0% 29,352 0%

      AuthenticationCallHandler.java:52  org.jboss.as.domain.http.server.DomainApiCheckHandler.handleRequest(HttpServerExchange)

39 0%  51,168 0%

      AuthenticationCallHandler.java:55  io.undertow.server.HttpServerExchange.endExchange()

1 0% 24 0%

   com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run()  7  0% 384 0%

   PeriodicRecovery.java:375 com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal()  7 0% 384 0%

    PeriodicRecovery.java:747  com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkFirstPass()

2 0% 48 0%

    PeriodicRecovery.java:747  com.arjuna.ats.internal.jta.recovery.arjunacore.CommitMarkableResourceRecordRecoveryModule.periodicWorkFirstPass()

2 0 % 104 0 %

    PeriodicRecovery.java:793  com.arjuna.ats.internal.jta.recovery.arjunacore.CommitMarkableResourceRecordRecoveryModule.periodicWorkSecondPass()

2 0% 152 0%

    PeriodicRecovery.java:747  com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass()

Thanks, Helen

[hchenkwi]

I did profiling on the web apps using yourkit tool, the following is the output of calltree stack. If you copy the following html block and paste to a file, use browser you could see the calltree.

It shows that the code from undertow holds big count of objects and used a lot of resources, which caused memory high.

After I took snapshot with yourkit, the memory stayed low. I can reproduce this. It is very strange, suppose yourkit should make Wildfly memory i=usage higher, not lower.  But this tells me there may exist some switch that make Wildfly not use so much memory. But I don't know what is this switch, and why Wildfly has to use so much memory.

<html>

  <head>

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">

    <title>Allocation call tree (all threads together)</title>

    <STYLE type="text/css">

      body{

        margin-top: 20px;

        margin-left: 50px;

        margin-right: 50px;

        background: white;

      }

      table{

        border: 0px;

        border-spacing: 0px;

      }

      td{

        padding-left: 3px;

        padding-right: 3px;

      }

      .tree-cell{

        vertical-align: middle;

      }

      .header{

        text-align: center;

        padding-top: 3px;

        padding-left: 10px;

        padding-bottom: 3px;

        padding-right: 10px;

        font-weight: bold;

        background-color: #DDDDDD;

      }

      .value{

        text-align: right;

        padding-left: 10px;

        padding-right: 10px;

      }

      .percents{

        font-size: 80%;

        color: gray;

        text-align: right;

        padding-left: 10px;

        padding-right: 10px;

      }

      .footer{

        font-size: 80%;

        text-align: right;

        padding-top: 20px;

      }

      .odd {background-color: #F2F2F2;}

    </STYLE>

</head>

  <body>

  <div><h2>Allocation call tree (all threads together)</h2></div>

  <p>

    <table width="100%">

<tr><td class="header" width="100%"> Call Tree </nobr></td><td class="header" colspan="2"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/desc.png"/> Objects </nobr></td><td class="header" colspan="2"> Size </nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/leaf-with-sibling.gif"/></td><td width="100%"><nobr><span class="tree-cell">&lt;Objects without allocation information&gt;</span></nobr></td></tr></table></td><td class="value"><nobr>90,623,310</nobr></td><td class="percents"><nobr></nobr></td><td class="value"><nobr>4,776,067,264</nobr></td><td class="percents"><nobr></nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><span class="tree-cell">&lt;All threads&gt;</span></nobr></td></tr></table></td><td class="value"><nobr>5,586,637</nobr></td><td class="percents"><nobr>100 %</nobr></td><td class="value"><nobr>282,133,712</nobr></td><td class="percents"><nobr>100 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">java.lang</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">Thread</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>5,585,940</nobr></td><td class="percents"><nobr>99 %</nobr></td><td class="value"><nobr>282,104,056</nobr></td><td class="percents"><nobr>99 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">ThreadPoolExecutor.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">1142</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange$1</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>61,933</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,471,864</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">HttpServerExchange.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">793</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">Connectors</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">executeRootHandler</span></font></b><span class="tree-cell">(HttpHandler, HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>61,933</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,471,864</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">Connectors.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">202</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.handlers</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">AuthenticationCallHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>61,933</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,471,864</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">AuthenticationCallHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">52</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.as.domain.http.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">DomainApiCheckHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,751</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,299,456</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainApiCheckHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">95</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server.handlers</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">BlockingHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,751</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,299,456</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">BlockingHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">56</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.as.domain.http.server.security</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SubjectDoAsHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,744</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,299,232</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SubjectDoAsHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">72</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server.handlers.encoding</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">EncodingHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,628</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,295,256</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">EncodingHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">66</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.as.domain.http.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">DomainApiHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,628</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,295,256</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainUtil.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">70</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.io</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">BlockingSenderImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">send</span></font></b><span class="tree-cell">(ByteBuffer)</span></nobr></td></tr></table></td><td class="value"><nobr>166</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>5,504</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainUtil.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">69</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.util</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HeaderMap</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">put</span></font></b><span class="tree-cell">(HttpString, long)</span></nobr></td></tr></table></td><td class="value"><nobr>42</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>1,072</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainUtil.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">70</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">getResponseSender</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>17</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>744</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainUtil.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">58</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.util</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HeaderMap</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">put</span></font></b><span class="tree-cell">(HttpString, String)</span></nobr></td></tr></table></td><td class="value"><nobr>12</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>288</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">Common.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">109</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.io</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">BlockingSenderImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">send</span></font></b><span class="tree-cell">(String, IoCallback)</span></nobr></td></tr></table></td><td class="value"><nobr>7</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>272</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">Common.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">109</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">getResponseSender</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>1</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>48</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">BlockingHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">52</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">startBlocking</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>7</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>224</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">AuthenticationCallHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">50</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">authenticate</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,175</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>172,032</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">89</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">authTransition</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,175</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>172,032</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">96</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">attemptAuthentication</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,166</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>170,944</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">121</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl$AuthAttempter</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">access$100</span></font></b><span class="tree-cell">(SecurityContextImpl$AuthAttempter)</span></nobr></td></tr></table></td><td class="value"><nobr>3,148</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>170,512</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">219</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl$AuthAttempter</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">transition</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,148</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>170,512</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">250</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl$AuthAttempter</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">transition</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,123</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>169,944</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">233</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.as.domain.http.server.security</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">AuthenticationMechanismWrapper</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">authenticate</span></font></b><span class="tree-cell">(HttpServerExchange, SecurityContext)</span></nobr></td></tr></table></td><td class="value"><nobr>25</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>568</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/leaf.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img2.png"/> <font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">121</span></font> </nobr></td></tr></table></td><td class="value"><nobr>18</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>432</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">104</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">authTransition</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>9</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>1,088</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">AuthenticationCallHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">55</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">endExchange</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>7</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>376</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">Executors.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">511</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">org.infinispan.expiration.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">ExpirationManagerImpl$ScheduledTask</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>144</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">ThreadPoolExecutor.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">1142</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">org.infinispan.persistence.file</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SingleFileStore$3</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>72</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">org.xnio.nio</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">WorkerThread</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>561</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>20,320</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">com.arjuna.ats.internal.arjuna.recovery</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">PeriodicRecovery</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>64</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>5,880</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr><td width="100%"></td><td class="value"><nobr>48</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>2,576</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/leaf.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.threads</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">JBossThread</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>24</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>880</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>

</table>

  </p>

  <div class="footer">Generated by <a href="http://www.yourkit.com">YourKit Java Profiler 2015 build 15086</a>    October 4, 2016 07:48:22 PM</div>

  </body>

</html>

[bcrowell2015]

Helen,

What JDK version are you using?  Can you provide specific minor point version with update level (e.g. 1.8.0_XX)?

How many applications are deployed to your Wildfly server?

I have a similar situation, and I believe it might be related to a Java NIO memory leak.

[hchenkwi]

Hi William,

The following is what we are using, and we are using Oracle certified java version.

java version "1.8.0_66"

Java(TM) SE Runtime Environment (build 1.8.0_66-b17)

Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)

we only have 4 applications deployed, but we do have a lot of files under each application.

From what I monitored, it seems the allocated memory can be Garbage collected, it is just allocating so much memory, and it will definitely be a problem when used in production.  But I'm not sure how to monitor the native memory leak, maybe the heap memory is garbage collected, but the native memory is still there.

In your case, what is the symptom? how do you monitor the memory usage for both heap memory and native memory usage?  

I saw an article talking about NIO memory leak: Fixing Java's ByteBuffer native memory "leak" (evanjones.ca)

Do you have work around fixing this issue?

Thanks a lot for your help,

Helen