-
1. Re: why Wildfly is using high memory when no one uses it
ctomc Sep 28, 2016 10:56 AM (in response to hchenkwi)What memory are you talking about? system memory? jvm memory? heap size? something else?
Did you check with profiler what is happening.
What are you xms/xmx settings or other GC related ones?
-
2. Re: why Wildfly is using high memory when no one uses it
hchenkwi Sep 28, 2016 11:38 AM (in response to ctomc)it is JVM memory going high and low. We are using java 8, I used Java Mission Control to monitor the JVM memory.
this is the JVM option I set:
-Xms8192m -Xmx8192m -XX:+UseG1GC -XX:MaxMetaspaceSize=256m -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true
-Djava.util.logging.manager=org.jboss.logmanager.LogManager
-Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.3.Final.jar
I'll try to do profiling today.
-
3. Re: why Wildfly is using high memory when no one uses it
hchenkwi Sep 28, 2016 12:29 PM (in response to hchenkwi)I suspect it is because I turned scanner on, because from heapdump, I see a lot of objects are I/O related. I think Wildfly keeps scanning the file system.
I have the following setting in standalone.xml:
<subsystem xmlns="urn:jboss:domain:deployment-scanner:2.0">
<deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-enabled="true" scan-interval="5000" auto-deploy-zipped="false" auto-deploy-exploded="false" auto-deploy-xml="false" runtime-failure-causes-rollback="${jboss.deployment.scanner.rollback.on.failure:false}"/>
</subsystem>
If I set scan-enabled="false", then applications won't be deployed when I start Wildfly.
Is there any work around like I turn off scan, but application can be deployed when Wildfly is restarted?
Thanks,
Helen
-
4. Re: why Wildfly is using high memory when no one uses it
hchenkwi Sep 28, 2016 1:57 PM (in response to hchenkwi)I turned off scanner, and added deployment from admin console. memory behaves the same as before. I'll try to do profile.
-
5. Re: why Wildfly is using high memory when no one uses it
ctomc Sep 28, 2016 2:00 PM (in response to hchenkwi)-Xms8192m -Xmx8192m -XX:+UseG1GC -XX:MaxMetaspaceSize=256m
What are you saying with this setting to JVM is that it should always utilize 8GB of RAM (+ metaspace + some overhead)
That is why you always have high memory.
you could consider setting Xms to 4G for example, which would than free memory when not needed to use only 4gb.
if you want to see "real" memory usage of WildFly and your application in it, you should look with jmx or via some memory profiler like jprofiler or YourKit, or even some jvm tools that do that.
-
6. Re: why Wildfly is using high memory when no one uses it
hchenkwi Sep 28, 2016 2:15 PM (in response to ctomc)I get impression that to improve performance, set Xms and Xmx to be the same will reduce overhead of heap memory allocation time.
So if I set Xms to be 4G, and memory is used to 7G, maybe at this time heapsize is 8G already. Then come to quite time, when no one is using it and objects are garbage collected, the heap size will be reduced to 4G? or heap size will remain as 8G? If heap size does not reduce to 4G, I will still see memory high when nobody uses it. is it right?
Thanks, Helen
-
7. Re: why Wildfly is using high memory when no one uses it
ctomc Sep 28, 2016 2:24 PM (in response to hchenkwi)Yes it is in general recommended to have xms and xmx set to same value, especially for high performance applications.
but it is all up to application itself, there is no real silver bullet on what works best for given scenario.
yes if you have Xms smaller, it would eventually also compact the heap and use less memory.
but that doesn't happen instantly, memory usage must be low for some time, exact details are up to your jvm impl and GC used.
so you should read docs of said jvm/gc to see what fine tuning you can use.
-
8. Re: why Wildfly is using high memory when no one uses it
hchenkwi Oct 1, 2016 5:40 PM (in response to ctomc)I used yourkit did profiling. One very strange thing is: before I collect snapshot, the memory was up and down bwetween 1G to 5G. after I collected snapshot, memory usage stayed around 1G, up and down a little bit, it didn't go up to 5G any more. But after I remove yourkit, restart Wildfly, memory behaves the old way up and down again.
Anyway I did profiling. I found there are a lot of small objects allocated (the object allocation rate can be 300/sec) which makes memory high; and they get garbage collected and memory went low. when memory is low, the object count can be around 8M; when memory is high, the object count can be 96M. The class are all low level java classes, I couldn't figure out what they really mean, I suspect that Wildfly has some process constantly checking something, maybe checking file system to see each file under project? that's the reason there were so many little objects allocated and then recycled. We do have one project which is in exploded directory structure and has a lot of file there.
If anyone can give me some help to figure out what is running, and how I can disable this process, it will be greatly appreciate. I listed details below. I can provide more info if needed.
I compared the snapshot between high memory and low memory, the following lists the each class object count difference and size different, "-" means count reduced when memory is low.
class object-cnt-diff object-size-diff
-----------------------------------------------------------------------------------------------
byte[] -21427442(-22%) -1672115304 -33%
sun.nio.fs.UnixPath -20,711,710(-22%) -662774,720 -13%
int[] -6,921,436(-7%) -467,545,160 -9%
java.nio.file.attribute.FileTime -6,903,903(-7%) -220,924,896 -4%
sun.nio.fs.UnixFileAttributes -6,903,903(-7%) -883699,584 -17%
sun.nio.fs.UnixFileAttributes$UnixAsBasicFileAttributes -6,903,903(-7%) -110,462,448 -2%
sun.nio.fs.UnixFileAttributeViews$Basic -6,903,903(-7%) -165,693,672 -3%
java.nio.file.LinkOption[] -6,903,902(-7%) -165,693,648 -3%
java.util.HashMap$EntryIterator -346,560(-0%) -13,862,400 -0%
java.util.concurrent.locks.ReentrantReadWriteLock -339,531(-0%) -8,148,744 -0%
java.util.concurrent.locks.ReentrantReadWriteLock$FairSync -339,531(-0%) -16,297,488 -0%
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock -339,531(-0%) -5,432,496 -0%
java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter -339,531(-0%) -5,432,496 -0%
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock -339,531(-0%) -5,432,496 -0%
sun.nio.fs.UnixDirectoryStream -339,531(-0%) -13,581,240 -0%
sun.nio.fs.UnixDirectoryStream$UnixDirectoryIterator -339,531(-0%) -10,864,992 -0%
sun.nio.fs.UnixSecureDirectoryStream -339,531(-0%) -8,148,744 -0%
java.lang.Object[] -190,815(-0%) -6,209,880 -0%
char[] -116,044(-0%) -10,226,680 -0%
and the following is the allocation stack for byte[]:
---------------------------------------------------------------------
<All threads> 1,350,016 100 % 105,243,048 100 %
java.lang.Thread.run() 1,350,009 99 % 105,242,664 99 %
ThreadPoolExecutor.java:1142 io.undertow.server.HttpServerExchange$1.run() 696 0% 80,544 0%
HttpServerExchange.java:793 io.undertow.server.Connectors.executeRootHandler(HttpHandler, HttpServerExchange) 696 0% 80,544 0%
Connectors.java:202 io.undertow.security.handlers.AuthenticationCallHandler.handleRequest(HttpServerExchange)
696 0% 80,544 0%
AuthenticationCallHandler.java:50 io.undertow.security.impl.SecurityContextImpl.authenticate() 656 0% 29,352 0%
AuthenticationCallHandler.java:52 org.jboss.as.domain.http.server.DomainApiCheckHandler.handleRequest(HttpServerExchange)
39 0% 51,168 0%
AuthenticationCallHandler.java:55 io.undertow.server.HttpServerExchange.endExchange()
1 0% 24 0%
com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run() 7 0% 384 0%
PeriodicRecovery.java:375 com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal() 7 0% 384 0%
PeriodicRecovery.java:747 com.arjuna.ats.internal.arjuna.recovery.AtomicActionRecoveryModule.periodicWorkFirstPass()
2 0% 48 0%
PeriodicRecovery.java:747 com.arjuna.ats.internal.jta.recovery.arjunacore.CommitMarkableResourceRecordRecoveryModule.periodicWorkFirstPass()
2 0 % 104 0 %
PeriodicRecovery.java:793 com.arjuna.ats.internal.jta.recovery.arjunacore.CommitMarkableResourceRecordRecoveryModule.periodicWorkSecondPass()
2 0% 152 0%
PeriodicRecovery.java:747 com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass()
Thanks, Helen
-
9. Re: why Wildfly is using high memory when no one uses it
hchenkwi Oct 4, 2016 8:01 PM (in response to hchenkwi)I did profiling on the web apps using yourkit tool, the following is the output of calltree stack. If you copy the following html block and paste to a file, use browser you could see the calltree.
It shows that the code from undertow holds big count of objects and used a lot of resources, which caused memory high.
After I took snapshot with yourkit, the memory stayed low. I can reproduce this. It is very strange, suppose yourkit should make Wildfly memory i=usage higher, not lower. But this tells me there may exist some switch that make Wildfly not use so much memory. But I don't know what is this switch, and why Wildfly has to use so much memory.
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Allocation call tree (all threads together)</title>
<STYLE type="text/css">
body{
margin-top: 20px;
margin-left: 50px;
margin-right: 50px;
background: white;
}
table{
border: 0px;
border-spacing: 0px;
}
td{
padding-left: 3px;
padding-right: 3px;
}
.tree-cell{
vertical-align: middle;
}
.header{
text-align: center;
padding-top: 3px;
padding-left: 10px;
padding-bottom: 3px;
padding-right: 10px;
font-weight: bold;
background-color: #DDDDDD;
}
.value{
text-align: right;
padding-left: 10px;
padding-right: 10px;
}
.percents{
font-size: 80%;
color: gray;
text-align: right;
padding-left: 10px;
padding-right: 10px;
}
.footer{
font-size: 80%;
text-align: right;
padding-top: 20px;
}
.odd {background-color: #F2F2F2;}
</STYLE>
</head>
<body>
<div><h2>Allocation call tree (all threads together)</h2></div>
<p>
<table width="100%">
<tr><td class="header" width="100%"> Call Tree </nobr></td><td class="header" colspan="2"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/desc.png"/> Objects </nobr></td><td class="header" colspan="2"> Size </nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/leaf-with-sibling.gif"/></td><td width="100%"><nobr><span class="tree-cell"><Objects without allocation information></span></nobr></td></tr></table></td><td class="value"><nobr>90,623,310</nobr></td><td class="percents"><nobr></nobr></td><td class="value"><nobr>4,776,067,264</nobr></td><td class="percents"><nobr></nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><span class="tree-cell"><All threads></span></nobr></td></tr></table></td><td class="value"><nobr>5,586,637</nobr></td><td class="percents"><nobr>100 %</nobr></td><td class="value"><nobr>282,133,712</nobr></td><td class="percents"><nobr>100 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">java.lang</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">Thread</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>5,585,940</nobr></td><td class="percents"><nobr>99 %</nobr></td><td class="value"><nobr>282,104,056</nobr></td><td class="percents"><nobr>99 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">ThreadPoolExecutor.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">1142</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange$1</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>61,933</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,471,864</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">HttpServerExchange.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">793</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">Connectors</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">executeRootHandler</span></font></b><span class="tree-cell">(HttpHandler, HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>61,933</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,471,864</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">Connectors.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">202</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.handlers</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">AuthenticationCallHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>61,933</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,471,864</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">AuthenticationCallHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">52</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.as.domain.http.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">DomainApiCheckHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,751</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,299,456</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainApiCheckHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">95</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server.handlers</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">BlockingHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,751</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,299,456</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">BlockingHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">56</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.as.domain.http.server.security</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SubjectDoAsHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,744</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,299,232</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SubjectDoAsHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">72</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server.handlers.encoding</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">EncodingHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,628</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,295,256</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">EncodingHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">66</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.as.domain.http.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">DomainApiHandler</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">handleRequest</span></font></b><span class="tree-cell">(HttpServerExchange)</span></nobr></td></tr></table></td><td class="value"><nobr>58,628</nobr></td><td class="percents"><nobr>1 %</nobr></td><td class="value"><nobr>2,295,256</nobr></td><td class="percents"><nobr>1 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainUtil.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">70</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.io</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">BlockingSenderImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">send</span></font></b><span class="tree-cell">(ByteBuffer)</span></nobr></td></tr></table></td><td class="value"><nobr>166</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>5,504</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainUtil.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">69</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.util</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HeaderMap</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">put</span></font></b><span class="tree-cell">(HttpString, long)</span></nobr></td></tr></table></td><td class="value"><nobr>42</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>1,072</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainUtil.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">70</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">getResponseSender</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>17</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>744</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">DomainUtil.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">58</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.util</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HeaderMap</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">put</span></font></b><span class="tree-cell">(HttpString, String)</span></nobr></td></tr></table></td><td class="value"><nobr>12</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>288</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">Common.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">109</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.io</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">BlockingSenderImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">send</span></font></b><span class="tree-cell">(String, IoCallback)</span></nobr></td></tr></table></td><td class="value"><nobr>7</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>272</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">Common.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">109</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">getResponseSender</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>1</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>48</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">BlockingHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">52</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">startBlocking</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>7</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>224</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">AuthenticationCallHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">50</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">authenticate</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,175</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>172,032</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">89</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">authTransition</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,175</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>172,032</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">96</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">attemptAuthentication</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,166</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>170,944</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">121</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl$AuthAttempter</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">access$100</span></font></b><span class="tree-cell">(SecurityContextImpl$AuthAttempter)</span></nobr></td></tr></table></td><td class="value"><nobr>3,148</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>170,512</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/expanded.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">219</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl$AuthAttempter</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">transition</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,148</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>170,512</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">250</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl$AuthAttempter</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">transition</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3,123</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>169,944</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">233</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.as.domain.http.server.security</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">AuthenticationMechanismWrapper</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">authenticate</span></font></b><span class="tree-cell">(HttpServerExchange, SecurityContext)</span></nobr></td></tr></table></td><td class="value"><nobr>25</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>568</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/leaf.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img2.png"/> <font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">121</span></font> </nobr></td></tr></table></td><td class="value"><nobr>18</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>432</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">SecurityContextImpl.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">104</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.security.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SecurityContextImpl</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">authTransition</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>9</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>1,088</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">AuthenticationCallHandler.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">55</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">io.undertow.server</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">HttpServerExchange</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">endExchange</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>7</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>376</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">Executors.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">511</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">org.infinispan.expiration.impl</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">ExpirationManagerImpl$ScheduledTask</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>144</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/vertical.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed.gif"/></td><td width="100%"><nobr><font color="#808080"><span class="tree-cell">ThreadPoolExecutor.java</span></font><font color="#808080"><span class="tree-cell">:</span></font><font color="#808080"><span class="tree-cell">1142</span></font> <img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">org.infinispan.persistence.file</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">SingleFileStore$3</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>3</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>72</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">org.xnio.nio</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">WorkerThread</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>561</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>20,320</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/collapsed-with-sibling.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img1.png"/> <span class="tree-cell">com.arjuna.ats.internal.arjuna.recovery</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">PeriodicRecovery</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>64</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>5,880</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr><td width="100%"></td><td class="value"><nobr>48</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>2,576</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
<tr class="odd"><td width="100%"><table><tr><td><img src="Allocation-call-tree--all-threads-together-files/empty.gif"/></td><td><img src="Allocation-call-tree--all-threads-together-files/leaf.gif"/></td><td width="100%"><nobr><img class="tree-cell" src="Allocation-call-tree--all-threads-together-files/img0.png"/> <span class="tree-cell">org.jboss.threads</span><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">JBossThread</span></font></b><span class="tree-cell">.</span><b><font color="#000000"><span class="tree-cell">run</span></font></b><span class="tree-cell">()</span></nobr></td></tr></table></td><td class="value"><nobr>24</nobr></td><td class="percents"><nobr>0 %</nobr></td><td class="value"><nobr>880</nobr></td><td class="percents"><nobr>0 %</nobr></td></tr>
</table>
</p>
<div class="footer">Generated by <a href="http://www.yourkit.com">YourKit Java Profiler 2015 build 15086</a> October 4, 2016 07:48:22 PM</div>
</body>
</html>
-
10. Re: why Wildfly is using high memory when no one uses it
bcrowell2015 Oct 11, 2016 7:45 AM (in response to hchenkwi)Helen,
What JDK version are you using? Can you provide specific minor point version with update level (e.g. 1.8.0_XX)?
How many applications are deployed to your Wildfly server?
I have a similar situation, and I believe it might be related to a Java NIO memory leak.
-
11. Re: why Wildfly is using high memory when no one uses it
hchenkwi Oct 11, 2016 9:47 AM (in response to bcrowell2015)Hi William,
The following is what we are using, and we are using Oracle certified java version.
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
we only have 4 applications deployed, but we do have a lot of files under each application.
From what I monitored, it seems the allocated memory can be Garbage collected, it is just allocating so much memory, and it will definitely be a problem when used in production. But I'm not sure how to monitor the native memory leak, maybe the heap memory is garbage collected, but the native memory is still there.
In your case, what is the symptom? how do you monitor the memory usage for both heap memory and native memory usage?
I saw an article talking about NIO memory leak: Fixing Java's ByteBuffer native memory "leak" (evanjones.ca)
Do you have work around fixing this issue?
Thanks a lot for your help,
Helen