Tracking high memory usage in thread local variable
Hi!
We're currently tracking down a memory usage issue that is keeping JBoss from completing some performance tests of our application. These currently run under WebLogic and WebSphere - but I'm pushing JBoss as a solution.
Our questions are:
1) Why is JBoss using so much memory in the thread local variable? Please see the memory map below for more info. Is this normal? Is there something special we have to do in order to clear this out?
2) Does JBoss pass by reference for local calls by default? For us WebLogic does and we had to enable a setting in WebSphere to enable this. Is there a similar setting in JBoss or is it the default behaviour?
In our latest test, we ran with the IBM JDK, so that we could get a heapdump and analyze it using their tools (useful for breaking down memory usage). Below is the memory graph/map created by IBMs heapdump analyzer tool.
Thanks in advance,
Chris Plough
Dumping object(s) : 63443 roots, sorted by total-size
Filter : total-size >= 10,485,760 and depth at most 100
Prune : N
<0> [1,019,475,328] 0x002d8d68 [304] class com/ibm/jvm/ExtendedSystem
<1> [1,019,090,504] 0x002d6c28 [304] class java/io/ObjectStreamClass
<2> [1,018,581,104] 0x101ec490 [40] sun/misc/SoftCache
<3> [1,018,581,064] 0x101ec430 [56] java/util/HashMap
<4> [1,018,581,008] 0x496c3850 [112] array of java/util/HashMap$Entry
<5> [1,018,517,232] 0x15711e08 [32] java/util/HashMap$Entry
<6> [1,018,516,712] 0x15711ea0 [32] java/io/ObjectStreamClass$FieldReflectorKey
<7> [1,018,516,552] 0x06341318 [304] class java/lang/reflect/Proxy
<8> [1,017,499,736] 0x10668168 [40] java/util/Collections$SynchronizedMap
<9> [1,017,499,696] 0x106681a8 [56] java/util/WeakHashMap
<10> [1,017,499,640] 0x1ebf7a38 [11,160] array of java/util/WeakHashMap$Entry
<11> [1,012,199,400] 0x1eaeab90 [40] java/util/WeakHashMap$Entry
<12> [1,012,189,720] 0x462546e8 [40] java/util/WeakHashMap$Entry
<13> [1,012,182,360] 0x1614f7b8 [40] java/util/WeakHashMap$Entry
<14> [1,012,182,320] 0x1062abb0 [304] class $Proxy2785
<15> [1,012,174,152] 0x161500b0 [72] java/lang/reflect/Method
<16> [1,012,174,080] 0x1cfaf948 [72] java/lang/reflect/Method
<17> [1,012,173,944] 0x020b8c38 [304] class java/util/ResourceBundle
<18> [1,012,173,520] 0x10269478 [40] sun/misc/SoftCache
<19> [1,012,173,480] 0x102861e0 [56] java/util/HashMap
<20> [1,012,173,424] 0x1a860890 [288] array of java/util/HashMap$Entry
<21> [1,012,029,248] 0x1b860e58 [32] java/util/HashMap$Entry
<22> [1,012,028,496] 0x1b860e78 [48] sun/misc/SoftCache$ValueCell
<23> [1,011,801,880] 0x10283208 [32] java/lang/ref/ReferenceQueue
<24> [1,011,801,832] 0x1b91f2a8 [48] sun/misc/SoftCache$ValueCell
<25> [1,011,801,712] 0x1b895408 [48] sun/misc/SoftCache$ValueCell
<26> [1,011,801,592] 0x1b91f218 [48] sun/misc/SoftCache$ValueCell
<27> [1,011,799,504] 0x1b91f240 [32] java/util/ResourceBundle$ResourceCacheKey
<28> [1,011,799,472] 0x1b91f260 [40] java/lang/ref/SoftReference
<29> [1,011,799,432] 0x101eb158 [176] org/jboss/mx/loading/UnifiedClassLoader3
<30> [999,482,456] 0x101eb5d0 [128] org/jboss/system/server/NoAnnotationURLClassLoader
<31> [999,456,200] 0x10313cb8 [32] java/util/Vector
<32> [999,456,168] 0x10a0d540 [2,368] array of java/lang/Object
<33> [999,338,480] 0x103137c8 [304] class org/jboss/util/timeout/TimeoutFactory
<34> [999,338,176] 0x11dc8fa0 [32] org/jboss/util/timeout/TimeoutFactory
<35> [999,335,560] 0x11903e90 [64] org/jboss/util/threadpool/BasicThreadPool
<36> [999,335,048] 0x143cf930 [56] java/lang/ThreadGroup
<37> [999,334,856] 0x1029b410 [56] java/lang/ThreadGroup
<38> [999,324,872] 0x101f61d8 [56] java/lang/ThreadGroup
<39> [999,324,584] 0x101f6280 [56] java/lang/ThreadGroup
<40> [612,919,104] 0x101ffd08 [24] array of java/lang/ThreadGroup
<41> [612,919,080] 0x153f5590 [56] java/lang/ThreadGroup
<42> [612,918,952] 0x19b7fcf8 [32] array of java/lang/Thread
<43> [612,917,408] 0x1063a920 [96] java/lang/Thread
<44> [612,916,464] 0x1fbbc008 [32] java/lang/ThreadLocal$ThreadLocalMap
<45> [612,916,432] 0x3872e510 [144] array of java/lang/ThreadLocal$ThreadLocalMap$Entry
<46> [612,900,360] 0x10fbbef8 [32] java/lang/ThreadLocal$ThreadLocalMap$Entry
<47> [612,900,328] 0x10fbbf18 [32] java/util/ArrayList
<48> [612,900,296] 0x4eb175a8 [9,224] array of java/lang/Object
- 2301 children of 0x4eb175a8 filtered.
- 2 children of 0x10fbbef8 filtered.
- 30 children of 0x3872e510 filtered.
<44> {612,919,080} 0x153f5590 [56] java/lang/ThreadGroup
- 5 children of 0x1063a920 filtered.
- 2 children of 0x19b7fcf8 filtered.
<42> {999,324,584} 0x101f6280 [56] java/lang/ThreadGroup
- 1 child of 0x153f5590 filtered.
<41> {999,324,872} 0x101f61d8 [56] java/lang/ThreadGroup
<40> [386,405,360] 0x1569aa28 [64] array of java/lang/Thread
<41> [386,396,040] 0x10d33c48 [96] java/lang/Thread
<42> [386,395,864] 0x19d48de8 [24] sun/rmi/transport/DGCImpl$LeaseChecker
<43> [386,395,840] 0x153f8cf8 [24] sun/rmi/transport/DGCImpl
<44> [386,395,816] 0x153e41e0 [48] java/util/Hashtable
<45> [386,395,768] 0x153e41a0 [24] array of java/util/Hashtable$Entry
<46> [386,395,464] 0x11595698 [32] java/util/Hashtable$Entry
<47> [386,395,352] 0x11595748 [32] sun/rmi/transport/DGCImpl$LeaseInfo
<48> [386,395,320] 0x11595738 [24] java/util/HashSet
<49> [386,395,296] 0x11595708 [56] java/util/HashMap
<50> [386,395,240] 0x115956b8 [24] array of java/util/HashMap$Entry
<51> [386,395,216] 0x115a9cc8 [32] java/util/HashMap$Entry
<52> [386,395,184] 0x156f9ea0 [64] sun/rmi/transport/Target
<53> [386,394,208] 0x1569abc0 [40] sun/rmi/transport/WeakRef
<54> [386,394,120] 0x14cd18e0 [72] org/jboss/invocation/jrmp/server/JRMPInvoker
<55> [386,393,304] 0x14c8a3e8 [56] org/jboss/invocation/jrmp/server/JRMPInvoker$1
<56> [386,393,080] 0x10798318 [32] org/jboss/mx/server/MBeanServerImpl
<57> [386,365,848] 0x10851230 [56] org/jboss/mx/server/registry/BasicMBeanRegistry
<58> [386,365,024] 0x10851200 [56] java/util/HashMap
<59> [386,364,944] 0x151f50e8 [96] array of java/util/HashMap$Entry
<60> [203,709,480] 0x16fed8f0 [32] java/util/HashMap$Entry
<61> [203,709,336] 0x16fed960 [56] java/util/HashMap
<62> [203,709,256] 0x16fed910 [32] array of java/util/HashMap$Entry
<63> [203,613,880] 0x17071338 [32] java/util/HashMap$Entry
<64> [203,613,728] 0x17065378 [40] org/jboss/mx/server/registry/MBeanEntry
<65> [203,612,984] 0x17052738 [104] org/jboss/mx/modelmbean/XMBean
<66> [203,524,240] 0x170529c0 [112] org/jboss/ejb/EJBDeployer
<67> [203,520,312] 0x170529a8 [24] org/jboss/mx/notification/ListenerRegistry
<68> [203,520,272] 0x17ddff68 [56] java/util/HashMap
<69> [203,520,192] 0x17ddff18 [32] array of java/util/HashMap$Entry
<70> [203,485,456] 0x17ddfed8 [32] java/util/HashMap$Entry
<71> [203,485,336] 0x1709b610 [24] $Proxy6
<72> [203,485,312] 0x1709b658 [32] org/jboss/mx/notification/NotificationListenerProxy
<73> [203,485,256] 0x141d7590 [120] org/jboss/management/j2ee/LocalJBossServerDomain
<74> [203,469,536] 0x14f80e38 [24] org/jboss/management/j2ee/factory/DefaultManagedObjectFactoryMap
<75> [203,469,416] 0x14f80e08 [56] java/util/HashMap
<76> [203,469,360] 0x14f80db8 [56] array of java/util/HashMap$Entry
<77> [201,711,512] 0x14fa3c28 [32] java/util/HashMap$Entry
<78> [201,711,480] 0x14fa3cf8 [24] org/jboss/management/j2ee/factory/WebModuleFactory
<79> [201,674,400] 0x14fa3cc8 [56] java/util/HashMap
<80> [201,674,344] 0x14fa3c78 [40] array of java/util/HashMap$Entry
<81> [201,674,112] 0x138d6f58 [32] java/util/HashMap$Entry
<82> [201,672,840] 0x130a5268 [144] org/jboss/deployment/DeploymentInfo
<83> [201,669,656] 0x18c0bdb0 [160] org/jboss/web/tomcat/tc5/Tomcat5
<84> [201,646,336] 0x18c0bb50 [56] java/util/HashMap
<85> [201,646,280] 0x18c0bb00 [40] array of java/util/HashMap$Entry
<86> [199,850,952] 0x1386c490 [32] java/util/HashMap$Entry
<87> [199,850,920] 0x131ed2c0 [40] org/jboss/web/WebApplication
<88> [199,850,880] 0x10327530 [128] java/net/FactoryURLClassLoader
<89> [199,849,936] 0x103275b0 [176] org/jboss/mx/loading/UnifiedClassLoader3
<90> [199,846,376] 0x1076e628 [80] org/jboss/mx/loading/UnifiedLoaderRepository3
<91> [186,852,336] 0x107a1108 [64] EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap
<92> [186,851,440] 0x42776e60 [64,968] array of EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry
<93> [49,729,840] 0x27b4c4d0 [32] EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry
<94> [49,729,808] 0x234f0b00 [32] EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry
<95> [49,729,776] 0x10385a20 [304] class org/jboss/ejb/plugins/LRUEnterpriseContextCachePolicy
<96> [49,729,424] 0x1a383510 [32] java/util/Timer
<97> [49,729,368] 0x10385500 [104] java/util/TimerThread
<98> [49,728,704] 0x1a433eb8 [24] java/util/TaskQueue
<99> [49,728,680] 0x284f9ed8 [8,400] array of java/util/TimerTask
- 2095 children of 0x284f9ed8 filtered.
<98> {999,334,856} 0x1029b410 [56] java/lang/ThreadGroup
- 5 children of 0x10385500 filtered.
<97> {49,728,704} 0x1a433eb8 [24] java/util/TaskQueue
- 1 child of 0x1a383510 filtered.
<96> {49,729,776} 0x10385a20 [304] class org/jboss/ejb/plugins/LRUEnterpriseContextCachePolicy
- 1 child of 0x10385a20 filtered.
- 1 child of 0x234f0b00 filtered.
- 2 children of 0x27b4c4d0 filtered.
<93> [47,368,640] 0x329bac60 [32] EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry
<94> [47,367,320] 0x05d30278 [304] class java/beans/Introspector
<95> [47,359,856] 0x1049ccc8 [56] java/util/WeakHashMap
<96> [47,359,800] 0x10464d48 [96] array of java/util/WeakHashMap$Entry
<97> [47,287,424] 0x3376fdb8 [40] java/util/WeakHashMap$Entry
<98> [47,274,520] 0x10d6d9e8 [304] class glog/util/event/EventQueue
<99> [47,266,520] 0x23311ca0 [48] java/util/TreeMap
<100> [47,266,448] 0x233c03d8 [40] java/util/TreeMap$Entry
- 4 children of 0x233c03d8 too deep.
- 1 child of 0x23311ca0 filtered.
- 5 children of 0x10d6d9e8 filtered.
- 2 children of 0x3376fdb8 filtered.
- 19 children of 0x10464d48 filtered.
- 1 child of 0x1049ccc8 filtered.
- 13 children of 0x05d30278 filtered.
- 2 children of 0x329bac60 filtered.
<93> [24,743,656] 0x322f7768 [32] EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry
<94> [24,743,504] 0x15450828 [32] EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry
<95> [24,743,472] 0x101adb30 [304] class org/jboss/invocation/MarshalledInvocation
<96> [24,743,168] 0x1544cbe8 [56] java/util/WeakHashMap
<97> [24,743,112] 0x2a5dece0 [7,752] array of java/util/WeakHashMap$Entry
- 1934 children of 0x2a5dece0 filtered.
- 1 child of 0x1544cbe8 filtered.
- 1 child of 0x15450828 filtered.
- 2 children of 0x322f7768 filtered.
<93> [11,546,808] 0x32eae468 [32] EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry
<94> [11,546,304] 0x10ac3250 [304] class glog/util/persistence/BusinessMap
- 3 children of 0x10ac3250 filtered.
- 2 children of 0x32eae468 filtered.
- 16234 children of 0x42776e60 filtered.
- 2 children of 0x107a1108 filtered.
<91> [11,386,072] 0x10657f98 [24] EDU/oswego/cs/dl/util/concurrent/CopyOnWriteArraySet
<92> [11,386,048] 0x1076e978 [24] EDU/oswego/cs/dl/util/concurrent/CopyOnWriteArrayList
<93> [11,386,024] 0x2a8a4010 [176] array of java/lang/Object
<94> [10,845,704] 0x101e81d8 [176] org/jboss/mx/loading/UnifiedClassLoader3
<95> [10,546,048] 0x18376c00 [32] java/util/Vector
<96> [10,546,016] 0x2a34daa0 [1,320] array of java/lang/Object
- 257 children of 0x2a34daa0 filtered.
<95> {999,482,456} 0x101eb5d0 [128] org/jboss/system/server/NoAnnotationURLClassLoader
<95> {999,482,456} 0x101eb5d0 [128] org/jboss/system/server/NoAnnotationURLClassLoader
<95> {199,846,376} 0x1076e628 [80] org/jboss/mx/loading/UnifiedLoaderRepository3
- 18 children of 0x101e81d8 filtered.
<94> {1,011,799,432} 0x101eb158 [176] org/jboss/mx/loading/UnifiedClassLoader3
<94> {199,849,936} 0x103275b0 [176] org/jboss/mx/loading/UnifiedClassLoader3
- 36 children of 0x2a8a4010 filtered.
- 11 children of 0x1076e628 filtered.
<90> {999,482,456} 0x101eb5d0 [128] org/jboss/system/server/NoAnnotationURLClassLoader
<90> {999,482,456} 0x101eb5d0 [128] org/jboss/system/server/NoAnnotationURLClassLoader
- 18 children of 0x103275b0 filtered.
- 12 children of 0x10327530 filtered.
<88> {201,672,840} 0x130a5268 [144] org/jboss/deployment/DeploymentInfo
- 4 children of 0x131ed2c0 filtered.
- 1 child of 0x1386c490 filtered.
- 5 children of 0x18c0bb00 filtered.
<84> {386,393,080} 0x10798318 [32] org/jboss/mx/server/MBeanServerImpl
- 17 children of 0x18c0bdb0 filtered.
<83> {386,393,080} 0x10798318 [32] org/jboss/mx/server/MBeanServerImpl
<83> {199,849,936} 0x103275b0 [176] org/jboss/mx/loading/UnifiedClassLoader3
- 16 children of 0x130a5268 filtered.
- 1 child of 0x138d6f58 filtered.
- 5 children of 0x14fa3c78 filtered.
- 1 child of 0x14fa3cf8 filtered.
- 1 child of 0x14fa3c28 filtered.
- 9 children of 0x14f80db8 filtered.
- 1 child of 0x14f80e38 filtered.
<74> {386,393,080} 0x10798318 [32] org/jboss/mx/server/MBeanServerImpl
- 20 children of 0x141d7590 filtered.
- 2 children of 0x1709b658 filtered.
- 1 child of 0x17ddfed8 filtered.
- 2 children of 0x17ddff18 filtered.
- 1 child of 0x17ddff68 filtered.
- 1 child of 0x170529a8 filtered.
<67> {386,393,080} 0x10798318 [32] org/jboss/mx/server/MBeanServerImpl
<67> <13,181,056 parent:0x1abc65d0> 0x156aa580 [64] org/jboss/tm/TxManager
- 10 children of 0x170529c0 filtered.
<66> {386,393,080} 0x10798318 [32] org/jboss/mx/server/MBeanServerImpl
<66> {203,613,728} 0x17065378 [40] org/jboss/mx/server/registry/MBeanEntry
- 10 children of 0x17052738 filtered.
<65> {203,524,240} 0x170529c0 [112] org/jboss/ejb/EJBDeployer
- 4 children of 0x17065378 filtered.
- 1 child of 0x17071338 filtered.
- 2 children of 0x16fed910 filtered.
- 1 child of 0x16fed960 filtered.
- 1 child of 0x16fed8f0 filtered.
<60> [104,077,296] 0x16f446d0 [32] java/util/HashMap$Entry
<61> [102,876,416] 0x16f446f0 [56] java/util/HashMap
<62> [102,876,336] 0x374a8c78 [10,872] array of java/util/HashMap$Entry
- 2713 children of 0x374a8c78 filtered.
- 1 child of 0x16f446f0 filtered.
- 2 children of 0x16f446d0 filtered.
<60> [71,109,192] 0x14291ae0 [32] java/util/HashMap$Entry
<61> [71,109,160] 0x14291b00 [56] java/util/HashMap
<62> [71,109,080] 0x369d5358 [3,944] array of java/util/HashMap$Entry
- 981 children of 0x369d5358 filtered.
- 1 child of 0x14291b00 filtered.
- 1 child of 0x14291ae0 filtered.
- 17 children of 0x151f50e8 filtered.
- 1 child of 0x10851200 filtered.
<58> {386,393,080} 0x10798318 [32] org/jboss/mx/server/MBeanServerImpl
<58> {199,846,376} 0x1076e628 [80] org/jboss/mx/loading/UnifiedLoaderRepository3
- 5 children of 0x10851230 filtered.
<57> {386,393,080} 0x10798318 [32] org/jboss/mx/server/MBeanServerImpl
<57> {199,846,376} 0x1076e628 [80] org/jboss/mx/loading/UnifiedLoaderRepository3
- 1 child of 0x10798318 filtered.
<56> {386,394,120} 0x14cd18e0 [72] org/jboss/invocation/jrmp/server/JRMPInvoker
<56> {386,394,120} 0x14cd18e0 [72] org/jboss/invocation/jrmp/server/JRMPInvoker
- 4 children of 0x14c8a3e8 filtered.
- 5 children of 0x14cd18e0 filtered.
<54> {386,394,120} 0x14cd18e0 [72] org/jboss/invocation/jrmp/server/JRMPInvoker
- 1 child of 0x1569abc0 filtered.
<53> {1,011,799,432} 0x101eb158 [176] org/jboss/mx/loading/UnifiedClassLoader3
- 7 children of 0x156f9ea0 filtered.
- 1 child of 0x115a9cc8 filtered.
- 1 child of 0x11595748 filtered.
- 1 child of 0x11595698 filtered.
- 1 child of 0x153e41a0 filtered.
<44> {386,396,040} 0x10d33c48 [96] java/lang/Thread
<42> {999,324,584} 0x101f6280 [56] java/lang/ThreadGroup
- 3 children of 0x10d33c48 filtered.
- 10 children of 0x1569aa28 filtered.
- 1 child of 0x101f6280 filtered.
- 3 children of 0x101f61d8 filtered.
- 3 children of 0x1029b410 filtered.
- 3 children of 0x143cf930 filtered.
- 7 children of 0x11903e90 filtered.
- 2 children of 0x11dc8fa0 filtered.
- 526 children of 0x10a0d540 filtered.
- 13 children of 0x101eb5d0 filtered.
<30> {999,482,456} 0x101eb5d0 [128] org/jboss/system/server/NoAnnotationURLClassLoader
<30> [10,526,232] 0x10fc0808 [64] EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap
<31> [10,525,664] 0x4bee3868 [63,600] array of EDU/oswego/cs/dl/util/concurrent/ConcurrentReaderHashMap$Entry
- 15895 children of 0x4bee3868 filtered.
- 2 children of 0x10fc0808 filtered.
<30> {199,846,376} 0x1076e628 [80] org/jboss/mx/loading/UnifiedLoaderRepository3
- 18 children of 0x101eb158 filtered.
- 1 child of 0x1b91f260 filtered.
- 2 children of 0x1b91f240 filtered.
- 2 children of 0x1b91f218 filtered.
- 2 children of 0x1b895408 filtered.
- 2 children of 0x1b91f2a8 filtered.
- 1 child of 0x10283208 filtered.
- 2 children of 0x1b860e78 filtered.
- 1 child of 0x1b860e58 filtered.
- 67 children of 0x1a860890 filtered.
<19> {1,011,801,880} 0x10283208 [32] java/lang/ref/ReferenceQueue
<18> {1,012,173,944} 0x020b8c38 [304] class java/util/ResourceBundle
- 3 children of 0x020b8c38 filtered.
- 4 children of 0x1cfaf948 filtered.
<16> {1,012,173,944} 0x020b8c38 [304] class java/util/ResourceBundle
- 4 children of 0x161500b0 filtered.
- 10 children of 0x1062abb0 filtered.
- 1 child of 0x1614f7b8 filtered.
- 2 children of 0x462546e8 filtered.
- 2 children of 0x1eaeab90 filtered.
- 2784 children of 0x1ebf7a38 filtered.
- 1 child of 0x106681a8 filtered.
<9> {1,017,499,736} 0x10668168 [40] java/util/Collections$SynchronizedMap
- 6 children of 0x06341318 filtered.
- 1 child of 0x15711ea0 filtered.
- 1 child of 0x15711e08 filtered.
- 23 children of 0x496c3850 filtered.
- 1 child of 0x101ec490 filtered.
<2> {1,019,090,504} 0x002d6c28 [304] class java/io/ObjectStreamClass
- 11 children of 0x002d6c28 filtered.
- 10 children of 0x002d8d68 filtered.
1/63443 roots were dumped. There were 138 objects expanded.