13 Replies Latest reply on Aug 12, 2009 10:44 AM by clebert.suconic

    Lots of largemessages

    timfox

      I noticed this at the weekend, when I did a kill -3:

      ...
      Heap
       [junit] PSYoungGen total 81600K, used 42070K [0xada40000, 0xb4af0000, 0xb4c00000)
       [junit] eden space 47104K, 67% used [0xada40000,0xaf96a830,0xb0840000)
       [junit] from space 34496K, 29% used [0xb2850000,0xb323b1d0,0xb4a00000)
       [junit] to space 32832K, 0% used [0xb0840000,0xb0840000,0xb2850000)
       [junit] PSOldGen total 498944K, used 367777K [0x74c00000, 0x93340000, 0xada40000)
       [junit] object space 498944K, 73% used [0x74c00000,0x8b328498,0x93340000)
       [junit] PSPermGen total 16384K, used 13329K [0x70c00000, 0x71c00000, 0x74c00000)
       [junit] object space 16384K, 81% used [0x70c00000,0x71904678,0x71c00000)
       [junit]
       [junit]
       [junit] num #instances #bytes class name
       [junit] ----------------------------------------------
       [junit] 1: 708550 109925912 [B
       [junit] 2: 450399 28825536 org.jboss.messaging.core.remoting.impl.wireformat.SessionReceiveMessage
       [junit] 3: 835287 26729184 org.jboss.messaging.core.server.impl.MessageReferenceImpl
       [junit] 4: 14687 24820128 [I
       [junit] 5: 1178059 18848944 java.util.concurrent.ConcurrentLinkedQueue$Node
       [junit] 6: 400336 12810752 java.util.concurrent.ConcurrentHashMap$Segment
       [junit] 7: 307046 12281840 org.jboss.messaging.core.remoting.impl.wireformat.SessionReceiveContinuationMessage
       [junit] 8: 458239 10997736 java.util.concurrent.locks.ReentrantLock$NonfairSync
       [junit] 9: 292037 7008888 org.jboss.messaging.utils.concurrent.LinkedBlockingDeque$Node
       [junit] 10: 400336 6609824 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
       [junit] 11: 73844 5316768 org.jboss.messaging.core.server.impl.ServerMessageImpl
       [junit] 12: 215287 5166888 org.jboss.messaging.utils.SimpleString
       [junit] 13: 43630 3897920 [C
       [junit] 14: 95968 3694112 [Ljava.util.HashMap$Entry;
       [junit] 15: 92082 3683280 java.util.HashMap
       [junit] 16: 29718 3456352 <constMethodKlass>
       [junit] 17: 170860 2733760 java.util.concurrent.atomic.AtomicInteger
       [junit] 18: 31652 2666920 [Ljava.lang.Object;
      


      There was a deadlock (which was not important). But the interesting thing was there is 450K messages in memory. Seems like a lot to me.

      The test in question was org.jboss.messaging.tests.integration.jms.bridge.JMSBridgeReconnectionTest.testCrashAndReconnectDestBasic_OnceAndOnlyOnce_P_LargeMessage

      Looking at that test, it does not send 450K messages. Perhaps something is leaking?

      Also I noticed this comment in the class:

      // The bridge won't work with largeMessages & failures
      // https://jira.jboss.org/jira/browse/JBMESSAGING-1601
      


      Is this still true?


        • 1. Re: Lots of largemessages
          timfox

          Here's the full dump

          Heap
           [junit] PSYoungGen total 81600K, used 42070K [0xada40000, 0xb4af0000, 0xb4c00000)
           [junit] eden space 47104K, 67% used [0xada40000,0xaf96a830,0xb0840000)
           [junit] from space 34496K, 29% used [0xb2850000,0xb323b1d0,0xb4a00000)
           [junit] to space 32832K, 0% used [0xb0840000,0xb0840000,0xb2850000)
           [junit] PSOldGen total 498944K, used 367777K [0x74c00000, 0x93340000, 0xada40000)
           [junit] object space 498944K, 73% used [0x74c00000,0x8b328498,0x93340000)
           [junit] PSPermGen total 16384K, used 13329K [0x70c00000, 0x71c00000, 0x74c00000)
           [junit] object space 16384K, 81% used [0x70c00000,0x71904678,0x71c00000)
           [junit]
           [junit]
           [junit] num #instances #bytes class name
           [junit] ----------------------------------------------
           [junit] 1: 708550 109925912 [B
           [junit] 2: 450399 28825536 org.jboss.messaging.core.remoting.impl.wireformat.SessionReceiveMessage
           [junit] 3: 835287 26729184 org.jboss.messaging.core.server.impl.MessageReferenceImpl
           [junit] 4: 14687 24820128 [I
           [junit] 5: 1178059 18848944 java.util.concurrent.ConcurrentLinkedQueue$Node
           [junit] 6: 400336 12810752 java.util.concurrent.ConcurrentHashMap$Segment
           [junit] 7: 307046 12281840 org.jboss.messaging.core.remoting.impl.wireformat.SessionReceiveContinuationMessage
           [junit] 8: 458239 10997736 java.util.concurrent.locks.ReentrantLock$NonfairSync
           [junit] 9: 292037 7008888 org.jboss.messaging.utils.concurrent.LinkedBlockingDeque$Node
           [junit] 10: 400336 6609824 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
           [junit] 11: 73844 5316768 org.jboss.messaging.core.server.impl.ServerMessageImpl
           [junit] 12: 215287 5166888 org.jboss.messaging.utils.SimpleString
           [junit] 13: 43630 3897920 [C
           [junit] 14: 95968 3694112 [Ljava.util.HashMap$Entry;
           [junit] 15: 92082 3683280 java.util.HashMap
           [junit] 16: 29718 3456352 <constMethodKlass>
           [junit] 17: 170860 2733760 java.util.concurrent.atomic.AtomicInteger
           [junit] 18: 31652 2666920 [Ljava.lang.Object;
           [junit] 19: 75911 2429152 org.jboss.messaging.core.buffers.HeapChannelBuffer
           [junit] 20: 75867 2427744 org.jboss.messaging.core.buffers.DynamicChannelBuffer
           [junit] 21: 100539 2412936 java.util.HashMap$Entry
           [junit] 22: 29718 2381984 <methodKlass>
           [junit] 23: 84381 2025144 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
           [junit] 24: 25021 2001680 [Ljava.util.concurrent.ConcurrentHashMap$Segment;
           [junit] 25: 40437 1747424 <symbolKlass>
           [junit] 26: 2675 1407432 <constantPoolKlass>
           [junit] 27: 13565 1302240 org.jboss.messaging.core.remoting.impl.ChannelImpl
           [junit] 28: 80321 1285136 org.jboss.messaging.utils.TypedProperties
           [junit] 29: 75622 1209952 java.util.HashMap$EntrySet
           [junit] 30: 2675 1136840 <instanceKlassKlass>
           [junit] 31: 27814 1112560 org.jboss.messaging.utils.concurrent.LinkedBlockingDeque
           [junit] 32: 43521 1044504 java.lang.String
           [junit] 33: 25021 1000840 java.util.concurrent.ConcurrentHashMap
           [junit] 34: 7678 982784 org.jboss.messaging.core.server.impl.ServerSessionImpl
           [junit] 35: 2428 892544 <constantPoolCacheKlass>
           [junit] 36: 52639 842224 java.util.concurrent.locks.ReentrantLock
           [junit] 37: 51662 826592 org.jboss.messaging.utils.TypedProperties$IntValue
           [junit] 38: 3375 788840 <methodDataKlass>
           [junit] 39: 27345 656280 org.jboss.messaging.utils.Pair
           [junit] 40: 37884 606144 org.jboss.messaging.utils.TypedProperties$StringValue
           [junit] 41: 22215 533160 java.util.concurrent.ConcurrentHashMap$HashEntry
           [junit] 42: 64456 515648 java.lang.Object
           [junit] 43: 21063 505512 org.jboss.messaging.core.remoting.impl.wireformat.NullResponseMessage
           [junit] 44: 4409 423264 org.jboss.messaging.core.persistence.impl.journal.JournalLargeServerMessage
           [junit] 45: 17011 408264 java.util.ArrayList
           [junit] 46: 17006 408144 org.jboss.messaging.core.server.impl.ServerConsumerImpl$2
           [junit] 47: 7065 395640 org.jboss.messaging.core.transaction.impl.TransactionImpl
           [junit] 48: 24146 386336 java.util.concurrent.ConcurrentLinkedQueue
           [junit] 49: 2733 349824 org.jboss.messaging.core.server.impl.QueueImpl
           [junit] 50: 6148 295104 java.lang.ThreadGroup
           [junit] 51: 2718 282672 org.jboss.messaging.core.server.impl.ServerConsumerImpl
           [junit] 52: 3918 282096 org.jboss.netty.util.internal.LinkedTransferQueue$PaddedAtomicReference
           [junit] 53: 2877 276192 java.lang.Class
           [junit] 54: 6135 266400 [Ljava.lang.Thread;
           [junit] 55: 11029 264696 java.util.LinkedList$Entry
           [junit] 56: 10982 263568 java.util.LinkedList
           [junit] 57: 10967 263208 org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor
           [junit] 58: 10653 255672 org.jboss.messaging.utils.SimpleIDGenerator
           [junit] 59: 3879 234392 [S
           [junit] 60: 6843 218976 java.lang.ref.Finalizer
           [junit] 61: 4233 195136 [[I
           [junit] 62: 2913 186432 org.jboss.messaging.core.remoting.impl.RemotingConnectionImpl
           [junit] 63: 3863 185424 java.util.LinkedHashMap
           [junit] 64: 4450 178000 org.jboss.messaging.core.journal.impl.NIOSequentialFile
           [junit] 65: 11036 176576 java.util.concurrent.ConcurrentHashMap$Values
           [junit] 66: 10967 175472 org.jboss.messaging.utils.OrderedExecutorFactory$OrderedExecutor$1
           [junit] 67: 5264 168448 org.jboss.netty.util.internal.ConcurrentHashMap$Segment
           [junit] 68: 2000 160000 java.lang.reflect.Method
           [junit] 69: 3243 155664 java.util.concurrent.FutureTask$Sync
           [junit] 70: 2058 148176 org.jboss.messaging.core.client.impl.ClientMessageImpl
           [junit] 71: 4485 143520 java.util.LinkedHashMap$Entry
           [junit] 72: 2913 139824 org.jboss.messaging.core.remoting.impl.Pinger
           [junit] 73: 3243 129720 java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
           [junit] 74: 7678 122848 org.jboss.messaging.core.server.impl.ServerSessionPacketHandler
           [junit] 75: 3012 120480 org.jboss.messaging.core.remoting.impl.wireformat.SessionSendMessage
           [junit] 76: 3391 108512 java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
           [junit] 77: 2586 103440 org.jboss.messaging.core.remoting.impl.wireformat.SessionXAResponseMessage
           [junit] 78: 6205 99280 java.util.concurrent.CopyOnWriteArrayList
           [junit] 79: 5998 95968 java.util.concurrent.atomic.AtomicLong
           [junit] 80: 5892 94272 org.jboss.messaging.utils.ConcurrentHashSet
           [junit] 81: 5622 89952 java.lang.Long
           [junit] 82: 2716 86912 org.jboss.messaging.core.postoffice.impl.LocalQueueBinding
           [junit] 83: 5264 86456 [Lorg.jboss.netty.util.internal.ConcurrentHashMap$HashEntry;
           [junit] 84: 632 85952 java.text.DecimalFormat
           [junit] 85: 5254 84064 java.util.HashSet
           [junit] 86: 692 83040 java.net.SocksSocketImpl
           [junit] 87: 3412 81888 org.jboss.messaging.core.journal.impl.JournalImpl$JournalRecord
           [junit] 88: 4838 77408 org.jboss.messaging.core.postoffice.impl.DuplicateIDCacheImpl$ByteArrayHolder
           [junit] 89: 4796 76736 java.util.concurrent.atomic.AtomicBoolean
           [junit] 90: 314 75360 org.jboss.messaging.core.config.impl.ConfigurationImpl
           [junit] 91: 4518 72288 java.io.File
           [junit] 92: 2253 72096 org.jboss.messaging.core.remoting.impl.invm.InVMConnection
           [junit] 93: 660 68640 sun.nio.ch.SocketChannelImpl
           [junit] 94: 2815 67560 java.util.concurrent.Semaphore$NonfairSync
           [junit] 95: 2765 66360 org.jboss.messaging.core.list.impl.PriorityLinkedListImpl
           [junit] 96: 2733 65592 org.jboss.messaging.core.server.impl.ScheduledDeliveryHandlerImpl
           [junit] 97: 2698 64752 java.net.InetSocketAddress
           [junit] 98: 627 60192 org.jboss.messaging.core.paging.impl.PagingStoreImpl
           [junit] 99: 549 57096 java.lang.Thread
           [junit] 100: 392 56448 org.jboss.messaging.core.client.impl.ConnectionManagerImpl
           [junit] 101: 173 55360 <objArrayKlassKlass>
           [junit] 102: 1349 53960 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext
           [junit] 103: 464 51968 org.jboss.netty.channel.socket.nio.NioClientSocketChannel
           [junit] 104: 3243 51888 java.util.concurrent.Executors$RunnableAdapter
           [junit] 105: 2877 46032 java.util.concurrent.Semaphore
           [junit] 106: 314 45216 org.jboss.messaging.core.server.impl.MessagingServerImpl
           [junit] 107: 2733 43728 org.jboss.messaging.core.server.impl.QueueImpl$DeliverRunner
           [junit] 108: 2733 43728 org.jboss.messaging.core.server.impl.RoundRobinDistributor
           [junit] 109: 2718 43488 org.jboss.messaging.core.server.impl.ServerConsumerImpl$3
           [junit] 110: 632 40448 java.text.DecimalFormatSymbols
           [junit] 111: 238 38080 org.jboss.messaging.core.client.impl.ClientSessionFactoryImpl
           [junit] 112: 146 37424 [Ljava.lang.ThreadGroup;
           [junit] 113: 2332 37312 java.util.HashMap$KeySet
           [junit] 114: 2125 37152 [Ljava.lang.Class;
           [junit] 115: 1153 36896 org.jboss.messaging.core.transaction.impl.XidImpl
           [junit] 116: 871 34840 org.jboss.messaging.core.remoting.impl.invm.InVMConnector
           [junit] 117: 483 34776 java.util.concurrent.ThreadPoolExecutor
           [junit] 118: 1020 32640 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
           [junit] 119: 660 31680 sun.nio.ch.SocketAdaptor
           [junit] 120: 1306 31344 org.jboss.netty.util.internal.LinkedTransferQueue$QNode
           [junit] 121: 1893 30288 org.jboss.messaging.utils.OrderedExecutorFactory
           [junit] 122: 1203 28872 java.util.Hashtable$Entry
           [junit] 123: 1780 28480 org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingFailureListener
           [junit] 124: 1780 28480 org.jboss.messaging.core.client.impl.ConnectionManagerImpl$Channel0Handler
           [junit] 125: 1780 28480 org.jboss.messaging.core.client.impl.ConnectionManagerImpl$FailedConnectionAction
           [junit] 126: 1771 28336 java.util.HashMap$Values
           [junit] 127: 318 27984 org.jboss.messaging.core.postoffice.impl.PostOfficeImpl
           [junit] 128: 318 27984 org.jboss.messaging.core.management.impl.ManagementServiceImpl
           [junit] 129: 850 27200 org.jboss.messaging.core.remoting.impl.wireformat.CreateSessionResponseMessage
           [junit] 130: 1133 27192 org.jboss.messaging.core.server.impl.MessagingServerPacketHandler
           [junit] 131: 660 26400 org.jboss.netty.channel.socket.nio.DefaultNioSocketChannelConfig
           [junit] 132: 329 26320 [Lorg.jboss.netty.util.internal.ConcurrentHashMap$Segment;
           [junit] 133: 326 26080 java.util.concurrent.ScheduledThreadPoolExecutor
           [junit] 134: 632 25280 java.text.DigitList
           [junit] 135: 1018 24432 org.jboss.messaging.core.remoting.impl.wireformat.PacketImpl
           [junit] 136: 979 23496 java.util.concurrent.locks.ReentrantReadWriteLock
           [junit] 137: 2913 23304 org.jboss.messaging.core.remoting.impl.PacketDecoder
           [junit] 138: 689 22048 org.jboss.netty.channel.UnfailingChannelFuture
           [junit] 139: 689 22048 org.jboss.netty.channel.DefaultChannelPipeline
           [junit] 140: 907 21768 org.jboss.messaging.core.config.TransportConfiguration
           [junit] 141: 897 21528 java.net.Inet4Address
           [junit] 142: 669 21408 java.lang.ref.SoftReference
           [junit] 143: 660 21120 org.jboss.netty.channel.socket.nio.NioSocketChannel$WriteBuffer
           [junit] 144: 660 21120 org.jboss.netty.channel.AdaptiveReceiveBufferSizePredictor
           [junit] 145: 1316 21056 org.jboss.messaging.core.remoting.impl.invm.InVMConnector$Listener
           [junit] 146: 196 20384 org.jboss.netty.channel.socket.nio.NioAcceptedSocketChannel
           [junit] 147: 318 20352 org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl
           [junit] 148: 632 20224 org.jboss.messaging.core.settings.impl.HierarchicalObjectRepository
           [junit] 149: 346 19376 org.jboss.messaging.core.settings.impl.AddressSettings
           [junit] 150: 756 18144 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$HoldCounter
           [junit] 151: 1133 18128 org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$FailedConnectionAction
           [junit] 152: 1116 17856 java.lang.Integer
           [junit] 153: 318 17808 org.jboss.messaging.core.paging.impl.PagingManagerImpl
           [junit] 154: 176 17536 [Ljava.util.Hashtable$Entry;
           [junit] 155: 720 17280 org.jboss.messaging.utils.JBMThreadFactory
           [junit] 156: 689 16536 [Ljava.nio.channels.SelectionKey;
           [junit] 157: 1020 16320 java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
           [junit] 158: 1020 16320 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
           [junit] 159: 660 15840 org.jboss.messaging.integration.transports.netty.NettyConnection
           [junit] 160: 660 15840 org.jboss.messaging.integration.transports.netty.MessagingFrameDecoder
           [junit] 161: 979 15664 java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
           [junit] 162: 977 15632 org.jboss.messaging.core.client.impl.ConnectionManagerImpl$DelegatingBufferHandler
           [junit] 163: 972 15552 org.jboss.messaging.core.server.impl.ServerSessionImpl$28
           [junit] 164: 486 15552 org.apache.xerces.dom.AttrImpl
           [junit] 165: 646 15504 org.jboss.netty.util.internal.LinkedTransferQueue
           [junit] 166: 241 15424 java.util.logging.Logger
           [junit] 167: 270 15120 org.jboss.netty.channel.socket.nio.NioWorker
           [junit] 168: 937 14992 org.jboss.messaging.core.remoting.impl.invm.InVMAcceptor$Listener
           [junit] 169: 464 14848 org.jboss.netty.channel.DefaultChannelFuture
           [junit] 170: 464 14848 org.jboss.messaging.integration.transports.netty.NettyConnector$MessagingClientChannelHandler
           [junit] 171: 928 14848 org.jboss.messaging.integration.transports.netty.NettyConnector$Listener
           [junit] 172: 262 14672 org.jboss.messaging.core.security.impl.SecurityStoreImpl
           [junit] 173: 888 14208 org.jboss.messaging.utils.TypedProperties$BytesValue
           [junit] 174: 120 13632 [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;
           [junit] 175: 564 13536 javax.management.MBeanParameterInfo
           [junit] 176: 330 13200 org.jboss.messaging.core.server.impl.ServerConsumerImpl$LargeMessageDeliverer
           [junit] 177: 329 13160 org.jboss.netty.util.internal.ConcurrentHashMap
           [junit] 178: 407 13024 java.util.Collections$SynchronizedMap
           [junit] 179: 318 12720 org.jboss.messaging.core.management.impl.MessagingServerControlImpl
           [junit] 180: 318 12720 org.jboss.messaging.core.management.impl.ReplicationOperationInvokerImpl
           [junit] 181: 318 12720 org.jboss.messaging.core.messagecounter.impl.MessageCounterManagerImpl
           [junit] 182: 318 12720 org.jboss.messaging.core.transaction.impl.ResourceManagerImpl
           [junit] 183: 437 12616 [Ljava.lang.String;
           [junit] 184: 188 12032 java.lang.reflect.Constructor
           [junit] 185: 243 11664 org.apache.xerces.dom.ElementImpl
           [junit] 186: 722 11552 java.io.FileDescriptor
           [junit] 187: 699 11184 java.nio.channels.spi.AbstractInterruptibleChannel$1
           [junit] 188: 689 11024 sun.nio.ch.SocketOptsImpl$IP$TCP
           [junit] 189: 689 11024 org.jboss.netty.channel.SucceededChannelFuture
           [junit] 190: 689 11024 sun.nio.ch.OptionAdaptor
           [junit] 191: 661 10576 org.jboss.netty.util.internal.ThreadLocalBoolean
           [junit] 192: 660 10560 org.jboss.netty.channel.socket.nio.NioSocketChannel$WriteTask
           [junit] 193: 660 10560 java.util.concurrent.atomic.AtomicReference
           [junit] 194: 660 10560 sun.nio.ch.SocketChannelImpl$1
           [junit] 195: 106 10176 org.jboss.messaging.integration.transports.netty.NettyConnector
           [junit] 196: 318 10176 org.jboss.messaging.core.paging.impl.PagingStoreFactoryNIO
           [junit] 197: 312 9984 org.jboss.messaging.core.remoting.impl.invm.InVMAcceptor
           [junit] 198: 244 9760 java.util.concurrent.LinkedBlockingQueue
           [junit] 199: 363 8712 javax.management.NotificationBroadcasterSupport
           [junit] 200: 542 8672 <compiledICHolderKlass>
           [junit] 201: 360 8640 java.util.concurrent.SynchronousQueue
           [junit] 202: 82 8528 org.jboss.messaging.core.journal.impl.JournalImpl
           [junit] 203: 175 8400 java.lang.Package
           [junit] 204: 347 8328 java.lang.ref.WeakReference
           [junit] 205: 347 8328 java.util.Vector
           [junit] 206: 260 8320 javax.management.MBeanAttributeInfo
           [junit] 207: 258 8256 org.jboss.messaging.core.remoting.impl.wireformat.ReattachSessionResponseMessage
           [junit] 208: 252 8064 java.lang.ThreadLocal$ThreadLocalMap$Entry
           [junit] 209: 335 8040 java.util.logging.LogManager$LogNode
           [junit] 210: 326 7824 java.util.concurrent.DelayQueue
           [junit] 211: 326 7824 java.util.PriorityQueue
           [junit] 212: 191 7640 javax.management.MBeanOperationInfo
           [junit] 213: 318 7632 org.jboss.messaging.core.transaction.impl.ResourceManagerImpl$TxTimeoutHandler
           [junit] 214: 318 7632 org.jboss.messaging.core.postoffice.impl.WildcardAddressManager
           [junit] 215: 318 7632 org.jboss.messaging.core.server.impl.QueueFactoryImpl
           [junit] 216: 314 7536 org.jboss.messaging.core.security.impl.JBMSecurityManagerImpl
           [junit] 217: 122 7488 [Ljava.lang.reflect.Method;
           [junit] 218: 81 6992 [Ljava.util.WeakHashMap$Entry;
           [junit] 219: 277 6648 org.jboss.messaging.core.persistence.impl.nullpm.NullStorageManager
           [junit] 220: 159 6360 java.util.WeakHashMap$Entry
           [junit] 221: 157 6280 java.util.Hashtable
           [junit] 222: 196 6272 org.jboss.messaging.integration.transports.netty.NettyAcceptor$MessagingServerChannelHandler
           [junit] 223: 392 6272 org.jboss.messaging.integration.transports.netty.NettyAcceptor$Listener
           [junit] 224: 385 6160 java.util.Collections$SynchronizedCollection
           [junit] 225: 93 5952 org.jboss.messaging.core.server.cluster.impl.ClusterManagerImpl
           [junit] 226: 366 5856 java.util.concurrent.ConcurrentHashMap$KeySet
           [junit] 227: 242 5808 org.apache.xerces.dom.AttributeMap
           [junit] 228: 240 5760 java.util.Random
           [junit] 229: 360 5760 java.util.concurrent.SynchronousQueue$TransferStack
           [junit] 230: 237 5688 org.jboss.messaging.core.client.impl.RoundRobinConnectionLoadBalancingPolicy
           [junit] 231: 98 5488 java.net.URL
           [junit] 232: 137 5480 java.util.IdentityHashMap
           [junit] 233: 62 5456 org.jboss.messaging.core.client.impl.ClientSessionImpl
           [junit] 234: 329 5264 org.jboss.messaging.core.server.impl.ServerConsumerImpl$1
           [junit] 235: 328 5248 org.jboss.netty.util.internal.ConcurrentHashMap$Values
           [junit] 236: 326 5216 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue
           [junit] 237: 108 5184 javax.management.MBeanInfo
           [junit] 238: 318 5088 org.jboss.messaging.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler
           [junit] 239: 632 5056 org.jboss.messaging.core.settings.impl.HierarchicalObjectRepository$MatchComparator
           [junit] 240: 75 4800 java.util.regex.Pattern
           [junit] 241: 186 4464 sun.reflect.NativeMethodAccessorImpl
           [junit] 242: 182 4368 javax.management.ImmutableDescriptor
           [junit] 243: 270 4320 org.jboss.messaging.core.postoffice.impl.PostOfficeImpl$Reaper
           [junit] 244: 135 4320 org.jboss.messaging.integration.transports.netty.VirtualExecutorService
           [junit] 245: 143 4256 [Ljavax.management.MBeanParameterInfo;
           [junit] 246: 256 4096 java.lang.Byte
           [junit] 247: 251 4016 java.util.concurrent.LinkedBlockingQueue$Node
           [junit] 248: 164 3936 org.jboss.netty.channel.group.DefaultChannelGroup
           [junit] 249: 237 3792 org.jboss.messaging.utils.Random
           [junit] 250: 237 3792 [Lorg.jboss.messaging.core.client.impl.ConnectionManager;
           [junit] 251: 76 3648 java.util.WeakHashMap
           [junit] 252: 41 3608 org.jboss.messaging.core.asyncio.impl.TimedBuffer
           [junit] 253: 223 3568 org.jboss.messaging.core.logging.Logger
           [junit] 254: 67 3456 [Ljavax.management.MBeanAttributeInfo;
           [junit] 255: 106 3392 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink
           [junit] 256: 106 3392 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss
           [junit] 257: 135 3240 [Lorg.jboss.netty.channel.socket.nio.NioWorker;
           [junit] 258: 96 3128 [Ljava.security.ProtectionDomain;
           [junit] 259: 78 3120 org.jboss.messaging.core.journal.impl.JournalFileImpl
           [junit] 260: 32 3072 org.jboss.messaging.core.client.impl.ClientConsumerImpl
           [junit] 261: 191 3056 sun.reflect.DelegatingMethodAccessorImpl
           [junit] 262: 29 3016 org.jboss.messaging.integration.transports.netty.NettyAcceptor
           [junit] 263: 370 2960 org.jboss.messaging.core.remoting.impl.invm.InVMConnectorFactory
           [junit] 264: 91 2912 java.util.TreeMap$Entry
           [junit] 265: 120 2880 java.lang.ThreadLocal$ThreadLocalMap
           [junit] 266: 72 2880 org.jboss.messaging.core.filter.impl.Token
           [junit] 267: 18 2872 [J
           [junit] 268: 87 2784 java.util.concurrent.ThreadPoolExecutor$Worker
           [junit] 269: 22 2728 [Z
           [junit] 270: 164 2624 org.jboss.netty.channel.group.DefaultChannelGroup$1
           [junit] 271: 8 2560 <typeArrayKlassKlass>
           [junit] 272: 29 2552 sun.nio.ch.ServerSocketChannelImpl
           [junit] 273: 106 2544 java.security.AccessControlContext
           [junit] 274: 106 2544 org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory
           [junit] 275: 105 2520 com.sun.jmx.mbeanserver.ConvertingMethod
           [junit] 276: 104 2496 com.sun.jmx.mbeanserver.PerInterface$MethodAndSig
           [junit] 277: 155 2480 org.jboss.messaging.core.server.impl.MessagingServerImpl$1
           [junit] 278: 44 2464 org.jboss.messaging.core.filter.impl.Operator
           [junit] 279: 77 2464 java.util.regex.Pattern$Curly
           [junit] 280: 34 2448 java.util.jar.JarFile$JarFileEntry
           [junit] 281: 37 2368 org.jboss.messaging.core.journal.impl.AIOSequentialFile
           [junit] 282: 60 2336 [Ljavax.management.MBeanOperationInfo;
           [junit] 283: 41 2296 org.jboss.messaging.core.persistence.impl.journal.JournalStorageManager
           [junit] 284: 95 2280 org.jboss.messaging.utils.UUID
           [junit] 285: 92 2208 sun.reflect.annotation.AnnotationInvocationHandler
           [junit] 286: 135 2160 java.util.IdentityHashMap$KeySet
           [junit] 287: 135 2160 org.jboss.netty.util.internal.MapBackedSet
           [junit] 288: 89 2136 java.util.concurrent.Executors$DefaultThreadFactory
           [junit] 289: 89 2136 java.lang.ref.ReferenceQueue
           [junit] 290: 4 2112 [Ljava.util.TimerTask;
           [junit] 291: 112 2008 [Lcom.sun.jmx.mbeanserver.OpenConverter;
           [junit] 292: 123 1968 java.util.concurrent.Executors$FinalizableDelegatedExecutorService
           [junit] 293: 117 1872 java.util.jar.Attributes$Name
           [junit] 294: 32 1792 javax.management.openmbean.OpenMBeanAttributeInfoSupport
           [junit] 295: 43 1720 sun.misc.URLClassPath$JarLoader
           [junit] 296: 43 1720 com.sun.jmx.mbeanserver.MBeanServerDelegateImpl
           [junit] 297: 43 1720 com.sun.jmx.interceptor.DefaultMBeanServerInterceptor
           [junit] 298: 41 1640 org.jboss.messaging.core.journal.impl.AIOSequentialFileFactory
           [junit] 299: 34 1632 java.util.jar.JarFile
           [junit] 300: 66 1584 org.jboss.messaging.core.settings.impl.Match
           [junit] 301: 66 1584 sun.reflect.generics.tree.SimpleClassTypeSignature
           [junit] 302: 98 1568 org.jboss.messaging.utils.TypedProperties$BooleanValue
           [junit] 303: 12 1536 org.jboss.messaging.core.server.cluster.impl.BridgeImpl
           [junit] 304: 38 1520 org.jboss.messaging.core.client.impl.ClientProducerImpl
           [junit] 305: 62 1488 org.jboss.messaging.core.filter.impl.FilterParser$JJCalls
           [junit] 306: 62 1488 java.util.concurrent.Semaphore$FairSync
           [junit] 307: 92 1472 org.jboss.messaging.core.journal.impl.NIOSequentialFileFactory
           [junit] 308: 61 1464 sun.reflect.NativeConstructorAccessorImpl
           [junit] 309: 29 1392 org.jboss.netty.channel.socket.nio.NioServerSocketChannel
           [junit] 310: 86 1376 com.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry
           [junit] 311: 43 1376 com.sun.jmx.mbeanserver.JmxMBeanServer
           [junit] 312: 17 1360 org.jboss.messaging.jms.client.JBossConnection
           [junit] 313: 12 1344 java.util.GregorianCalendar
           [junit] 314: 24 1344 java.security.Provider$Service
           [junit] 315: 41 1312 org.jboss.messaging.core.persistence.impl.journal.JournalStorageManager$BatchingIDGenerator
           [junit] 316: 41 1312 org.jboss.messaging.core.journal.impl.AIOSequentialFileFactory$ReuseBuffersController
           [junit] 317: 39 1248 java.util.zip.Inflater
           [junit] 318: 77 1232 sun.reflect.DelegatingConstructorAccessorImpl
           [junit] 319: 22 1232 java.nio.DirectByteBuffer
           [junit] 320: 75 1200 java.util.Collections$SingletonList
           [junit] 321: 21 1176 sun.reflect.DelegatingClassLoader
           [junit] 322: 72 1152 $Proxy5
           [junit] 323: 12 1152 sun.util.calendar.Gregorian$Date
           [junit] 324: 47 1128 java.security.Provider$ServiceKey
           [junit] 325: 10 1120 <klassKlass>
           [junit] 326: 28 1120 org.jboss.messaging.core.config.cluster.DivertConfiguration
           [junit] 327: 69 1104 java.util.regex.Pattern$Dot
           [junit] 328: 66 1080 [Lsun.reflect.generics.tree.TypeArgument;
           [junit] 329: 45 1080 javax.management.ObjectName$Property
           [junit] 330: 2 1056 [Ljava.lang.Integer;
           [junit] 331: 66 1056 com.sun.jmx.mbeanserver.NamedObject
           [junit] 332: 44 1048 [Lcom.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry;
           [junit] 333: 1 1040 [Ljava.lang.Byte;
           [junit] 334: 1 1040 [Ljava.lang.Long;
           [junit] 335: 49 1040 [Ljavax.management.ObjectName$Property;
           [junit] 336: 43 1032 com.sun.jmx.mbeanserver.ClassLoaderRepositorySupport
           [junit] 337: 43 1032 com.sun.jmx.mbeanserver.Repository
           [junit] 338: 32 1024 java.io.ObjectStreamField
           [junit] 339: 64 1024 java.util.regex.Pattern$Start
           [junit] 340: 14 1008 java.lang.reflect.Field
           [junit] 341: 21 1008 org.jboss.messaging.core.management.impl.ReplicationOperationInvokerImpl$1
           [junit] 342: 62 992 org.jboss.messaging.core.client.impl.ClientSessionPacketHandler
           [junit] 343: 41 984 org.jboss.messaging.utils.VariableLatch$CountSync
           [junit] 344: 20 960 sun.nio.cs.UTF_8$Encoder
           [junit] 345: 29 928 org.jboss.netty.bootstrap.ServerBootstrap
           [junit] 346: 29 928 sun.nio.ch.ServerSocketAdaptor
           [junit] 347: 23 920 org.jboss.messaging.core.remoting.impl.wireformat.SessionConsumerFlowCreditMessage
           [junit] 348: 19 912 java.util.TreeMap
           [junit] 349: 38 912 org.jboss.messaging.core.server.impl.QueueImpl$RefsOperation
           [junit] 350: 28 896 org.jboss.messaging.core.remoting.impl.wireformat.SessionXAStartMessage
           [junit] 351: 22 880 sun.misc.Cleaner
           [junit] 352: 34 816 org.apache.xerces.xni.QName
           [junit] 353: 17 816 org.jboss.messaging.core.remoting.impl.wireformat.SessionAcknowledgeMessage
           [junit] 354: 25 800 javax.management.ObjectName
           [junit] 355: 10 800 org.jboss.messaging.core.client.impl.LargeMessageBufferImpl
           [junit] 356: 11 792 org.jboss.messaging.core.messagecounter.MessageCounter
           [junit] 357: 49 784 [Ljavax.management.MBeanNotificationInfo;
           [junit] 358: 32 768 com.sun.jmx.mbeanserver.OpenConverter$IdentityConverter
           [junit] 359: 16 768 sun.util.calendar.ZoneInfo
           [junit] 360: 47 752 java.util.jar.Attributes
           [junit] 361: 45 728 [Ljava.lang.reflect.Constructor;
           [junit] 362: 91 728 java.lang.ref.ReferenceQueue$Lock
           [junit] 363: 15 720 java.util.Properties
           [junit] 364: 10 720 org.jboss.messaging.core.persistence.impl.nullpm.NullStorageLargeServerMessage
           [junit] 365: 15 720 javax.management.openmbean.ArrayType
           [junit] 366: 15 720 java.util.ResourceBundle$CacheKey
           [junit] 367: 30 720 java.security.Provider$EngineDescription
           [junit] 368: 29 696 org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink
           [junit] 369: 29 696 org.jboss.netty.bootstrap.ServerBootstrap$Binder
           [junit] 370: 29 696 org.jboss.netty.channel.socket.DefaultServerSocketChannelConfig
           [junit] 371: 29 696 org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory
           [junit] 372: 43 688 com.sun.jmx.mbeanserver.MBeanInstantiator
           [junit] 373: 43 688 com.sun.jmx.mbeanserver.SecureClassLoaderRepository
           [junit] 374: 42 672 org.jboss.messaging.core.server.impl.ServerSessionImpl$30
           [junit] 375: 21 672 java.util.Locale
           [junit] 376: 41 656 org.jboss.messaging.utils.VariableLatch
           [junit] 377: 41 656 org.jboss.messaging.core.asyncio.impl.TimedBuffer$CheckTimer
           [junit] 378: 82 656 org.jboss.messaging.core.journal.impl.Reclaimer
           [junit] 379: 41 656 org.jboss.messaging.core.journal.impl.AIOSequentialFileFactory$ReuseBuffersController$LocalBufferCallback
           [junit] 380: 2 640 <arrayKlassKlass>
           [junit] 381: 40 640 org.jboss.messaging.core.server.impl.ServerSessionImpl$22
           [junit] 382: 39 624 java.util.regex.Pattern$CharPropertyNames$1
           [junit] 383: 7 616 org.jboss.messaging.core.filter.impl.FilterParser
           [junit] 384: 25 600 org.jboss.messaging.core.filter.impl.Identifier
           [junit] 385: 15 600 java.util.ResourceBundle$BundleReference
           [junit] 386: 37 592 org.jboss.messaging.core.journal.impl.AIOSequentialFile$LocalBufferObserver
           [junit] 387: 37 592 java.lang.ProcessEnvironment$Variable
           [junit] 388: 37 592 java.lang.ProcessEnvironment$Value
           [junit] 389: 12 576 java.nio.HeapByteBuffer
           [junit] 390: 71 568 org.jboss.messaging.integration.transports.netty.NettyConnectorFactory
           [junit] 391: 14 560 javax.management.openmbean.SimpleType
           [junit] 392: 35 560 org.jboss.messaging.utils.TypedProperties$LongValue
           [junit] 393: 17 544 java.io.FilePermission
           [junit] 394: 17 544 java.security.CodeSource
           [junit] 395: 17 544 com.sun.jmx.mbeanserver.PerInterface
           [junit] 396: 17 544 com.sun.jmx.mbeanserver.WeakIdentityHashMap$IdentityWeakReference
           [junit] 397: 17 544 java.security.ProtectionDomain
           [junit] 398: 11 528 sun.nio.ch.FileChannelImpl
           [junit] 399: 22 528 java.nio.DirectByteBuffer$Deallocator
           [junit] 400: 32 512 org.jboss.messaging.core.client.impl.ClientConsumerImpl$Runner
           [junit] 401: 16 512 com.sun.jmx.mbeanserver.MXBeanSupport
           [junit] 402: 21 504 org.jboss.messaging.core.client.ClientRequestor
           [junit] 403: 31 496 org.jboss.messaging.core.cluster.impl.DiscoveryGroupImpl$UniqueIDEntry
           [junit] 404: 20 480 org.jboss.messaging.tests.integration.cluster.failover.PreserveOrderDuringFailoverTest$1
           [junit] 405: 30 480 [Ljavax.management.MBeanConstructorInfo;
           [junit] 406: 20 480 java.lang.RuntimePermission
           [junit] 407: 10 480 org.jboss.messaging.core.client.impl.LargeMessageBufferImpl$FileCache
           [junit] 408: 12 480 org.jboss.messaging.core.postoffice.impl.DuplicateIDCacheImpl
           [junit] 409: 15 480 java.util.ResourceBundle$LoaderReference
           [junit] 410: 10 480 sun.nio.cs.StreamEncoder
           [junit] 411: 10 480 org.jboss.messaging.tests.integration.jms.bridge.JMSBridgeReconnectionTest
           [junit] 412: 29 464 org.jboss.messaging.integration.transports.netty.NettyAcceptor$1
           [junit] 413: 29 464 sun.nio.ch.ServerSocketChannelImpl$1
           [junit] 414: 19 456 java.util.regex.Pattern$GroupTail
           [junit] 415: 4 448 java.util.TimerThread
           [junit] 416: 7 448 org.jboss.messaging.core.filter.impl.SimpleCharStream
           [junit] 417: 8 448 javax.management.openmbean.OpenMBeanParameterInfoSupport
           [junit] 418: 11 440 org.jboss.messaging.core.remoting.impl.wireformat.SessionSendContinuationMessage
           [junit] 419: 9 432 javax.management.openmbean.CompositeType
           [junit] 420: 27 432 org.jboss.messaging.core.server.impl.ServerSessionImpl$16
           [junit] 421: 18 432 java.security.Permissions
           [junit] 422: 1 416 [Lorg.apache.xerces.util.SymbolTable$Entry;
           [junit] 423: 13 416 javax.management.MBeanConstructorInfo
           [junit] 424: 13 416 com.arjuna.ats.arjuna.common.Uid
           [junit] 425: 17 408 java.security.BasicPermissionCollection
           [junit] 426: 10 400 org.jboss.messaging.jms.client.JBossBytesMessage
           [junit] 427: 5 400 org.jboss.messaging.core.server.cluster.impl.ClusterConnectionImpl
           [junit] 428: 7 392 org.jboss.messaging.core.filter.impl.FilterParserTokenManager
           [junit] 429: 5 376 [D
           [junit] 430: 23 368 java.security.Provider$UString
           [junit] 431: 15 360 org.jboss.messaging.core.cluster.DiscoveryEntry
           [junit] 432: 9 360 org.jboss.messaging.core.remoting.impl.wireformat.SessionCreateConsumerMessage
           [junit] 433: 22 352 org.jboss.messaging.core.remoting.impl.invm.InVMConnection$1
           [junit] 434: 6 336 org.jboss.messaging.core.server.cluster.impl.RemoteQueueBindingImpl
           [junit] 435: 7 336 [Lorg.jboss.messaging.core.filter.impl.FilterParser$JJCalls;
           [junit] 436: 21 336 sun.reflect.generics.tree.ClassTypeSignature
           [junit] 437: 10 320 org.jboss.messaging.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl
           [junit] 438: 5 320 java.net.PlainDatagramSocketImpl
           [junit] 439: 10 320 java.io.PrintStream
           [junit] 440: 8 320 java.io.BufferedWriter
           [junit] 441: 8 320 org.jboss.messaging.core.postoffice.QueueInfo
           [junit] 442: 13 312 [Ljava.io.ObjectStreamField;
           [junit] 443: 13 312 org.apache.log4j.Level
           [junit] 444: 19 304 java.util.regex.Pattern$GroupHead
           [junit] 445: 19 304 java.util.regex.Pattern$Slice
           [junit] 446: 2 304 org.jboss.messaging.jms.bridge.impl.JMSBridgeImpl
           [junit] 447: 12 288 [Ljava.util.regex.Pattern$Node;
           [junit] 448: 6 288 org.jboss.messaging.jms.server.impl.JMSServerManagerImpl
           [junit] 449: 12 288 java.io.FileOutputStream
           [junit] 450: 18 288 java.text.DateFormat$Field
           [junit] 451: 12 288 java.util.regex.Pattern$Branch
           [junit] 452: 9 288 java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
           [junit] 453: 2 288 com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction
           [junit] 454: 12 288 org.jboss.messaging.core.management.NotificationType
           [junit] 455: 18 288 org.jboss.messaging.core.server.cluster.impl.BridgeImpl$1
           [junit] 456: 9 288 java.util.concurrent.SynchronousQueue$TransferStack$SNode
           [junit] 457: 12 288 org.apache.xerces.util.XMLStringBuffer
           [junit] 458: 7 280 com.sun.jmx.mbeanserver.OpenConverter$CompositeConverter
           [junit] 459: 17 272 [Ljava.security.Principal;
           [junit] 460: 17 272 org.jboss.messaging.jms.client.JBossConnection$JMSFailureListener
           [junit] 461: 17 272 javax.management.StandardMBean
           [junit] 462: 17 272 java.io.FilePermissionCollection
           [junit] 463: 8 256 org.jboss.messaging.core.postoffice.impl.BindingsImpl
           [junit] 464: 8 256 sun.security.jca.ProviderConfig
           [junit] 465: 16 256 java.util.regex.Pattern$CharPropertyNames$4
           [junit] 466: 16 256 sun.reflect.BootstrapConstructorAccessorImpl
           [junit] 467: 4 256 org.jboss.messaging.core.cluster.impl.DiscoveryGroupImpl
           [junit] 468: 8 256 org.jboss.messaging.core.management.impl.QueueControlImpl
           [junit] 469: 8 256 sun.misc.ProxyGenerator$PrimitiveTypeInfo
           [junit] 470: 3 248 [Lorg.apache.xerces.xni.QName;
           [junit] 471: 6 240 org.jboss.messaging.core.deployers.impl.FileDeploymentManager
           [junit] 472: 10 240 org.jboss.messaging.tests.unit.util.InVMContext
           [junit] 473: 15 240 com.arjuna.ats.internal.arjuna.gandiva.inventory.InventoryList
           [junit] 474: 6 240 org.jboss.messaging.core.config.cluster.BroadcastGroupConfiguration
           [junit] 475: 5 240 sun.management.MemoryPoolImpl
           [junit] 476: 10 240 java.lang.StringCoding$StringEncoder
           [junit] 477: 10 240 java.io.OutputStreamWriter
           [junit] 478: 10 240 java.io.RandomAccessFile
           [junit] 479: 6 240 com.arjuna.common.internal.util.propertyservice.PropertyManagerImpl
           [junit] 480: 1 224 org.apache.xerces.impl.XMLNSDocumentScannerImpl
           [junit] 481: 14 224 java.util.Arrays$ArrayList
           [junit] 482: 14 224 org.jboss.netty.logging.JdkLogger
           [junit] 483: 14 224 org.jboss.netty.logging.InternalLoggerFactory$1
           [junit] 484: 7 224 sun.reflect.annotation.AnnotationType
           [junit] 485: 9 216 java.util.logging.Level
           [junit] 486: 13 208 java.lang.ThreadLocal
           [junit] 487: 7 200 [Ljava.lang.Boolean;
           [junit] 488: 5 200 sun.management.MemoryPoolImpl$PoolSensor
           [junit] 489: 5 200 sun.management.MemoryPoolImpl$CollectionSensor
           [junit] 490: 5 200 org.jboss.messaging.core.config.cluster.ClusterConnectionConfiguration
           [junit] 491: 8 192 org.jboss.messaging.core.management.impl.AddressControlImpl
           [junit] 492: 8 192 java.lang.OutOfMemoryError
           [junit] 493: 12 192 java.util.regex.Pattern$BranchConn
           [junit] 494: 6 192 sun.reflect.generics.repository.MethodRepository
           [junit] 495: 6 192 org.jboss.messaging.jms.server.impl.JMSServerDeployer
           [junit] 496: 8 192 org.jboss.messaging.core.postoffice.impl.AddressImpl
           [junit] 497: 8 192 org.jboss.messaging.jms.JBossQueue
           [junit] 498: 8 192 java.math.RoundingMode
           [junit] 499: 2 192 com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord
           [junit] 500: 4 192 java.net.MulticastSocket
           [junit] 501: 12 192 org.jboss.messaging.core.management.impl.AcceptorControlImpl
           [junit] 502: 3 192 sun.net.www.protocol.jar.URLJarFile
           [junit] 503: 3 192 java.security.SecureRandom
           [junit] 504: 8 192 org.apache.xerces.xni.XMLString
           [junit] 505: 1 192 org.apache.xerces.parsers.XIncludeAwareParserConfiguration
           [junit] 506: 10 184 [Ljava.lang.reflect.Type;
           [junit] 507: 1 176 org.apache.xerces.impl.dtd.XMLNSDTDValidator
           [junit] 508: 11 176 java.text.NumberFormat$Field
           [junit] 509: 11 176 org.apache.log4j.CategoryKey
           [junit] 510: 2 176 sun.security.provider.Sun
           [junit] 511: 11 176 org.jboss.messaging.core.messagecounter.MessageCounter$DayCounter
           [junit] 512: 7 176 [Ljava.lang.annotation.ElementType;
           [junit] 513: 11 176 sun.nio.ch.NativeThreadSet
           [junit] 514: 2 168 [Ljava.text.DateFormat$Field;
           [junit] 515: 7 168 org.jboss.messaging.core.filter.impl.FilterParser$LookaheadSuccess
           [junit] 516: 7 168 org.jboss.messaging.utils.SimpleStringReader
           [junit] 517: 7 168 com.sun.jmx.mbeanserver.StandardMBeanSupport
           [junit] 518: 7 168 org.jboss.messaging.core.filter.impl.FilterImpl
           [junit] 519: 7 168 org.apache.log4j.ProvisionNode
           [junit] 520: 7 168 java.util.Stack
           [junit] 521: 5 160 org.jboss.messaging.core.remoting.impl.wireformat.SessionXAEndMessage
           [junit] 522: 5 160 org.jboss.messaging.core.exception.MessagingException
           [junit] 523: 8 160 [Lorg.jboss.messaging.utils.SimpleString;
           [junit] 524: 4 160 java.util.PropertyResourceBundle
           [junit] 525: 4 160 org.apache.xerces.util.XMLAttributesImpl$Attribute
           [junit] 526: 4 160 org.apache.log4j.Logger
           [junit] 527: 10 160 java.util.Formatter$Flags
           [junit] 528: 5 160 java.lang.ClassLoader$NativeLibrary
           [junit] 529: 6 144 org.jboss.messaging.core.messagecounter.impl.MessageCounterManagerImpl$MessageCountersPinger
           [junit] 530: 3 144 javax.management.openmbean.OpenMBeanOperationInfoSupport
           [junit] 531: 1 144 org.apache.xerces.parsers.DOMParser
           [junit] 532: 9 144 sun.misc.MetaIndex
           [junit] 533: 9 144 [Lsun.reflect.generics.tree.FieldTypeSignature;
           [junit] 534: 1 144 org.apache.xerces.impl.XMLDTDScannerImpl
           [junit] 535: 6 144 sun.reflect.generics.tree.MethodTypeSignature
           [junit] 536: 9 144 org.jboss.messaging.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl$2
           [junit] 537: 6 144 sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl
           [junit] 538: 9 144 java.util.jar.Manifest
           [junit] 539: 8 136 [Lsun.reflect.generics.tree.FormalTypeParameter;
           [junit] 540: 4 128 org.jboss.messaging.jms.client.JBossSession
           [junit] 541: 8 128 sun.reflect.generics.factory.CoreReflectionFactory
           [junit] 542: 8 128 java.util.regex.Pattern$BitClass
           [junit] 543: 2 128 com.arjuna.common.internal.util.logging.LogImpl
           [junit] 544: 2 128 org.jboss.messaging.core.config.cluster.BridgeConfiguration
           [junit] 545: 8 128 java.util.regex.Pattern$CharPropertyNames$3
           [junit] 546: 2 128 org.jboss.messaging.core.remoting.impl.wireformat.CreateSessionMessage
           [junit] 547: 1 128 com.arjuna.ats.internal.arjuna.recovery.Listener
           [junit] 548: 1 128 org.apache.xerces.dom.DocumentImpl
           [junit] 549: 2 128 [Lorg.jboss.messaging.core.management.NotificationType;
           [junit] 550: 4 128 java.net.DatagramPacket
           [junit] 551: 8 128 java.lang.annotation.ElementType
           [junit] 552: 4 128 com.arjuna.ats.arjuna.common.Mutex
           [junit] 553: 3 120 sun.nio.cs.UTF_8$Decoder
           [junit] 554: 1 120 com.arjuna.ats.internal.arjuna.coordinator.ReaperThread
           [junit] 555: 5 120 sun.util.LocaleServiceProviderPool
           [junit] 556: 5 120 java.io.BufferedOutputStream
           [junit] 557: 5 120 org.apache.xerces.util.AugmentationsImpl$SmallContainer
           [junit] 558: 3 120 org.apache.tools.ant.taskdefs.optional.junit.FormatterElement
           [junit] 559: 1 112 java.util.logging.LogManager$Cleaner
           [junit] 560: 1 112 com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread
           [junit] 561: 14 112 org.jboss.messaging.utils.TypedProperties$NullValue
           [junit] 562: 7 112 java.util.regex.Pattern$Dollar
           [junit] 563: 7 112 java.util.LinkedHashSet
           [junit] 564: 1 112 org.apache.xerces.impl.XMLEntityManager
           [junit] 565: 1 112 org.jboss.messaging.core.remoting.impl.ChannelImpl$1
           [junit] 566: 7 112 org.jboss.messaging.core.client.impl.ConnectionManagerImpl$ConnectionEntry
           [junit] 567: 7 112 java.util.regex.Pattern$Begin
           [junit] 568: 2 112 org.jboss.messaging.jms.client.JBossMessageProducer
           [junit] 569: 2 112 java.io.ExpiringCache$1
           [junit] 570: 1 104 java.lang.ref.Reference$ReferenceHandler
           [junit] 571: 1 104 java.lang.ref.Finalizer$FinalizerThread
           [junit] 572: 1 104 com.arjuna.ats.arjuna.coordinator.TxControl$Shutdown
           [junit] 573: 3 96 sun.reflect.generics.reflectiveObjects.TypeVariableImpl
           [junit] 574: 3 96 sun.misc.URLClassPath
           [junit] 575: 6 96 org.jboss.messaging.jms.server.management.impl.JMSManagementServiceImpl
           [junit] 576: 2 96 javax.management.openmbean.TabularType
           [junit] 577: 6 96 java.util.regex.Pattern$Single
           [junit] 578: 4 96 org.jboss.messaging.core.config.cluster.QueueConfiguration
           [junit] 579: 6 96 java.util.Collections$UnmodifiableRandomAccessList
           [junit] 580: 6 96 org.jboss.messaging.core.filter.impl.RegExp
           [junit] 581: 6 96 java.util.regex.Pattern$CharPropertyNames$2
           [junit] 582: 2 96 com.arjuna.common.internal.util.logging.LogNoi18nImpl
           [junit] 583: 6 96 [Lsun.reflect.generics.tree.TypeSignature;
           [junit] 584: 4 96 com.arjuna.ats.arjuna.xa.XID
           [junit] 585: 3 96 org.jboss.messaging.core.remoting.impl.wireformat.MessagingExceptionMessage
           [junit] 586: 3 96 org.apache.xerces.util.XMLResourceIdentifierImpl
           [junit] 587: 2 96 com.sun.jmx.mbeanserver.MBeanIntrospector$PerInterfaceMap
           [junit] 588: 2 96 com.sun.jmx.mbeanserver.MBeanIntrospector$MBeanInfoMap
           [junit] 589: 6 96 sun.misc.URLClassPath$FileLoader
           [junit] 590: 4 96 java.util.regex.Pattern$BnM
           [junit] 591: 3 96 javax.jms.JMSException
           [junit] 592: 6 96 org.jboss.messaging.jms.server.management.impl.JMSServerControlImpl$NotificationType
           [junit] 593: 6 96 java.util.regex.Pattern$Ctype
           [junit] 594: 6 96 org.jboss.messaging.jms.bridge.impl.JMSBridgeImpl$BridgeExceptionListener
           [junit] 595: 6 96 org.jboss.messaging.core.transaction.Transaction$State
           [junit] 596: 4 96 com.arjuna.ats.internal.jta.xa.TxInfo
           [junit] 597: 6 96 org.jboss.messaging.core.server.cluster.impl.ClusterConnectionImpl$MessageFlowRecordImpl$4
           [junit] 598: 4 96 com.sun.jmx.mbeanserver.OpenConverter$ArrayConverter
           [junit] 599: 2 96 org.jboss.messaging.jms.client.JBossMessageConsumer
           [junit] 600: 6 96 java.util.concurrent.Executors$DelegatedScheduledExecutorService
           [junit] 601: 4 96 com.arjuna.ats.arjuna.StateManagerAttribute
           [junit] 602: 1 96 org.apache.xerces.impl.dtd.XMLDTDProcessor
           [junit] 603: 4 96 java.util.Timer
           [junit] 604: 3 96 javax.management.MBeanNotificationInfo
           [junit] 605: 6 96 sun.reflect.generics.scope.MethodScope
           [junit] 606: 2 96 java.text.ChoiceFormat
           [junit] 607: 1 88 sun.security.jca.ProviderList$1
           [junit] 608: 1 88 java.util.logging.FileHandler
           [junit] 609: 2 80 org.jboss.messaging.core.remoting.impl.wireformat.CreateQueueMessage
           [junit] 610: 5 80 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4
           [junit] 611: 10 80 org.jboss.messaging.tests.unit.util.InVMNameParser
           [junit] 612: 5 80 org.jboss.messaging.core.server.impl.ServerSessionImpl$11
           [junit] 613: 2 80 com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple
           [junit] 614: 5 80 org.apache.xerces.util.AugmentationsImpl
           [junit] 615: 2 80 org.apache.xerces.impl.dtd.XMLSimpleType
           [junit] 616: 5 80 org.jboss.messaging.core.server.impl.ServerSessionImpl$6
           [junit] 617: 5 80 $Proxy6
           [junit] 618: 5 80 $Proxy0
           [junit] 619: 5 80 [Ljava.lang.annotation.Annotation;
           [junit] 620: 2 80 org.apache.xerces.impl.dtd.XMLEntityDecl
           [junit] 621: 1 80 org.apache.tools.ant.taskdefs.optional.junit.JUnitTest
           [junit] 622: 5 80 $Proxy2
           [junit] 623: 3 72 org.jboss.messaging.jms.bridge.QualityOfServiceMode
           [junit] 624: 1 72 sun.misc.Launcher$AppClassLoader
           [junit] 625: 1 72 [Ljavax.management.openmbean.SimpleType;
           [junit] 626: 3 72 java.util.regex.Pattern$2
           [junit] 627: 3 72 com.sun.jmx.mbeanserver.OpenConverter$EnumConverter
           [junit] 628: 3 72 org.jboss.messaging.jms.server.management.impl.JMSQueueControlImpl
           [junit] 629: 1 72 sun.misc.Launcher$ExtClassLoader
           [junit] 630: 4 64 java.util.Timer$1
           [junit] 631: 1 64 java.util.logging.LogManager$RootLogger
           [junit] 632: 2 64 com.sun.jmx.mbeanserver.OpenConverter$CollectionConverter
           [junit] 633: 2 64 org.jboss.messaging.jms.client.JBossConnectionFactory
           [junit] 634: 2 64 com.sun.jmx.mbeanserver.OpenConverter$TabularConverter
           [junit] 635: 2 64 sun.management.GarbageCollectorImpl
           [junit] 636: 2 64 [Lcom.arjuna.ats.arjuna.coordinator.ActionInfo;
           [junit] 637: 4 64 java.util.concurrent.ConcurrentHashMap$EntrySet
           [junit] 638: 2 64 [Lsun.security.jca.ProviderConfig;
           [junit] 639: 2 64 com.arjuna.ats.internal.arjuna.objectstore.HashedActionStore
           [junit] 640: 4 64 org.jboss.netty.channel.ChannelState
           [junit] 641: 2 64 org.jboss.messaging.core.remoting.impl.wireformat.SessionXASetTimeoutResponseMessage
           [junit] 642: 4 64 com.arjuna.common.internal.util.logging.jakarta.Log4JLogger
           [junit] 643: 4 64 com.arjuna.ats.arjuna.gandiva.ClassName
           [junit] 644: 2 64 java.io.ExpiringCache
           [junit] 645: 4 64 org.apache.tools.ant.util.TeeOutputStream
           [junit] 646: 4 64 com.arjuna.ats.jta.xa.XidImple
           [junit] 647: 4 64 com.arjuna.common.internal.util.logging.jakarta.JakartaLogger
           [junit] 648: 2 64 sun.reflect.generics.repository.ClassRepository
           [junit] 649: 1 64 java.text.SimpleDateFormat
           [junit] 650: 4 64 java.util.TaskQueue
           [junit] 651: 4 64 java.io.FileInputStream
           [junit] 652: 1 56 [F
           [junit] 653: 1 56 sun.security.provider.NativePRNG$RandomIO
           [junit] 654: 1 56 sun.security.provider.SHA
           [junit] 655: 1 56 [Ljava.lang.reflect.Field;
           [junit] 656: 1 56 java.util.ResourceBundle$RBClassLoader
           [junit] 657: 1 56 org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner
           [junit] 658: 3 56 [Ljava.io.File;
           [junit] 659: 1 56 java.util.logging.ConsoleHandler
           [junit] 660: 1 56 [Ljava.text.Format;
           [junit] 661: 1 48 [[Ljava.lang.Object;
           [junit] 662: 3 48 java.nio.charset.CodingErrorAction
           [junit] 663: 3 48 java.text.AttributedCharacterIterator$Attribute
           [junit] 664: 2 48 java.util.regex.Pattern$Ques
           [junit] 665: 1 48 org.apache.commons.logging.impl.WeakHashtable
           [junit] 666: 1 48 com.arjuna.ats.arjuna.coordinator.TransactionReaper
           [junit] 667: 1 48 com.sun.jmx.mbeanserver.OpenConverter$ConverterMap
           [junit] 668: 2 48 com.arjuna.ats.arjuna.coordinator.ActionHierarchy
           [junit] 669: 3 48 org.jboss.messaging.tests.integration.jms.bridge.BridgeTestBase$1
           [junit] 670: 2 48 sun.security.jca.ProviderList
           [junit] 671: 3 48 org.jboss.messaging.tests.integration.jms.bridge.BridgeTestBase$2
           [junit] 672: 3 48 org.jboss.messaging.jms.server.management.impl.TopicControlImpl$DurabilityType
           [junit] 673: 3 48 org.jboss.messaging.core.server.impl.ServerSessionImpl$1
           [junit] 674: 3 48 org.jboss.messaging.tests.integration.jms.bridge.BridgeTestBase$3
           [junit] 675: 1 48 java.io.BufferedReader
           [junit] 676: 3 48 org.jboss.messaging.tests.integration.jms.bridge.BridgeTestBase$4
           [junit] 677: 3 48 sun.reflect.generics.tree.FormalTypeParameter
           [junit] 678: 2 48 org.jboss.messaging.jms.JBossTopic
           [junit] 679: 3 48 org.jboss.messaging.core.postoffice.BindingType
           [junit] 680: 2 48 org.jboss.messaging.jms.server.management.impl.JMSServerControlImpl
           [junit] 681: 2 48 org.jboss.messaging.core.server.impl.MessagingServerPacketHandler$1
           [junit] 682: 2 48 java.util.concurrent.CountDownLatch$Sync
           [junit] 683: 3 48 java.util.Hashtable$EntrySet
           [junit] 684: 2 48 java.lang.ref.ReferenceQueue$Null
           [junit] 685: 1 48 [Ljava.math.RoundingMode;
           [junit] 686: 3 48 java.util.Collections$SynchronizedSet
           [junit] 687: 2 48 org.jboss.messaging.tests.integration.jms.server.management.NullInitialCont


          • 2. Re: Lots of largemessages
            timfox

            I logged out the number of QueueImpl instances with an AtomicLong, incrementing in the constructor and decrementing in a finalize.

            The queue count stays pretty low until LargeMessageMultiThreadFailoverTest where it starts to go through the roof.

            • 3. Re: Lots of largemessages
              clebert.suconic

              I created a few reports on the jboss-profiler, where you can navigate through the memory, following references, using a simple web interface.

              I ran the testsuite with the changes and I couldn't find any leaks.


              I also run without the changes, and the only leak I found was the Pinger. (which is fixed now)
              I don't think QueueImpl was leaking (as I tested it without the fixes).


              However, there is one thing that is puzzing me.


              If I changed MultiThreadRandomFailoverTest::tearDown as follows:

              protected void tearDown() throws Exception
               {
               for (int i = 0 ; i < 10; i ++)
               log.info("************* Ending test " + getName());
              
               if (liveServer != null && liveServer.isStarted())
               {
               liveServer.stop();
               }
               liveServer = null;
               if (backupServer != null && backupServer.isStarted())
               {
               backupServer.stop();
               }
               backupServer = null;
              
               super.tearDown();
              
               }
              


              .... it seems that Queues and ServerMessages would be GCed faster. I don't know.. It seems that Junit is holding a couple of instances of a JUnit test until the suite (the JUnit class) is completed.. and it seemed that clearing huge objects on the tearDown would help GC.

              Maybe I'm mistaken, and maybe the tearDown change just helped my reports.. but since it doesn't hurt.. I will keep the set null on the tearDown.


              I will confirm that tomorrow. if that is true, we will have to start clearing big objects on tearDown in our code.

              • 4. Re: Lots of largemessages
                timfox

                 

                "clebert.suconic@jboss.com" wrote:
                I created a few reports on the jboss-profiler, where you can navigate through the memory, following references, using a simple web interface.

                I ran the testsuite with the changes and I couldn't find any leaks.


                I also run without the changes, and the only leak I found was the Pinger. (which is fixed now)
                I don't think QueueImpl was leaking (as I tested it without the fixes).




                You couldn't find the leaks because I fixed them (as discussed on IRC) in revision 7704 last night ;)


                • 5. Re: Lots of largemessages
                  clebert.suconic

                   

                  "timfox" wrote:
                  "clebert.suconic@jboss.com" wrote:

                  I also run without the changes, and the only leak I found was the Pinger. (which is fixed now)
                  I don't think QueueImpl was leaking (as I tested it without the fixes).



                  You couldn't find the leaks because I fixed them (as discussed on IRC) in revision 7704 last night ;)


                  I ran the profiler without your changes, and I couldn't see QueueImpl leaking. I can confirm that Pinger was leaking.

                  And indeed I couldn't find any leaks after your changes.

                  • 6. Re: Lots of largemessages
                    timfox

                    Well I saw a lot of queue instances which were not GC'd when I run System.gc() and Runtime.runFinalizers() several times.

                    Those instances didn't occur after my fixes.

                    • 7. Re: Lots of largemessages
                      clebert.suconic

                      Yeah.. ok...

                      I was wondering if those were because of changes you had at your workspace. I couldn't replicate that.


                      it's not important... It's fixed anyway now :-)

                      • 8. JUnit holds the test references until the end of the suite c
                        clebert.suconic

                        This is to confirm what I suspected here


                        Say that you have this test:

                        public class SomeTest extends TestCase
                        {
                        
                         // Constants -----------------------------------------------------
                        
                         // Attributes ----------------------------------------------------
                        
                         Server server;
                        
                         // Static --------------------------------------------------------
                        
                         // Constructors --------------------------------------------------
                        
                         // Public --------------------------------------------------------
                        
                         public void setUp()
                         {
                         ts = new SomeClass();
                         }
                        
                         public void testOne() throws Exception
                         {
                         System.gc();
                         Thread.sleep(1000);
                         }
                        
                         public void testOneA() throws Exception
                         {
                         System.gc();
                         Thread.sleep(1000);
                         }
                        


                        JUnit will hold one reference of the Server for each method on the test. Those referencess will be released only at the end of the class.

                        So it would be great if as a best practice, we set any big object to null on tearDown. (Things like servers, connections... etc).

                        • 9. Re: Lots of largemessages
                          timfox

                          I don't mean to spoil your discovery, but isn't that kind of obvious?

                          All JUnit does is call:

                          setUp()
                          testFoo()
                          tearDown()

                          Once for each test.

                          So if you set a reference only in setUp(), the next time it gets set again will be in the next time setUp() is called.

                          When one reference overwrites the other, then the old reference will be eligible for garbage collection.

                          That's nothing special about how JUnit works, it's just Java references and gc.

                          • 10. Re: Lots of largemessages
                            timfox

                            I just did a test, both in eclipse and junit via ant gave the same results.

                            The following test class

                            public class MyTest extends TestCase
                            {
                             private static final Logger log = Logger.getLogger(MyTest.class);
                            
                             public MyTest()
                             {
                             log.info("Constructor");
                             }
                            
                             protected void setUp()
                             {
                             log.info("Setup");
                             }
                            
                             protected void tearDown()
                             {
                             log.info("TearDown");
                             }
                            
                             public void test1()
                             {
                             log.info("test1");
                             }
                            
                             public void test2()
                             {
                             log.info("test2");
                             }
                            
                             public void test3()
                             {
                             log.info("test3");
                             }
                            
                            
                            }
                            


                            Produces the following output:

                            [main] 09:54:20,814 INFO [org.jboss.messaging.tests.integration.MyTest] Constructor
                            [main] 09:54:21,185 INFO [org.jboss.messaging.tests.integration.MyTest] Constructor
                            [main] 09:54:21,186 INFO [org.jboss.messaging.tests.integration.MyTest] Constructor
                            [main] 09:54:21,231 INFO [org.jboss.messaging.tests.integration.MyTest] Setup
                            [main] 09:54:21,232 INFO [org.jboss.messaging.tests.integration.MyTest] test1
                            [main] 09:54:21,233 INFO [org.jboss.messaging.tests.integration.MyTest] TearDown
                            [main] 09:54:21,233 INFO [org.jboss.messaging.tests.integration.MyTest] Setup
                            [main] 09:54:21,234 INFO [org.jboss.messaging.tests.integration.MyTest] test2
                            [main] 09:54:21,234 INFO [org.jboss.messaging.tests.integration.MyTest] TearDown
                            [main] 09:54:21,235 INFO [org.jboss.messaging.tests.integration.MyTest] Setup
                            [main] 09:54:21,235 INFO [org.jboss.messaging.tests.integration.MyTest] test3
                            [main] 09:54:21,235 INFO [org.jboss.messaging.tests.integration.MyTest] TearDown
                            


                            So JUnit instantiates a new instance of the test class for running each test (I didn't know that), but it does that at the beginning.

                            Not sure if that was the point you were trying to make.

                            But if so, then yes we need to set objects to null in teardown.

                            • 11. Re: Lots of largemessages
                              timfox

                              I am in the process of making sure all members variables in tests are set to null in tearDown().

                              So please, don't anyone else do it.

                              • 12. Re: Lots of largemessages
                                timfox
                                • 13. Re: Lots of largemessages
                                  clebert.suconic

                                   

                                  Not sure if that was the point you were trying to make.

                                  But if so, then yes we need to set objects to null in teardown.



                                  I didn't check / know about the order on Constructors... but yes.. the point I was trying to make is JUnit will hold one instance of the class per test method.

                                  So we must make sure we set fields to null on the test.


                                  We could maybe make some simple and generic code on UnitTestCase::tearDown, using reflection. But we would need to make sure we all called super.tearDown at the end of our tearDown methods.