GC hell ?

[vettori]

Hi,

I'm running JBoss 4.0.5 on Redhat Linux 64. The server has two dual core processor and ten gigabyte of ram.

I start JBoss with :

-Xms4096m -Xmx5120m -XX:PermSize=128m -XX:MaxPermSize=256m -XX:NewSize=2048m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Xloggc:/tmp/gc.log

I have the following problem. At server start the load is very small and the gc is called not very often, for example :

4.356: [GC 314590K->8523K(3932160K), 0.0177040 secs]
4.374: [Full GC 8523K->8318K(3932160K), 0.0849990 secs]
47.272: [GC 1581182K->59216K(3932160K), 0.0860340 secs]
314.685: [GC 1632080K->61497K(3932160K), 0.0628620 secs]
834.124: [GC 1634361K->66492K(3932160K), 0.0693030 secs]
1235.281: [GC 1639356K->66265K(3932160K), 0.0582550 secs]
1519.406: [GC 1639129K->65943K(4109248K), 0.0779130 secs]
2211.883: [GC 1815895K->72202K(3932160K), 0.0790670 secs]
2478.914: [GC 1822154K->75776K(4103296K), 0.0777540 secs]
2725.280: [GC 1993984K->75759K(4031872K), 0.0193570 secs]
2794.302: [GC 1993967K->74001K(4102336K), 0.0129670 secs]
3026.186: [GC 1989009K->77645K(4017088K), 0.0134620 secs]
3604.464: [GC 737189K->79960K(4102464K), 0.0168140 secs]
3604.481: [Full GC 79960K->79901K(4102464K), 0.3914900 secs]
4350.582: [GC 1993373K->86781K(4102464K), 0.0103160 secs]
4634.533: [GC 2000253K->88447K(4105664K), 0.0127350 secs]
5441.497: [GC 2006655K->96467K(4104000K), 0.0134960 secs]
5736.696: [GC 2014675K->94458K(4110528K), 0.0101000 secs]
6554.140: [GC 2022074K->100914K(4108544K), 0.0102490 secs]
6952.151: [GC 2028530K->104049K(4116992K), 0.0095100 secs]


Than after some time that I found depends on the value of NewSize parameter the GC is called a lot of times and the load reach 100% and sometimes 200%. For example :

143806.289: [GC 2258900K->162852K(4193728K), 0.0046960 secs]
143817.558: [GC 2258916K->162884K(4193792K), 0.0044520 secs]
143829.238: [GC 2259012K->162916K(4193792K), 0.0043090 secs]
143840.359: [GC 2259044K->163174K(4193728K), 0.0064450 secs]
143851.252: [GC 2259302K->164061K(4193792K), 0.0050780 secs]
143862.932: [GC 2260189K->164197K(4193152K), 0.0049390 secs]
143874.515: [GC 2259685K->164197K(4193472K), 0.0042450 secs]
143885.381: [GC 2259685K->164773K(4193216K), 0.0052980 secs]
143896.948: [GC 2260005K->164797K(4193344K), 0.0044450 secs]
143908.172: [GC 2260029K->164805K(4193408K), 0.0060260 secs]
143917.971: [GC 2260165K->164893K(4193408K), 0.0053640 secs]
143928.683: [GC 2260253K->164933K(4193536K), 0.0047570 secs]
143939.384: [GC 2260485K->164909K(4193472K), 0.0039900 secs]
143950.040: [GC 2260461K->164909K(4193600K), 0.0040570 secs]
143960.082: [GC 2260589K->164981K(4193536K), 0.0052420 secs]
143970.591: [GC 2260661K->164997K(4193664K), 0.0043150 secs]
143979.495: [GC 1933704K->167118K(4193600K), 0.0110230 secs]

The server is responsive even under such load but I really want to understand why this happens and how to solve this.

The applications makes heavy use of persistence and caches a lot of data (in web sessions) that it's probably not used very frequently but it has a limit on the cache size so I think it's not consuming all the memory.

Any ideas ?

How should I read the gc log ? I don't know how GC works, I tired to read the documentation but it's not very clear to me.

Thank you

[peterj]

Do you really need a 4GB heap with a 2GB young generation? Bigger is not necessarily better when it comes to Java heaps. It does look like you are freeing up 2GB of garbage every 10 seconds, that is a lot of garbage. Some code must be generating that garbage. It also looks like your working set size is around 160MB. I would try setting Xms/x to 900MB and Max/NewSize to 300MB. Or perhaps 1200M/400M. Of course, at the rate your app is generating garbage that will cause GCs to occur more often.

[vettori]

Thanks.

How can I look at what is consuming that 2GB ?
It's not the data I cache in the session since that lives more than ten seconds... (a lot more).

Thank you

[peterj]

Unfortunately I do not know of any way of asking the JVM to state what objects it is removing as part of a GC. Actually, during a GC the JVM considers only live objects - it never even looks at dead objects that it collects.

But you can take a heap dump - see http://java.sun.com/javase/6/docs/technotes/tools/share/jhat.html.

If you can take a dump before a GC, that should give some idea of what is in the heap.

[vettori]

I tried hat (jdk5) but it's hard to understand...

I have a lot of char[], String and Object[]. But i see the number of classes not their size! So I can't understand who's eating the memory.

As I suspected I have a limited number of mine classes.

Any other hint ?

[peterj]

Sorry, but you have reached the limit of what I can do via a forum. Perhaps you should look at a commercial tool, such as JProbe, or consider paying someone to analyze your system.

[vettori]

I tried to profile the memory with various tool but I can't see anything.

I've tried to use jrockit and set it with 1gb newsize... The problem is still there. It's somethink like if a process on "timer" starts eating all the memory.

Does the default configuration of jboss have some process that starts to do something after 13 hours of boot ? Or does it exist some process that starts on a specific condition ?

If i restart jboss the problem is solved for some hours and then it comes back. So it can't be something related to the applications client, it must be something else!

It's not a memory leak because the system keeps running and the occupied memory varies (it's not always growing).

Any hint ?

[INFO ][memory ] 3606.617-3607.456: GC 1300258K->84978K (5242880K), 838.964 ms
[INFO ][memory ] 7207.491-7208.338: GC 1015477K->93056K (5242880K), 846.489 ms
[INFO ][memory ] 10808.386-10809.143: GC 780919K->93621K (5242880K), 756.863 ms
[INFO ][memory ] 14409.166-14410.062: GC 859083K->102183K (5242880K), 896.407 ms
[INFO ][memory ] 18010.091-18010.828: GC 738553K->93361K (5242880K), 736.318 ms
[INFO ][memory ] 21610.866-21611.479: GC 783016K->93796K (5242880K), 612.593 ms
[INFO ][memory ] 25211.527-25212.377: GC 877385K->97377K (5242880K), 850.318 ms
[INFO ][memory ] 28355.565-28355.750: GC 1182130K->125094K (5242880K), 185.104 ms
[INFO ][memory ] 31190.734-31191.483: GC 1086929K->121687K (5242880K), 749.435 ms
[INFO ][memory ] 33812.292-33812.424: GC 1169857K->136251K (5242880K), 132.776 ms
[INFO ][memory ] 36163.268-36163.471: GC 1147925K->131655K (5242880K), 202.951 ms
[INFO ][memory ] 38463.711: parallel nursery GC 1614098K->831714K (5242880K), 494.306 ms
[INFO ][memory ] 38480.740: parallel nursery GC 1613714K->830594K (5242880K), 414.318 ms
[INFO ][memory ] 38496.893: parallel nursery GC 1613394K->831394K (5242880K), 432.489 ms
[INFO ][memory ] 38519.054: parallel nursery GC 1613394K->830594K (5242880K), 423.293 ms
[INFO ][memory ] 39757.593-39757.751: GC 1547900K->126597K (5242880K), 157.734 ms
[INFO ][memory ] 41986.769-41986.913: GC 1187874K->140813K (5242880K), 144.724 ms
[INFO ][memory ] 44345.904-44346.064: GC 1261056K->153070K (5242880K), 160.614 ms
[INFO ][memory ] 45160.828: parallel nursery GC 1599799K->824135K (5242880K), 168.815 ms
[INFO ][memory ] 45163.988: parallel nursery GC 1599735K->821287K (5242880K), 37.270 ms
[INFO ][memory ] 45166.991: parallel nursery GC 1599287K->823368K (5242880K), 42.019 ms
[INFO ][memory ] 45169.997: parallel nursery GC 1598968K->820968K (5242880K), 38.009 ms
[INFO ][memory ] 45173.002: parallel nursery GC 1598968K->823260K (5242880K), 35.458 ms
[INFO ][memory ] 45176.005: parallel nursery GC 1598860K->820860K (5242880K), 37.036 ms
[INFO ][memory ] 45179.009: parallel nursery GC 1598860K->823224K (5242880K), 35.218 ms
[INFO ][memory ] 45182.002: parallel nursery GC 1598824K->820720K (5242880K), 37.749 ms
[INFO ][memory ] 45185.004: parallel nursery GC 1598720K->823120K (5242880K), 35.726 ms
[INFO ][memory ] 45187.998: parallel nursery GC 1598720K->820592K (5242880K), 37.297 ms
[INFO ][memory ] 45190.995: parallel nursery GC 1598592K->822992K (5242880K), 35.212 ms
[INFO ][memory ] 45193.988: parallel nursery GC 1598592K->820592K (5242880K), 37.718 ms
[INFO ][memory ] 45196.985: parallel nursery GC 1598592K->822868K (5242880K), 35.630 ms
[INFO ][memory ] 45199.978: parallel nursery GC 1598468K->820468K (5242880K), 37.431 ms
[INFO ][memory ] 45202.982: parallel nursery GC 1598468K->822681K (5242880K), 35.342 ms
[INFO ][memory ] 45205.973: parallel nursery GC 1598281K->820281K (5242880K), 37.172 ms
[INFO ][memory ] 45208.925: parallel nursery GC 1598281K->822529K (5242880K), 39.648 ms
[INFO ][memory ] 45211.904: parallel nursery GC 1598129K->819688K (5242880K), 41.357 ms

[vettori]

First time today, after restarting the server it has started early to eat memory.

What can produce 1 GB of garbage in three seconds ?????

[INFO ][memory ] 6.383-6.580: GC 519614K->8472K (5242880K), 197.359 ms
[INFO ][memory ] 1367.161: parallel nursery GC 1475865K->689001K (5242880K), 315.806 ms
[INFO ][memory ] 1371.696: parallel nursery GC 1475801K->688809K (5242880K), 36.619 ms
[INFO ][memory ] 1375.094: parallel nursery GC 1475609K->688425K (5242880K), 35.300 ms
[INFO ][memory ] 1378.180: parallel nursery GC 1475225K->687977K (5242880K), 31.062 ms
[INFO ][memory ] 1381.143: parallel nursery GC 1474777K->687977K (5242880K), 30.965 ms
[INFO ][memory ] 1384.020: parallel nursery GC 1474777K->687786K (5242880K), 34.746 ms
[INFO ][memory ] 1386.890: parallel nursery GC 1474586K->687530K (5242880K), 31.365 ms

[vettori]

Here is the thread dump. Is something strange to you jboss experts ???

Thanks

===== FULL THREAD DUMP ===============
Wed Jun 27 16:01:17 2007
BEA JRockit(R) R27.2.0-131-78843-1.5.0_10-20070320-1507-linux-x86_64

"Thread-0" id=12 idx=0x4 tid=29535 prio=5 alive, in native, daemon

"(Signal Handler)" id=2 idx=0x8 tid=29536 prio=5 alive, in native, daemon

"(Code Generation Thread 1)" id=3 idx=0xc tid=29537 prio=5 alive, in native, native_waiting, daemon

"(Code Optimization Thread 1)" id=4 idx=0x10 tid=29538 prio=10 alive, in native, native_waiting, daemon

"(Code Optimization Thread 2)" id=5 idx=0x14 tid=29539 prio=10 alive, in native, native_waiting, daemon

"(GC Main Thread)" id=6 idx=0x18 tid=29540 prio=5 alive, in native, daemon

"(GC Worker Thread 1)" id=? idx=0x1c tid=29541 prio=5 alive, in native, native_waiting, daemon

"(GC Worker Thread 2)" id=? idx=0x20 tid=29542 prio=5 alive, in native, native_waiting, daemon

"(GC Worker Thread 3)" id=? idx=0x24 tid=29543 prio=5 alive, in native, native_waiting, daemon

"(GC Worker Thread 4)" id=? idx=0x28 tid=29544 prio=5 alive, in native, native_waiting, daemon

"(VM Periodic Task)" id=7 idx=0x2c tid=29545 prio=10 alive, in native, daemon

"Finalizer" id=8 idx=0x30 tid=29546 prio=8 alive, in native, native_waiting, daemon
 at jrockit/memory/Finalizer.getPendingFinalizee(Z)Ljava/lang/Object;(Native Method)
 at jrockit/memory/Finalizer.access$100(Z)Ljava/lang/Object;(Unknown Source)
 at jrockit/memory/Finalizer$4.run()V(Unknown Source)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Reference Handler" id=9 idx=0x34 tid=29547 prio=10 alive, in native, native_waiting, daemon
 at java/lang/ref/Reference.getPending()Ljava/lang/ref/Reference;(Native Method)
 at java/lang/ref/Reference.access$000()Ljava/lang/ref/Reference;(Unknown Source)
 at java/lang/ref/Reference$ReferenceHandler.run()V(Unknown Source)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"(Sensor Event Thread)" id=10 idx=0x38 tid=29548 prio=5 alive, in native, daemon

"http-0.0.0.0-8080-3" id=89 idx=0x3c tid=29628 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae5f7c538[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae5f7c538[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Timer-0" id=15 idx=0x40 tid=29550 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/TaskQueue@0x2a95c01608[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/util/TimerThread.mainLoop(Timer.java:509)
 ^-- Lock released while waiting: java/util/TaskQueue@0x2a95c01608[fat lock]
 at java/util/TimerThread.run(Timer.java:462)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"ScannerThread" id=16 idx=0x44 tid=29551 prio=5 alive, in native, sleeping, native_waiting, daemon
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/deployment/scanner/AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:280)
 at org/jboss/deployment/scanner/AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SocketServerInvoker#0-3873" id=18 idx=0x48 tid=29552 prio=5 alive, in native
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2a95d54f58[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at org/jboss/remoting/transport/socket/SocketServerInvoker.run(SocketServerInvoker.java:383)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SubscriptionWatchDog" id=19 idx=0x4c tid=29553 prio=5 alive, in native, sleeping, native_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ws/eventing/mgmt/SubscriptionManager$WatchDog.run(SubscriptionManager.java:535)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"RMI TCP Accept-1098" id=20 idx=0x50 tid=29554 prio=5 alive, in native, daemon
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2a96107ff0[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at sun/rmi/transport/tcp/TCPTransport.run(TCPTransport.java:340)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Timer-1" id=21 idx=0x54 tid=29555 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/TaskQueue@0x2a96108d80[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at java/util/TimerThread.mainLoop(Timer.java:483)
 ^-- Lock released while waiting: java/util/TaskQueue@0x2a96108d80[fat lock]
 at java/util/TimerThread.run(Timer.java:462)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"RMI Reaper" id=22 idx=0x58 tid=29556 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/ref/ReferenceQueue$Lock@0x2a96107b78[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:116)
 ^-- Lock released while waiting: java/lang/ref/ReferenceQueue$Lock@0x2a96107b78[fat lock]
 at java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:132)
 at sun/rmi/transport/ObjectTable$Reaper.run(ObjectTable.java:336)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"GC Daemon" id=23 idx=0x5c tid=29557 prio=2 alive, in native, waiting, daemon
 -- Waiting for notification on: sun/misc/GC$LatencyLock@0x2a9610a600[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at sun/misc/GC$Daemon.run(GC.java:100)
 ^-- Lock released while waiting: sun/misc/GC$LatencyLock@0x2a9610a600[fat lock]
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"JBoss System Threads(1)-1" id=24 idx=0x60 tid=29558 prio=5 alive, in native, daemon
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2ae5abc328[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at org/jnp/server/Main$AcceptHandler.run(Main.java:464)
 at org/jboss/util/threadpool/RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:743)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"JBoss System Threads(1)-2" id=25 idx=0x64 tid=29559 prio=5 alive, in native, daemon
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2ae5adecc0[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at org/jboss/web/WebServer.run(WebServer.java:324)
 at org/jboss/util/threadpool/RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:743)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"TimeoutFactory-0" id=26 idx=0x68 tid=29560 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5abcac8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/jboss/util/timeout/TimeoutPriorityQueueImpl.poll(TimeoutPriorityQueueImpl.java:271)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5abcac8[fat lock]
 at org/jboss/util/timeout/TimeoutPriorityQueueImpl.take(TimeoutPriorityQueueImpl.java:230)
 at org/jboss/util/timeout/TimeoutFactory.doWork(TimeoutFactory.java:215)
 at org/jboss/util/timeout/TimeoutFactory.access$000(TimeoutFactory.java:41)
 at org/jboss/util/timeout/TimeoutFactory$1.run(TimeoutFactory.java:136)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"JBossMQ Cache Reference Softner" id=27 idx=0x6c tid=29561 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/ref/ReferenceQueue$Lock@0x2ae5ae1838[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:116)
 ^-- Lock released while waiting: java/lang/ref/ReferenceQueue$Lock@0x2ae5ae1838[fat lock]
 at org/jboss/mq/server/MessageCache.run(MessageCache.java:245)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Timer-2" id=28 idx=0x70 tid=29562 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/TaskQueue@0x2ae5acad30[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/util/TimerThread.mainLoop(Timer.java:509)
 ^-- Lock released while waiting: java/util/TaskQueue@0x2ae5acad30[fat lock]
 at java/util/TimerThread.run(Timer.java:462)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"RMI TCP Accept-4444" id=29 idx=0x74 tid=29563 prio=5 alive, in native, daemon
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2ae5aefb80[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at sun/rmi/transport/tcp/TCPTransport.run(TCPTransport.java:340)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"PooledInvokerAcceptor#0-4445" id=30 idx=0x78 tid=29564 prio=5 alive, in native
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2ae5ad5a88[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at org/jboss/invocation/pooled/server/PooledInvoker.run(PooledInvoker.java:262)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"ContainerBackgroundProcessor[StandardEngine[jboss.web]]" id=31 idx=0x7c tid=29565 prio=5 alive, in native, sleeping, native_waiting, daemon
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/apache/catalina/core/ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1547)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"Thread-5" id=32 idx=0x80 tid=29566 prio=5 alive, in native, sleeping, native_waiting, daemon
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/console/plugins/AOPLister$1.run(AOPLister.java:882)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-0" id=33 idx=0x84 tid=29567 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-1" id=34 idx=0x88 tid=29568 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-2" id=35 idx=0x8c tid=29569 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-3" id=36 idx=0x90 tid=29570 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-4" id=37 idx=0x94 tid=29571 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-5" id=38 idx=0x98 tid=29572 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"DefaultQuartzScheduler_Worker-6" id=39 idx=0x9c tid=29573 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-7" id=40 idx=0xa0 tid=29574 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-8" id=41 idx=0xa4 tid=29575 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_Worker-9" id=42 idx=0xa8 tid=29576 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5af01d0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/quartz/simpl/SimpleThreadPool.getNextRunnable(SimpleThreadPool.java:428)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5af01d0[fat lock]
 at org/quartz/simpl/SimpleThreadPool.access$000(SimpleThreadPool.java:47)
 at org/quartz/simpl/SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:518)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"DefaultQuartzScheduler_QuartzSchedulerThread" id=43 idx=0xac tid=29577 prio=5 alive, in native, sleeping, native_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/quartz/core/QuartzSchedulerThread.run(QuartzSchedulerThread.java:394)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"HSQLDB Timer @53c548fe" id=44 idx=0xb0 tid=29578 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/hsqldb/lib/HsqlTimer@0x2ae5ad7110[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/hsqldb/lib/HsqlTimer.nextTask()Lorg/hsqldb/lib/HsqlTimer$Task;(Unknown Source)
 ^-- Lock released while waiting: org/hsqldb/lib/HsqlTimer@0x2ae5ad7110[fat lock]
 at org/hsqldb/lib/HsqlTimer$TaskRunner.run()V(Unknown Source)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"JCA PoolFiller" id=45 idx=0xb4 tid=29579 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/util/LinkedList@0x2ae5af9938[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/resource/connectionmanager/PoolFiller.run(PoolFiller.java:85)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae5af9938[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"TimeoutFactory-1" id=46 idx=0xb8 tid=29580 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5afb348[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/util/timeout/TimeoutPriorityQueueImpl.poll(TimeoutPriorityQueueImpl.java:253)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5afb348[fat lock]
 at org/jboss/util/timeout/TimeoutPriorityQueueImpl.take(TimeoutPriorityQueueImpl.java:230)
 at org/jboss/util/timeout/TimeoutFactory.doWork(TimeoutFactory.java:215)
 at org/jboss/util/timeout/TimeoutFactory.access$000(TimeoutFactory.java:41)
 at org/jboss/util/timeout/TimeoutFactory$1.run(TimeoutFactory.java:136)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"UILServerILService Accept Thread" id=47 idx=0xbc tid=29581 prio=5 alive, in native
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2ae5a2df78[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at org/jboss/mq/il/uil2/UILServerILService.run(UILServerILService.java:151)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"IdleRemover" id=48 idx=0xc0 tid=29582 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/ArrayList@0x2ae5a50950[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/jboss/resource/connectionmanager/IdleRemover$IdleRemoverRunnable.run(IdleRemover.java:161)
 ^-- Lock released while waiting: java/util/ArrayList@0x2ae5a50950[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=RicercaAnagrafeClienteBean,service=EJB3" id=49 idx=0xc4 tid=29588 prio=5 alive, in native, sleeping, native
_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=RicercaOrdiniClienteBean,service=EJB3" id=50 idx=0xc8 tid=29589 prio=5 alive, in native, sleeping, native_w
aiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=FileReaderBean,service=EJB3" id=51 idx=0xcc tid=29590 prio=5 alive, in native, sleeping, native_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=FileWriterBean,service=EJB3" id=52 idx=0xd0 tid=29591 prio=5 alive, in native, sleeping, native_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=RicercaSpedizioniBean,service=EJB3" id=53 idx=0xd4 tid=29592 prio=5 alive, in native, sleeping, native_wait
ing
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=ManagerGenerazioneListiniBean,service=EJB3" id=54 idx=0xd8 tid=29593 prio=5 alive, in native, sleeping, nat
ive_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=RicercaFattureBean,service=EJB3" id=55 idx=0xdc tid=29594 prio=5 alive, in native, sleeping, native_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=RicercaPrezzoBean,service=EJB3" id=56 idx=0xe0 tid=29595 prio=5 alive, in native, sleeping, native_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SFSB Passivation Thread - jboss.j2ee:jar=01-ecf3.jar,name=RicercaVenditeBean,service=EJB3" id=57 idx=0xe4 tid=29596 prio=5 alive, in native, sleeping, native_waiting
 at java/lang/Thread.sleep(J)V(Native Method)
 at org/jboss/ejb3/cache/simple/SimpleStatefulCache$SessionTimeoutTask.run(SimpleStatefulCache.java:101)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"UIL2(SocketManager.MsgPool@53cdf038 client=10.1.2.8:40754)#4" id=792 idx=0xe8 tid=30767 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: EDU/oswego/cs/dl/util/concurrent/LinkedNode@0x2a98404eb8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/SynchronousChannel.poll(SynchronousChannel.java:353)
 ^-- Lock released while waiting: EDU/oswego/cs/dl/util/concurrent/LinkedNode@0x2a98404eb8[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor.getTask(PooledExecutor.java:723)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:747)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"UIL2(SocketManager.MsgPool@565cb6ee client=10.1.2.8:8093)#32" id=853 idx=0xec tid=30900 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: EDU/oswego/cs/dl/util/concurrent/LinkedNode@0x2a98918e78[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/SynchronousChannel.poll(SynchronousChannel.java:353)
 ^-- Lock released while waiting: EDU/oswego/cs/dl/util/concurrent/LinkedNode@0x2a98918e78[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor.getTask(PooledExecutor.java:723)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:747)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"JMSThread(4)-209" id=854 idx=0xf0 tid=30930 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5afb850[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5afb850[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor.getTask(PooledExecutor.java:723)
 at org/jboss/util/threadpool/MinPooledExecutor.getTask(MinPooledExecutor.java:106)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:747)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"WorkManager(3)-256" id=855 idx=0xf4 tid=30931 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5921340[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5921340[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor.getTask(PooledExecutor.java:723)
 at org/jboss/util/threadpool/MinPooledExecutor.getTask(MinPooledExecutor.java:106)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:747)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"JBoss System Threads(1)-19" id=865 idx=0xf8 tid=30990 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2a95c653e8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
 ^-- Lock released while waiting: java/lang/Object@0x2a95c653e8[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor.getTask(PooledExecutor.java:723)
 at org/jboss/util/threadpool/MinPooledExecutor.getTask(MinPooledExecutor.java:106)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:747)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-14" id=216 idx=0xfc tid=29831 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627ce70[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627ce70[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"JMSThread(4)-213" id=867 idx=0x100 tid=30996 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5afb850[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5afb850[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor.getTask(PooledExecutor.java:723)
 at org/jboss/util/threadpool/MinPooledExecutor.getTask(MinPooledExecutor.java:106)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:747)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"(Attach Listener)" id=869 idx=0x104 tid=31011 prio=5 alive, in native, daemon

"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ID:9/60d7390f4982202adab51e9e1e81cb3e destination=QUEUE.ETWebLogger messageSele
ctor=null Local Create] id=1" id=66 idx=0x108 tid=29605 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/LinkedList@0x2ae5884960[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/mq/SpyConnectionConsumer.run(SpyConnectionConsumer.java:284)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae5884960[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ModRegReinviaAttivazione/d0a8e00bf38a154300ebcf7b3ab24e48 destination=TOPIC.ETR
einviaAttivazione.DurableSubscription[clientId=ModRegReinviaAttivazione name=ETReinviaAttivazione selector=null] messageSelector=null Local Create] id=4" id=69 idx=0x
10c tid=29608 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/LinkedList@0x2ae59811c8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/mq/SpyConnectionConsumer.run(SpyConnectionConsumer.java:284)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae59811c8[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ModuloRegistrazioneAlert/e977f93f50929be30a724b7201b49ef6 destination=TOPIC.ETM
oduloRegistrazione.DurableSubscription[clientId=ModuloRegistrazioneAlert name=ETModuloRegistrazione selector=null] messageSelector=null Local Create] id=2" id=67 idx=
0x110 tid=29606 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/LinkedList@0x2ae59cf7d8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/mq/SpyConnectionConsumer.run(SpyConnectionConsumer.java:284)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae59cf7d8[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:Wishing/88f1518c67eba3098d65dcb3dc813e0f destination=TOPIC.ETWishing.DurableSub
scription[clientId=Wishing name=ETWishing selector=null] messageSelector=null Local Create] id=3" id=68 idx=0x114 tid=29607 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/LinkedList@0x2ae58285b8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/mq/SpyConnectionConsumer.run(SpyConnectionConsumer.java:284)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae58285b8[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:OrdineEffettuato/c7d4804a3b8a52b12c15454ef6bff43b destination=TOPIC.ETOrdini.Du
rableSubscription[clientId=OrdineEffettuato name=ETOrdini selector=null] messageSelector=null Local Create] id=6" id=70 idx=0x118 tid=29609 prio=5 alive, in native, w
aiting, daemon
 -- Waiting for notification on: java/util/LinkedList@0x2ae588b2c8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/mq/SpyConnectionConsumer.run(SpyConnectionConsumer.java:284)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae588b2c8[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:InviaAmico/f09d7899aff95ed499e11db040b9b54f destination=TOPIC.ETInviaAmico.Dura
bleSubscription[clientId=InviaAmico name=ETInviaAmico selector=null] messageSelector=null Local Create] id=7" id=72 idx=0x11c tid=29610 prio=5 alive, in native, waiti
ng, daemon
 -- Waiting for notification on: java/util/LinkedList@0x2ae592eea8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/mq/SpyConnectionConsumer.run(SpyConnectionConsumer.java:284)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae592eea8[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ModRegRichiestaConferma/2486f04f03ede32c7e71a402e54b483b destination=TOPIC.ETMo
duloRegistrazione.DurableSubscription[clientId=ModRegRichiestaConferma name=ETModuloRegistrazione selector=null] messageSelector=null Local Create] id=8" id=73 idx=0x
120 tid=29612 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/LinkedList@0x2ae58109e0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/mq/SpyConnectionConsumer.run(SpyConnectionConsumer.java:284)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae58109e0[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Connection Consumer for dest Subscription[subId=-2147483648connection=ConnectionToken:ModRegInviaPwdPersa/472536b0735ddcc305f5074b2ff7bedf destination=TOPIC.ETInviaP
wdPersa.DurableSubscription[clientId=ModRegInviaPwdPersa name=ETInviaPwdPersa selector=null] messageSelector=null Local Create] id=5" id=71 idx=0x124 tid=29611 prio=5
 alive, in native, waiting, daemon
 -- Waiting for notification on: java/util/LinkedList@0x2ae58cd9f8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/mq/SpyConnectionConsumer.run(SpyConnectionConsumer.java:284)
 ^-- Lock released while waiting: java/util/LinkedList@0x2ae58cd9f8[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-6" id=132 idx=0x128 tid=29720 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cc30[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cc30[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"RMI RenewClean-[10.1.2.8:1098]" id=75 idx=0x12c tid=29614 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/ref/ReferenceQueue$Lock@0x2ae5501a70[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/ref/ReferenceQueue.remove(ReferenceQueue.java:116)
 ^-- Lock released while waiting: java/lang/ref/ReferenceQueue$Lock@0x2ae5501a70[fat lock]
 at sun/rmi/transport/DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"RMI LeaseChecker" id=77 idx=0x134 tid=29616 prio=5 alive, in native, sleeping, native_waiting, daemon
 at java/lang/Thread.sleep(J)V(Native Method)
 at sun/rmi/transport/DGCImpl$LeaseChecker.run(DGCImpl.java:310)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"JMSThread(4)-210" id=856 idx=0x138 tid=30935 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5afb850[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5afb850[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor.getTask(PooledExecutor.java:723)
 at org/jboss/util/threadpool/MinPooledExecutor.getTask(MinPooledExecutor.java:106)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:747)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SocketServerInvokerThread-10.1.2.8-27" id=79 idx=0x13c tid=29618 prio=5 alive, in native
 at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
 at jrockit/net/SocketNativeIO.socketRead(Ljava/io/FileDescriptor;[BIII)I(Unknown Source)
 at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
 at java/net/SocketInputStream.read(SocketInputStream.java:129)
 at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)[inlined]
 at java/io/BufferedInputStream.read(BufferedInputStream.java:235)[optimized]
 ^-- Holding lock: java/io/BufferedInputStream@0x2af1efc118[thin lock]
 at java/io/ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
 at java/io/ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2376)
 at java/io/ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2443)
 at java/io/ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2515)
 at java/io/ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2664)
 at java/io/ObjectInputStream.readByte(ObjectInputStream.java:875)
 at org/jboss/remoting/transport/socket/ServerSocketWrapper.checkConnection(ServerSocketWrapper.java:77)
 at org/jboss/remoting/transport/socket/ServerThread.acknowledge(ServerThread.java:317)
 at org/jboss/remoting/transport/socket/ServerThread.dorun(ServerThread.java:411)
 at org/jboss/remoting/transport/socket/ServerThread.run(ServerThread.java:239)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"JBossLifeThread" id=80 idx=0x140 tid=29619 prio=5 alive, in native, waiting
 -- Waiting for notification on: java/lang/Object@0x2ae5532ab8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/system/server/ServerImpl$LifeThread.run(ServerImpl.java:932)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5532ab8[fat lock]
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080" id=81 idx=0x144 tid=29620 prio=5 alive, in native, daemon
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2ae5f7d820[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at org/apache/tomcat/util/net/DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
 at org/apache/tomcat/util/net/PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
 at org/apache/tomcat/util/net/PoolTcpEndpoint.run(PoolTcpEndpoint.java:647)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-1" id=82 idx=0x148 tid=29621 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cb10[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cb10[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-2" id=83 idx=0x14c tid=29622 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cb58[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cb58[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"TP-Processor1" id=84 idx=0x150 tid=29623 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0x2ae5cf2810[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:656)
 ^-- Lock released while waiting: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0x2ae5cf2810[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"TP-Processor2" id=85 idx=0x154 tid=29624 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0x2ae5cf2840[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:656)
 ^-- Lock released while waiting: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0x2ae5cf2840[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"TP-Processor3" id=86 idx=0x158 tid=29625 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0x2ae5cf2870[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:656)
 ^-- Lock released while waiting: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable@0x2ae5cf2870[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"TP-Processor4" id=87 idx=0x15c tid=29626 prio=5 alive, in native, daemon
 at java/net/PlainSocketImpl.socketAccept(Ljava/net/SocketImpl;)V(Native Method)
 at java/net/PlainSocketImpl.accept(PlainSocketImpl.java:384)
 ^-- Holding lock: java/net/SocksSocketImpl@0x2ae5cf1830[thin lock]
 at java/net/ServerSocket.implAccept(ServerSocket.java:450)
 at java/net/ServerSocket.accept(ServerSocket.java:421)
 at org/apache/jk/common/ChannelSocket.accept(ChannelSocket.java:306)
 at org/apache/jk/common/ChannelSocket.acceptConnections(ChannelSocket.java:660)
 at org/apache/jk/common/ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:870)
 at org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run(ThreadPool.java:684)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"TP-Monitor" id=88 idx=0x160 tid=29627 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/threads/ThreadPool$MonitorRunnable@0x2ae5cf1788[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at org/apache/tomcat/util/threads/ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
 ^-- Lock released while waiting: org/apache/tomcat/util/threads/ThreadPool$MonitorRunnable@0x2ae5cf1788[fat lock]
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"UIL2.SocketManager.ReadTask#1 client=10.1.2.8:40754" id=90 idx=0x164 tid=29629 prio=5 alive, in native, daemon
 at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
 at jrockit/net/SocketNativeIO.socketRead(Ljava/io/FileDescriptor;[BIII)I(Unknown Source)
 at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
 at java/net/SocketInputStream.read(SocketInputStream.java:129)
 at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)[inlined]
 at java/io/BufferedInputStream.read(BufferedInputStream.java:235)[optimized]
 ^-- Holding lock: org/jboss/util/stream/NotifyingBufferedInputStream@0x2ae5a2f5f0[thin lock]
 at org/jboss/util/stream/NotifyingBufferedInputStream.read(NotifyingBufferedInputStream.java:79)
 at java/io/ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
 at java/io/ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2376)
 at java/io/ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2443)
 at java/io/ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2515)
 at java/io/ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2664)
 at java/io/ObjectInputStream.readByte(ObjectInputStream.java:875)
 at org/jboss/mq/il/uil2/SocketManager$ReadTask.run(SocketManager.java:316)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"UIL2.SocketManager.WriteTask#2 client=10.1.2.8:40754" id=91 idx=0x168 tid=29630 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5a2eed0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at EDU/oswego/cs/dl/util/concurrent/LinkedQueue.take(LinkedQueue.java:122)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5a2eed0[fat lock]
 at org/jboss/mq/il/uil2/SocketManager$WriteTask.run(SocketManager.java:518)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"UIL2.SocketManager.ReadTask#1 client=10.1.2.8:8093" id=92 idx=0x16c tid=29631 prio=5 alive, in native, daemon
 at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
 at jrockit/net/SocketNativeIO.socketRead(Ljava/io/FileDescriptor;[BIII)I(Unknown Source)
 at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)
 at java/net/SocketInputStream.read(SocketInputStream.java:129)
 at java/io/BufferedInputStream.fill(BufferedInputStream.java:218)[inlined]
 at java/io/BufferedInputStream.read(BufferedInputStream.java:235)[optimized]
 ^-- Holding lock: org/jboss/util/stream/NotifyingBufferedInputStream@0x2ae5c4e850[thin lock]
 at org/jboss/util/stream/NotifyingBufferedInputStream.read(NotifyingBufferedInputStream.java:79)
 at java/io/ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2196)
 at java/io/ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2376)
 at java/io/ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2443)
 at java/io/ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2515)
 at java/io/ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2664)
 at java/io/ObjectInputStream.readByte(ObjectInputStream.java:875)
 at org/jboss/mq/il/uil2/SocketManager$ReadTask.run(SocketManager.java:316)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"UIL2.SocketManager.WriteTask#2 client=10.1.2.8:8093" id=93 idx=0x170 tid=29632 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5c4f2f8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at EDU/oswego/cs/dl/util/concurrent/LinkedQueue.take(LinkedQueue.java:122)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5c4f2f8[fat lock]
 at org/jboss/mq/il/uil2/SocketManager$WriteTask.run(SocketManager.java:518)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace


"WorkManager(3)-257" id=857 idx=0x174 tid=30936 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: java/lang/Object@0x2ae5921340[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/BoundedLinkedQueue.poll(BoundedLinkedQueue.java:253)
 ^-- Lock released while waiting: java/lang/Object@0x2ae5921340[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor.getTask(PooledExecutor.java:723)
 at org/jboss/util/threadpool/MinPooledExecutor.getTask(MinPooledExecutor.java:106)
 at EDU/oswego/cs/dl/util/concurrent/PooledExecutor$Worker.run(PooledExecutor.java:747)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"Connection Monitor Thread" id=95 idx=0x178 tid=29634 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: EDU/oswego/cs/dl/util/concurrent/ClockDaemon@0x2ae5c521d8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at EDU/oswego/cs/dl/util/concurrent/ClockDaemon.nextTask(ClockDaemon.java:328)
 ^-- Lock released while waiting: EDU/oswego/cs/dl/util/concurrent/ClockDaemon@0x2ae5c521d8[fat lock]
 at EDU/oswego/cs/dl/util/concurrent/ClockDaemon$RunLoop.run(ClockDaemon.java:362)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-8" id=208 idx=0x17c tid=29823 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627ccc0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627ccc0[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-9" id=209 idx=0x180 tid=29824 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cd08[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cd08[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-10" id=210 idx=0x184 tid=29825 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cd50[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cd50[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-5" id=130 idx=0x188 tid=29717 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cbe8[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cbe8[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-4" id=100 idx=0x18c tid=29644 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cba0[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cba0[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"SocketServerInvokerThread-10.1.2.8-31" id=131 idx=0x190 tid=29718 prio=5 alive, in native, waiting
 -- Waiting for notification on: org/jboss/remoting/transport/socket/ServerThread@0x2ae556ff98[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/jboss/remoting/transport/socket/ServerThread.run(ServerThread.java:280)
 ^-- Lock released while waiting: org/jboss/remoting/transport/socket/ServerThread@0x2ae556ff98[fat lock]
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-11" id=211 idx=0x194 tid=29826 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cd98[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cd98[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-7" id=135 idx=0x198 tid=29723 prio=5 alive, in native, waiting, daemon
 -- Waiting for notification on: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cc78[fat lock]
 at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
 at java/lang/Object.wait(J)V(Native Method)
 at java/lang/Object.wait(Object.java:474)
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
 ^-- Lock released while waiting: org/apache/tomcat/util/net/MasterSlaveWorkerThread@0x2ae627cc78[fat lock]
 at org/apache/tomcat/util/net/MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
 at java/lang/Thread.run(Thread.java:595)
 at jrockit/vm/RNI.c2java(JJJJ)V(Native Method)
 -- end of trace

"http-0.0.0.0-8080-12" id=212 idx=0x19c tid=29827 prio=5 alive, in native, daemon
 at jrockit/net/SocketNativeIO.writeBytesPinned(Ljava/io/FileDescriptor;[BII)V(Native Method)
 at jrockit/net/SocketNativeIO.socketWrite(Ljava/io/FileDescriptor;[BII)V(Unknown Source)
 at java/net/SocketOutputStream.socketWrite0(Ljava/io/FileDescriptor;[BII)V(SocketOutputStream.java)
 at java/net/SocketOutputStream.socketWrite(SocketOutputStream.java:92)
 at java/net/SocketOutputStream.write(SocketOutputStream.java:136)
 at org/apache/coyote/http11/InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:746)
 at org/apache/tomcat/util/buf/ByteChunk.flushBuffer(ByteChunk.java:433)
 at org/apache/coyote/http11/InternalOutputBuffer.flush(InternalOutputBuffer.java:304)
 at org/apache/coyote/http11/Http11Processor.action(Http11Processor.java:991)
 at org/apache/coyote/Response.action(Response.java:182)
 at org/apache/catalina/connector/OutputBuffer.doFlush(OutputBuffer.java:322)
 at org/apache/catalina/connector/OutputBuffer.flush(OutputBuffer.java:293)
 at org/apache/catalina/connector/CoyoteWriter.flush(CoyoteWriter.java:94)
 at org/apache/jasper/runtime/JspWriterImpl.flush(JspWriterImpl.java:174)
 at freemarker/core/Environment.process(Environment.java:178)
 at freemarker/template/Template.process(Template.java:232)
 at org/apache/struts2/components/template/FreemarkerTemplateEngine.renderTemplate(FreemarkerTemplateEngine.java:148)
 at org/apache/struts2/components/UIBean.mergeTemplate(UIBean.java:530)[inlined]
 at org/apache/struts2/components/UIBean.end(UIBean.java:484)[optimized]
 at org/apache/struts2/views/jsp/ComponentTagSupport.doEndTag(ComponentTagSupport.java:43)[optimized]
 at org/apache/jsp/listaArticoli_jsp._jspx_meth_s_select_0(listaArticoli_jsp.java:1379)
 at org/apache/jsp/listaArticoli_jsp._jspx_meth_s_form_1(listaArticoli_jsp.java:1340)
 at org/apache/jsp/listaArticoli_jsp._jspService(listaArticoli_jsp.java:260)
 at org/apache/jasper/runtime/HttpJspBase.service(HttpJspBase.java:97)
 at javax/servlet/http/HttpServlet.service(HttpServlet.java:810)
 at org/apache/jasper/servlet/JspServletWrapper.service(JspServletWrapper.java:334)
 at org/apache/jasper/servlet/JspServlet.serviceJspFile(JspServlet.java:314)
 at org/apache/jasper/servlet/JspServlet.service(JspS

[tobias]

Is that a thread dump when it is "quite" on the server? That would be interesting, cut the LAN connection, wait a minute and then do the thread dump.

The SLSB passivation might be something interesting to look into. JBossCache might be involved (cluster?).

Also there is a thread cpu utilization tool in the JDK. Get the server in a quite state check the values, wait for 5 mins and check the values again to see which threads cause the CPU utilization.

In the jmx-console you'll find the ServerInfo-MBean. Invoke the listThreadCpuUtilization() operation. HTH.

[vettori]

Hi,

the dump is when the jvm is continually running the gc.
Yesterday I activated a log on the web connections. The only port reachable from the internet is the http port. In the night the server received less than 20 request.
But the jvm run the gc every three seconds all night long. So it can't be something that's generated by requests.

There is no JBossCache and I disabled the hibernate second level cache.
The passivation occours on some stateful session beans.

I'm really lost. It seems something internal to JBoss but can't understand what. Yesterday I also removed all the unneeded services but that don't helped.

Any other hint ?

Thank you

[vettori]

After experimenting I found that the problem can be an Hibernate problem.

Here is the post I've done

http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4059217#4059217

[vettori]

Bad news...

Even with the upgrade it only takes longer but it happens...

[INFO ][memory ] 41369.000-41369.141: GC 973949K->153965K (5242880K), 140.800 ms
[INFO ][memory ] 43678.758-43678.893: GC 876646K->145950K (5242880K), 135.191 ms
[INFO ][memory ] 46058.278-46058.425: GC 847984K->140040K (5242880K), 147.288 ms
[INFO ][memory ] 48400.393-48400.529: GC 973458K->152902K (5242880K), 136.187 ms
[INFO ][memory ] 50376.968-50377.117: GC 1008382K->171014K (5242880K), 148.453 ms
[INFO ][memory ] 52293.828-52293.963: GC 911852K->149894K (5242880K), 135.565 ms
[INFO ][memory ] 52766.219: parallel nursery GC 1597712K->819792K (5242880K), 1066.246 ms
[INFO ][memory ] 52785.264: parallel nursery GC 1963059K->1184403K (5242880K), 825.711 ms
[INFO ][memory ] 52805.516: parallel nursery GC 1962803K->1184947K (5242880K), 822.961 ms
[INFO ][memory ] 52828.219: parallel nursery GC 2290690K->1512290K (5242880K), 833.181 ms
[INFO ][memory ] 55071.548-55073.111: GC 1862290K->206785K (5242880K), 1563.780 ms
[INFO ][memory ] 57584.100-57584.879: GC 888041K->93430K (5242880K), 779.164 ms
[INFO ][memory ] 59435.050: parallel nursery GC 1552296K->766040K (5242880K), 219.950 ms
[INFO ][memory ] 59438.151: parallel nursery GC 1552040K->765260K (5242880K), 36.980 ms
[INFO ][memory ] 59441.070: parallel nursery GC 1551660K->765404K (5242880K), 39.158 ms
[INFO ][memory ] 59443.997: parallel nursery GC 1551404K->764556K (5242880K), 34.195 ms
[INFO ][memory ] 59446.923: parallel nursery GC 1550956K->764637K (5242880K), 32.322 ms
[INFO ][memory ] 59449.843: parallel nursery GC 1550637K->764237K (5242880K), 34.257 ms
[INFO ][memory ] 59452.738: parallel nursery GC 1550637K->764533K (5242880K), 36.508 ms
[INFO ][memory ] 59455.669: parallel nursery GC 1550533K->764133K (5242880K), 35.539 ms
[INFO ][memory ] 59458.597: parallel nursery GC 1550533K->764533K (5242880K), 33.265 ms
[INFO ][memory ] 59461.509: parallel nursery GC 1550533K->764133K (5242880K), 34.502 ms
[INFO ][memory ] 59464.414: parallel nursery GC 1550533K->764533K (5242880K), 32.673 ms
[INFO ][memory ] 59467.293: parallel nursery GC 1550533K->764133K (5242880K), 38.018 ms
[INFO ][memory ] 59470.192: parallel nursery GC 1550533K->764277K (5242880K), 33.602 ms
[INFO ][memory ] 59473.077: parallel nursery GC 1550277K->763877K (5242880K), 34.551 ms
[INFO ][memory ] 59475.945: parallel nursery GC 1550277K->764149K (5242880K), 39.169 ms
[INFO ][memory ] 59478.842: parallel nursery GC 1550149K->763365K (5242880K), 34.903 ms
[INFO ][memory ] 59481.727: parallel nursery GC 1549765K->763477K (5242880K), 32.807 ms
[INFO ][memory ] 59484.612: parallel nursery GC 1549477K->762629K (5242880K), 34.387 ms
[INFO ][memory ] 59487.473: parallel nursery GC 1549029K->763029K (5242880K), 36.790 ms
[INFO ][memory ] 59490.376: parallel nursery GC 1549029K->762437K (5242880K), 39.065 ms
[INFO ][memory ] 59493.275: parallel nursery GC 1548837K->762453K (5242880K), 33.966 ms
[INFO ][memory ] 59496.155: parallel nursery GC 1548453K->762053K (5242880K), 35.931 ms
[INFO ][memory ] 59499.040: parallel nursery GC 1548453K->762350K (5242880K), 36.593 ms
[INFO ][memory ] 59501.933: parallel nursery GC 1548350K->761950K (5242880K), 33.835 ms



ANYONE CAN HELP ?

I read everything but can't find nothing explaining this...

[philjboss]

I have the same issue. Starts slowly and then over time the GC gets faster and faster and eventually brings down JBOSS. With this log:

345059.959: [GC 160936K->90857K(777856K), 0.0043550 secs]
345060.905: [GC 160937K->90857K(777856K), 0.0042820 secs]
345061.842: [GC 160937K->90858K(777856K), 0.0042990 secs]
345062.781: [GC 160938K->90857K(777856K), 0.0044080 secs]
345063.703: [GC 160937K->90858K(777856K), 0.0043070 secs]
345064.703: [GC 160938K->90851K(777856K), 0.0042030 secs]
345065.625: [GC 160931K->90862K(777856K), 0.0043180 secs]
345066.555: [GC 160942K->90868K(777856K), 0.0043630 secs]
345067.467: [GC 160948K->90860K(777856K), 0.0043340 secs]
345068.400: [GC 160940K->90861K(777856K), 0.0044370 secs]
345069.350: [GC 160941K->90875K(777856K), 0.0042140 secs]

[philjboss]

For information. I'm using Hibernate within the WAR files rather than as the jboss service. Apart from that we have nothing really out of the ordinary.