connection close problem.

[sentcs]

Hi,

We have deployed jboss-4.2.3.GA with jboss-messaging-1.4.0.SP3 in our production environment.

In our application,we have developed with recivers are restarting automatically for every 15 minutes(configurable). In this case, when restarting, we are closing all the objects are session, subscriber, connection.

Problem as when we close the connection obejct, it takes around 1 hour at some time(not all the time). We don't know the reason why it takes that much of time.

Please some help me.

Thanks in advance.

[gaohoward]

Hi, can you provide the thread dump when that happens, it will give us where the process got stuck. Thanks.

[sentcs]

Thanks for your response.

We have analyzed and find out that, we are getting the same problem for below scenarios,

1. To get the connection from JBoss.
2. To register the subscribers after the getting the connection
3. Close the connection.

Here i pasted the Thread dump when get the connection(receiver gets hang).

Full thread dump Java HotSpot(TM) Server VM (10.0-b23 mixed mode):

"Timer-2" daemon prio=10 tid=0x4c6c7800 nid=0x5751 in Object.wait() [0x4d8f1000..0x4d8f20a0]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x4745abc8> (a java.util.TaskQueue)
 at java.lang.Object.wait(Object.java:485)
 at java.util.TimerThread.mainLoop(Timer.java:483)
 - locked <0x4745abc8> (a java.util.TaskQueue)
 at java.util.TimerThread.run(Timer.java:462)

"Timer-1" daemon prio=10 tid=0x4d145c00 nid=0x574f in Object.wait() [0x4dbf9000..0x4dbf91a0]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x47448df8> (a java.util.TaskQueue)
 at java.util.TimerThread.mainLoop(Timer.java:509)
 - locked <0x47448df8> (a java.util.TaskQueue)
 at java.util.TimerThread.run(Timer.java:462)

"Timer-0" daemon prio=10 tid=0x082a8800 nid=0x574e in Object.wait() [0x4daab000..0x4daac120]

java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x4744a118> (a java.util.TaskQueue)
 at java.lang.Object.wait(Object.java:485)
 at java.util.TimerThread.mainLoop(Timer.java:483)
 - locked <0x4744a118> (a java.util.TaskQueue)
 at java.util.TimerThread.run(Timer.java:462)

"RMI Scheduler(0)" daemon prio=10 tid=0x081f8400 nid=0x574d waiting on condition [0x4d8a0000..0x4d8a0ea0]
 java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for <0x4740d308> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
 at java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:582)
 at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:575)
 at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
 at java.lang.Thread.run(Thread.java:619)

"GC Daemon" daemon prio=10 tid=0x082dc400 nid=0x574b in Object.wait() [0x4d84f000..0x4d84fe20]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x47375058> (a sun.misc.GC$LatencyLock)
 at sun.misc.GC$Daemon.run(GC.java:100)
 - locked <0x47375058> (a sun.misc.GC$LatencyLock)

"RMI RenewClean-[172.19.6.5:1098]" daemon prio=10 tid=0x08431000 nid=0x574a in Object.wait() [0x4d0bb000..0x4d0bbfa0]
 java.lang.Thread.State: TIMED_WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x473750c8> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 - locked <0x473750c8> (a java.lang.ref.ReferenceQueue$Lock)
 at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:516)
 at java.lang.Thread.run(Thread.java:619)

"DestroyJavaVM" prio=10 tid=0x4d74d400 nid=0x573b waiting on condition [0x00000000..0x401cf0f0]
 java.lang.Thread.State: RUNNABLE

"Thread-0" prio=10 tid=0x4d507c00 nid=0x5748 runnable [0x4d06a000..0x4d06af20]

java.lang.Thread.State: RUNNABLE
 at java.net.SocketInputStream.socketRead0(Native Method)
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
 - locked <0x48f06e60> (a java.io.BufferedInputStream)
 at java.io.FilterInputStream.read(FilterInputStream.java:66)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.readVersion(MicroSocketClientInvoker.java:962)
 at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:601)
 at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:418)
 at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:122)
 at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:373)
 - locked <0x48f06fa8> (a java.lang.Object)
 at org.jboss.remoting.Client.setupClientLease(Client.java:1625)
 at org.jboss.remoting.Client.connect(Client.java:1525)
 at org.jboss.remoting.Client.connect(Client.java:462)
 at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:319)
 at java.security.AccessController.doPrivileged(Native Method)
 at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:315)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactory
Delegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:154)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeN
ext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
 at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:83)
 at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect0.invoke(StateCreationAspect0.java)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeN
ext(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
 at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDeleg
ate.java)
 at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
 at org.jboss.jms.client.JBossConnectionFactory.createConnection(JBossConnectionFactory.java:87)
 at com.tcs.retail.framework.messaging.lookup.ConnectionForReceiver.getConnection(ConnectionForReceiver.java:51)
 at com.tcs.retail.framework.messaging.lookup.ConnectionForReceiver.getConnectionForReceiver(ConnectionForReceiver.java
:65)
 at com.tcs.retail.framework.messaging.service.MsgReceiverService.checkConnection(MsgReceiverService.java:183)
 at com.tcs.retail.framework.messaging.service.MsgReceiverService.startListener(MsgReceiverService.java:49)
 at com.tcs.retail.framework.messaging.service.MsgReceiverService.run(MsgReceiverService.java:227)
 at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=10 tid=0x4c6b7c00 nid=0x5746 runnable [0x00000000..0x00000000]
 java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x4c6b6000 nid=0x5745 waiting on condition [0x00000000..0x4cb5d668]
 java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x4c6b4c00 nid=0x5744 waiting on condition [0x00000000..0x4cadc5e8]
 java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x4c6b3800 nid=0x5743 waiting on condition [0x00000000..0x00000000]
 java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x4c69b800 nid=0x5742 in Object.wait() [0x4c9de000..0x4c9dee20]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x47373ca8> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
 - locked <0x47373ca8> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
 at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x4c69a800 nid=0x5741 in Object.wait() [0x4c98d000..0x4c98dfa0]
 java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x47375048> (a java.lang.ref.Reference$Lock)
 at java.lang.Object.wait(Object.java:485)
 at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
 - locked <0x47375048> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x4c697800 nid=0x5740 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x08060c00 nid=0x573c runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x08061c00 nid=0x573d runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x08062c00 nid=0x573e runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x08063c00 nid=0x573f runnable

"VM Periodic Task Thread" prio=10 tid=0x4c6c1400 nid=0x5747 waiting on condition

JNI global references: 636

Heap

PSYoungGen total 4032K, used 93K [0x4ba50000, 0x4bf50000, 0x4c330000)
 eden space 3136K, 2% used [0x4ba50000,0x4ba67668,0x4bd60000)
 from space 896K, 0% used [0x4bd60000,0x4bd60000,0x4be40000)
 to space 960K, 0% used [0x4be60000,0x4be60000,0x4bf50000)
 PSOldGen total 43328K, used 28923K [0x47330000, 0x49d80000, 0x4ba50000)
 object space 43328K, 66% used [0x47330000,0x48f6ee70,0x49d80000)
 PSPermGen total 34560K, used 26615K [0x43330000, 0x454f0000, 0x47330000)
 object space 34560K, 77% used [0x43330000,0x44d2dc60,0x454f0000)

Please let us know if any correction.

Thanks in advance.

[bodrin]

Hi,

Sometimes my consumer also gets disconnected for some reason and is unable to reconnect. I'm using the ExceptionListener. When I detect this I try to connect another consumer and it gets stuck on the same line :

...
MicroSocketClientInvoker.java:962
...

[gaohoward]

Hi,

Do you have a working test with it? It would be much easier to find the problem if there is some test.

I suggest you also post it to remoting forum as I think it's like a remoting issue.

[bodrin]

Hi,

Unfortunatelly it happens randomly and when I detect it I just do restart the JBM server.

[psvn]

Hi there!

Sentcs, did you solve your problem?

I'm using jboss-4.2.3.GA with jboss-messaging-1.4.4.GA and randomly
i have the same problem "java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)" and messages got stucked with state "+".

This just happen when i use Transactional connections to the queues.

Anyone can help?

Thnks in advance.