HornetQ 2.2.5.Final, closeSession sometimes takes forever!

[yannywang]

with HornetQ 2.2.5.Final, I try to closeSession which is created by clientSessionFactory.createSession(true,true,1). when exception happens, we try to close the session but sometime it takes forever.

Here is the stack trace:

"Thread-1 (group:HornetQ-client-global-threads-27692793)" daemon prio=6 tid=0x043c3400 nid=0x1cb4 waiting on condition [0x0501f000]

   java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

- parking to wait for  <0x23d53420> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2116)

at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:256)

- locked <0x23d53458> (a java.lang.Object)

at org.hornetq.core.client.impl.ClientSessionImpl.close(ClientSessionImpl.java:857)

at org.hornetq.core.client.impl.ClientSessionFactoryImpl.close(ClientSessionFactoryImpl.java:438)

- locked <0x29287678> (a java.lang.Object)

- locked <0x29287670> (a java.lang.Object)

at com.alibaba.napoli.mqImpl.hornetQ.HQConnection.close(HQConnection.java:66)

- locked <0x29269630> (a java.lang.Object)

at com.alibaba.napoli.mqImpl.hornetQ.HornetQTransportConsumer$JMSFailureListener.connectionFailed(HornetQTransportConsumer.java:390)

- locked <0x29269630> (a java.lang.Object)

- locked <0x23d4b028> (a com.alibaba.napoli.mqImpl.hornetQ.HornetQTransportConsumer$JMSFailureListener)

at org.hornetq.core.client.impl.ClientSessionFactoryImpl.callFailureListeners(ClientSessionFactoryImpl.java:818)

at org.hornetq.core.client.impl.ClientSessionFactoryImpl.failoverOrReconnect(ClientSessionFactoryImpl.java:605)

- locked <0x29287678> (a java.lang.Object)

at org.hornetq.core.client.impl.ClientSessionFactoryImpl.handleConnectionFailure(ClientSessionFactoryImpl.java:482)

at org.hornetq.core.client.impl.ClientSessionFactoryImpl.access$800(ClientSessionFactoryImpl.java:78)

at org.hornetq.core.client.impl.ClientSessionFactoryImpl$DelegatingFailureListener.connectionFailed(ClientSessionFactoryImpl.java:1318)

at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.callFailureListeners(RemotingConnectionImpl.java:528)

at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.fail(RemotingConnectionImpl.java:298)

at org.hornetq.core.client.impl.ClientSessionFactoryImpl$Channel0Handler$1.run(ClientSessionFactoryImpl.java:1262)

at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

at java.lang.Thread.run(Thread.java:662)

ChannelImpl.java:256 points to:

sendCondition.await(toWait, TimeUnit.MILLISECONDS);

which I've checked the toWait (come from serverLocator's call timeout by default is 30000(30 seconds)). and I've checked by config and make sure  didn't changed it.

My understanding is the await will timeout in 30 seconds which should not hang for ever.

any idea?

[gaohoward]

Do you have some client code showing how you catch the exception and close the session?

[yannywang]

Hi Yong Hao.

after create session, I do:

session.addFailureListener((SessionFailureListener) (new JMSFailureListener(handler))

which handler.getSession() will return session.

the JMSFailureListener is an inner class which have clientSessionFactory, I call clientSessionFactory.close() in the connectionFailed method.

During my test, I run HornetQ's stop.sh to stop the serivce which will then trigger the callFailureListener.

my clientSessionFactory may create mutliple session, when connectionFailed is called, I do sychronize close clientSessionFactory when it's not closed.

any suggestion?

[gaohoward]

The stack shows the client is waiting for response from the server and the response never come back. Did you try to test it using 'kill -9' to kill the server and see if it happens too?

Howard

[yannywang]

This doesn't always happen when I test with stop.sh.

I didn't try kill -9since I'm afraid it may mess up the pag files.

[gaohoward]

It shouldn't mess up the page files.

[yannywang]

my Question to "The stack shows the client is waiting for response from the server and the response never come back.". is I suppose the secondCondition.wait(long time, Unit) means only wait for given time. it should not wait forever. am I right? This is what confuses me.

[gaohoward]

It shouldn't. I don't know why the code gets stuck here. Is your test reproducible? I'd like to have a try myself.

Howard

[yannywang]

H Howard,

unfortunately i's not a stabl repo. I did see it several time thogh.

It may be related to I created the queue with JMS interface and then operate use Core API.

The "pagin file mess" is also due to this. Now change back to create queue also in CoreAPI. the page file mess went away.

I will let you know more information if I figure out wit the cloeSessin hang

Thanks for looking into this.