JMS hangs after IOException in the server

[richieb]

We're running Jboss 3.2.3 on Linux (JDK 1.4.1). Our application uses publishes messages on several topics (non-persistent messages). The load is pretty heavy - we have about 40 clients publishing and receiving messages every 10 seconds.

Few times the whole JMS just hanged, after an exception in server. The IO exception was from "internalSend" in org.jboss.mq.il.uil2.SocketManager and it said:

"Client not connected"

After the initial exception eveything just stops. Except the same exception keeps being thrown in the server log. The clients don't seem to get any exceptions at all.

Restarting the clients has no effect and only restarting JBoss fixes the problem.

The JMS is configured to use file based persistance manager, as we normally do not expect message to hang around.

I'm unable to post extract from the log at this point, because these are production machines and getting logs copied over gets difficult.

Does this problem sound familiar to anyone? Any suggestions where to look will be greatly appreciated.

What other information would help in finding what the problem is?

TIA

...richie

[priyapravas]

It is probably easier and better to look for a suitable database file in the doc directory of the jboss install.

[adrian.brock]

The server should disconnect the client when it cannot send a message
to the client. I would need to see the logs to find out what is going on.

Regards,
Adrian

[richieb]

Here is a small extract from the server log:

2004-02-20 18:26:10,946 ERROR [org.jboss.mq.server.JMSDestinationManager] The connection to client ID
:7 failed.
2004-02-20 18:26:10,948 ERROR [org.jboss.mq.il.uil2.SocketManager] Failed to handle: org.jboss.mq.il.
uil2.msgs.AcknowledgementRequestMsg1649195744[msgType: m_acknowledge, msgID: 171372, error: null]
java.io.IOException: Client is not connected
at org.jboss.mq.il.uil2.SocketManager.internalSendMessage(SocketManager.java:225)
at org.jboss.mq.il.uil2.SocketManager.sendReply(SocketManager.java:208)
at org.jboss.mq.il.uil2.ServerSocketManagerHandler.handleMsg(ServerSocketManagerHandler.java:
82)
at org.jboss.mq.il.uil2.SocketManager$ReadTask.handleMsg(SocketManager.java:355)
at org.jboss.mq.il.uil2.msgs.BaseMsg.run(BaseMsg.java:377)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:732)
at java.lang.Thread.run(Thread.java:568)


We see no exceptions on the client side.

The client seems to be able to publish without errors, but does not see any new messages.

I haven't been able to reproduce this problem with a small test case.

...richie

[adrian.brock]

The first message says the server is going to close the connection.

The second message says the client was still trying to acknowledge
receipt of a message.

Have you set an ExceptionListener on the connection so you can monitor for
errors in the transport?

Regards,
Adrian

[richieb]

Yes we have. Here are snippets of our code:


tcf_ = (TopicConnectionFactory) context_.lookup(connectionFactory_);
conn_ = tcf_.createTopicConnection ();
conn_.setExceptionListener(new ExceptionListener() {
public void onException(JMSException e)
{
if (!restarting_)
JMSInterface.shutdownAndRestart(e);
}
});


In "shutdownAndRestart" we attempt to disconnect and reconnect to JMS. We log several messages - which I do not see in this case. Here are more snipets of the code:


private static void shutdownAndRestart(Throwable t) {

TSLogger.getMainLogger().debug("JMS Service Failure", t);
restarted_ = false;
synchronized (JMSInterface.class) {
if (restarted_)
return;
restarting_ = true;
for (;;) {
TSLogger.getMainLogger().warn("JMS service not reachable ... will try again in a few seconds");
try {
Thread.sleep (reconnectTimeout_);
}
catch (InterruptedException ie) {}

... etc


We don not see the warning message. In a more typical operation, when we shut down JBoss we see the warnings logged by the clients. When JBoss is restarted the clients reconnect and continue.

...richie

[adrian.brock]

Since you didn't get a disconnect message on the client it sounds
like the server failed to close the connection.

Unfortunatly, the logging of an error from this operation has been disabled
// Close the clientIL
try
{
if (dc.clientIL != null)
dc.clientIL.close();
}
catch (Exception ex)
{
// We skipp warning, to often the client will allways
// have gone when we get here
//log.warn("Could not close clientIL: " +ex,ex);
}

Another option is that it got stuck in a deadlock before hitting this code.
This would be consistent with you saying "JMS hangs".
I don't suppose you took a thread dump while it hung?

Regards,
Adrian

[richieb]

I'm afraid I don't have a thread dump. This is a production system, so I only find out after a JBoss restart.


...richie