Failed to get a server session

[imaeses]

Hello all,

We are using JBoss Messaging 1.0.1.SP4 with JBoss Remoting 2.2.0 in JBoss 4.0.5.GA. We have a series of MDB's which answer to queues. The queues are deployed on a remote machine from the JBoss instance hosting the MDB's.

We have already noticed that sometimes when shutting down JBoss, the server hangs on undeployment of MDB's.

Today we were investigating the fact that certain queues were populated with messages that were not being delivered to their target MDB's. Using the JMX console, we noticed that the number of messages in MessageCount was not the same as that from querying the database directly for these queues. Sometimes it was more; sometimes it was less. When we tried to undeploy one of the MDB's affected, we saw the following stracktrace:

2007-03-26 13:48:00,227 WARN [org.jboss.tm.TransactionImpl] Transaction TransactionImpl:XidImpl[FormatId=257, GlobalId=Franco/39484616, BranchQual=, localId=39484616] timed out. status=STATUS_COMMITING
2007-03-26 13:48:11,364 WARN [org.jboss.jms.client.JBossConnectionConsumer] Connection consumer closing due to error in listening thread JBossConnectionConsumer[-2147480495, 44]
javax.jms.JMSException: Failed to get a server session: javax.jms.JMSException: Cannot get session after pool has been closed down.
at org.jboss.jms.asf.StdServerSessionPool.getServerSession(StdServerSessionPool.java:206)
at org.jboss.jms.client.JBossConnectionConsumer.run(JBossConnectionConsumer.java:291)
at java.lang.Thread.run(Thread.java:595)

After this, we cannot undeploy anything, even applications that have nothing to do with MDB's. The only way to terminate the server is to send it a kill -9, which is very bad.

Looking in the JBoss code, it seems that the server session pool is empty and is never refilled. When we perform the undeployment, my guess is a notify() is sent to this thread. Seeing that the pool is closing, it throws the JMSException "cannot get session after pool has been closed".

The question I have is, why isn't the session pool being refilled and why does this hang the undeployment thread?

Many thanks,
Adam

[imaeses]

We solved this problem. Turns out it has nothing to do with JBoss Messaging and everything to do with our code. I'm relating our experiences here in case anyone else encounters a similar problem.

Queue clients are not only a JBoss server hosting MDB's listening to those queues, but also freestanding programs. Because there is a fair number of these freestanding programs, they each cache a QueueConnection. These programs are multithreaded. In short, there was a race condition in our code which caused the QueueConnection.close() to hang. After this occurred enough times, the JBoss session pool ran out of sessions. We observed other strange behavior as well, including the fact that we could not undeploy applications from JBoss listening in on queue's affected by this problem. Our initial approach was to interrupt() one of the deadlocked threads, drop the reference, and create a new one with a fresh QueueConnection. However, onMessage() was called immediately once again, before the deadlock condition could resolve itself. A side-effect of this nonsense was that the JVM filled up with orphaned threads all waiting on QueueConnection.close(). Eventually we had OutOfMemoryErrors.

The race condition never occurred in JBossMQ because it seems that the thread which calls onMessage() in a MessageListener was the same thread as calls onException() in an ExceptionListener. In JBoss Messaging, it appears these are different threads. Our problem was that an onException() called while onMessage() was being processed caused a deadlock under a particular, application-specific condition. After this, QueueConnection.close() would wait for onMessage() to complete, which was forever.

[mskonda]

Good analysis.

How did you resolve?

[imaeses]

Well the solution was breaking the deadlock, which is application specific. As long as the QueueConnection could be reestablished, the deadlock would be broken.

We stopped implementing message receive as a MessageListener and did it ourselves. This had the effect that we could control when receive() was called and how long it would block for. If onException() was called, we could set a flag in the receive thread. The receive thread would eventually return from receive() and close it's QueueSession object. So QueueConnection.close() always works. Then we reestablish the QueueConnection and the deadlock is cleared.

(The deadlock had to do with two threads. Thread A wanted to add a message to a data structure but had to wait() because it was currently full. Thread B wanted to read from this data structure but would first wait() on the same monitor if there was no valid QueueConnection.)

[imaeses]

Thread A was the onMessage() thread.

[santhoshreddy747]

HI,

Can you please send me your MDB code to fix this problem.Im also having same issues.Im also recieving the message as message listener.

Im attaching my MDB code please let me know if i did anything wrong.

Thanks

santhosh

mdb main code:

// Source File Name:   EskomMDB.java

//@author santhosh pulichinthala

package za.eskom.jms.ejb;

import java.util.logging.Level;
import java.util.logging.Logger;

import javax.ejb.EJBException;
import javax.ejb.MessageDrivenBean;
import javax.ejb.MessageDrivenContext;

import javax.jms.MessageListener;
import javax.jms.Queue;

import javax.jms.Message;

import javax.jms.TextMessage;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;

// Referenced classes of package za.eskom.jms.ejb:
//            EscapeChars, GTXSOAPCalling , GetSoapXmlByMsgType , PersistentSoapTarget

public class GtxEskomFourthInstanceMDB implements MessageDrivenBean, MessageListener {

private MessageDrivenContext mdcontext;

String soapTargetValue = null;

private boolean instantiated;

String messageType = "";  

InitialContext initialContext = null;

Thread currentThread = null;

public GtxEskomFourthInstanceMDB() {
  mdcontext = null;
  instantiated = getVariables();
}

public void setMessageDrivenContext(MessageDrivenContext mdc)
   throws EJBException {
  mdcontext = mdc;
}

public void ejbCreate() {

}

public void ejbRemove() {

}

private boolean getVariables() {
  // TODO Auto-generated method stub
  String variableName = "";
  try {
   // createQueueConnetion();
   initialContext = new InitialContext();
   Context context = (Context) initialContext.lookup("java:comp/env");

   variableName = "SoapTargetValue";
   soapTargetValue = (String) context.lookup(variableName);

   variableName = "MessageType";
   messageType = (String) context.lookup(variableName);

   return true;

  } catch (NamingException excepton) {
   logger.log(Level.SEVERE, "Missing environment variable ("
     + variableName + ") from ejb-jar.xml .");
   return false;
  }
}

public void onMessage(Message inMessage) {
  TextMessage msg = null;
  String xmlString = "";
  String xmlToSend = "";
  Queue subscribedQueue;
  try {
      
   subscribedQueue = (Queue) inMessage.getJMSDestination();
      
   logger.log(Level.INFO, "Received JMS message ("
     + inMessage.getJMSMessageID() + ") from Queue ("
     + subscribedQueue.getQueueName() + ").");    

   if (inMessage instanceof TextMessage) {

    msg = (TextMessage) inMessage;

    xmlToSend = msg.getText().toString();

    if (instantiated) {
     logger.log(Level.INFO,
       "MESSAGE BEAN: Got the message Type :  "
         + messageType + " and SoapTargetValue :  "
         + soapTargetValue);

    } else {
     logger.log(Level.SEVERE,
       "MESSAGE BEAN: Failed read values from ejb-jar.xml for xml Message"
         + msg.getText().toString());
     mdcontext.setRollbackOnly();
    }

    xmlString = GetSoapXmlByMsgType.getMessageType(messageType,
      xmlToSend);

    logger.log(Level.INFO,
      "MESSAGE BEAN: Calling Gtx Soap process :");

    GTXSOAPCalling
      .callingGtxSoapProcess(soapTargetValue, xmlString);

   }
  } catch (Throwable e) {
   // TODO Auto-generated catch block
   try {
    mdcontext.setRollbackOnly();
   } catch (Exception f) {
    logger.log(Level.SEVERE,
      "Failure when trying to rollback transaction");
   }
   logger.log(Level.SEVERE,
     "Failed to submit JMS message  with an exception - "
       + e.getMessage());
  }
}

private static final long serialVersionUID = 1L;

private static Logger logger = Logger.getLogger("GtxEskomMessageBean");
}