I'm hosting JBM as an HA-Singleton service on a six node cluster. If a node dies, I could see stuck messages at the database. Is this a bug ?

If the master node dies while messages are being processed, these messages would remain stuck with the given node, until it joins the cluster again. This is because, even though the messages associated with the failed transactions are visible to the new master node, it doesn't have sufficient information (transaction log of the failed node) to recover these messages. So the failed node has to re-join the cluster to recover these messages.

If you're using JBoss-EAP-4.3_CP4 or a higher release, this is the expected behavior. Not a bug. The reason is, JBossTM is configured to recover JBM resources by default from JBoss-EAP-4.3_CP4. The JBossTM uses a transaction log (this is also known as the object store) to write information on the ongoing transaction, during the prepare phase. This transaction log contains sufficient information to provide for the transaction recovery manager, during the recovery process to recover failed transactions. If a transaction fails (due to a server crash or for any other reason) during the prepare phase. The periodic recovery thread (which is a part of recovery manager) scans the object store (transaction log) seeking failed transactions to recover them.

When the failed node rejoins the cluster, periodic recovery thread attached to the recovery manager starts scanning the object store, seeking failed transactions. If there are any records on the transaction log, it realizes that there are failed transactions and starts the transaction recovery process. At this stage however, JBM service is hosted on a different node and it's not running on the same node. But still, it's possible for the newly joined cluster member to obtain an XA resource using the local DefaultJMSProvider. Once this node obtains an instance of the JMSProviderAdapter using the DefaultJMSProvider; it's then possible to obtain an XAConnectionFactory object using the InitialContext of the JMSProviderAdapter. The XAConnectionFactory is all what matters to obtain an XA resource to recover failed transactions.

As explained above, in this case. You don't have to create multiple RemoteJMSProviders for every other JBoss-EAP instance. If you want to verify this, please use the following code snippet on a servlet or a JSP. And make sure to run this code on a node where the JBM service isn't hosted at.

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.transaction.xa.XAResource;
import org.jboss.jms.jndi.JMSProviderAdapter;
import org.jboss.naming.Util;
import javax.jms.XAConnection;
import javax.jms.XAConnectionFactory;
import javax.jms.XASession;

....
....
        XAConnection xac = null;
    try
    {

        String jmsProvider = "java:/DefaultJMSProvider";
        Context ctx = new InitialContext();
            JMSProviderAdapter jmspa = (JMSProviderAdapter) ctx.lookup(jmsProvider);
            String connectionFactoryRef = jmspa.getFactoryRef();
            ctx = jmspa.getInitialContext();
            XAConnectionFactory xacf =  (XAConnectionFactory)Util.lookup(ctx, connectionFactoryRef, XAConnectionFactory.class);
            xac = xacf.createXAConnection();
            XASession xas = xac.createXASession();
            XAResource xar = xas.getXAResource();
        out.println("XA Resource clazz : " + xar.getClass());
    }
    catch(Exception e)
    {
        e.printStackTrace();
    }
    finally
    {
        try
        {
            if(xac != null)
                xac.close();
        }
        catch(Exception ex)
        {
        }    
    }
....
....