JMS Error in cluster

[irfanmasood_015]

Hi Dears,

I have start receiving below error in server.log file suddenly, this happens while the application tries to send a JMS message to queue

2011-07-10 10:24:19,042 ERROR [org.jboss.resource.adapter.jms.JmsSessionFactoryImpl] could not create session

org.jboss.resource.JBossResourceException: Unexpected throwable while trying to create a connection: null; - nested throwable: (java.lang.IllegalArgumentException: factory is null)

    at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:246)

    at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:534)

    at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:444)

    at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:312)

    at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

    at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:887)

    at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:369)

    at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createQueueSession(JmsSessionFactoryImpl.java:137)

.........................

.............................

The part of log also contains

javax.jms.JMSException: Error creating the dlq connection: null

    at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:171)

    at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:237)

    at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:164)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerCreate(JMSContainerInvoker.java:542)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker.startService(JMSContainerInvoker.java:764)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl.onException(JMSContainerInvoker.java:1267)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker$1.run(JMSContainerInvoker.java:776)

2011-07-10 06:45:14,155 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider

2011-07-10 06:45:14,155 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider

2011-07-10 06:45:14,155 WARN  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] JMS provider failure detected:

javax.jms.JMSException: Error creating the dlq connection: null

    at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:171)

    at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:237)

    at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:164)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerCreate(JMSContainerInvoker.java:542)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker.startService(JMSContainerInvoker.java:764)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl.onException(JMSContainerInvoker.java:1267)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker$1.run(JMSContainerInvoker.java:776)

2011-07-10 06:45:14,155 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider

This all happened when we restarted the Jboss nodes running in cluster

BEFORE restartting the nodes below error was printing heavily in log file

2011-07-10 02:18:04,850 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackLocalException in method: public abstract void javax.jms.MessageListener.onMessage(javax.jms.Message), causedBy:

javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=5000

    at org.jboss.ejb.plugins.AbstractInstancePool.get(AbstractInstancePool.java:132)

    at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:48)

    at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)

    at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:315)

    at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:148)

    at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:90)

    at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)

    at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)

    at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:372)

    at org.jboss.ejb.Container.invoke(Container.java:723)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:914)

    at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1208)

    at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:276)

    at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:871)

    at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:159)

    at org.jboss.mq.SpySession.run(SpySession.java:347)

    at org.jboss.jms.asf.StdServerSession.run0(StdServerSession.java:200)

    at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180)

    at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)

    at java.lang.Thread.run(Thread.java:534)

I will really gratefull for help!

Best Regards!

Irfan

[irfanmasood_015]

I also see below warning in the logs many times

2011-07-12 10:47:49,744 WARN  [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] javax.naming.NameNotFoundException: XAConnectionFactory not bound

This warning is printed after jmsXA is deployed as shown below

2011-07-12 10:47:49,166 INFO  [org.jboss.resource.adapter.jms.JmsManagedConnectionFactory.JmsXA] Bound connection factory for resource adapter for ConnectionManager 'jboss.jca:service=TxCM,name=JmsXA to JNDI name 'java:/JmsXA'

But before the MDBs are deployed.

I am using JBoss 3.2.5 with JDK 1.4.2.

In hajndi-jms-ds.xml file factory is configured as shown below:

<tx-connection-factory>

    <jndi-name>JmsXA</jndi-name>

    <xa-transaction/>

    <adapter-display-name>JMS Adapter</adapter-display-name>

    <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>

    <config-property name="JmsProviderAdapterJNDI" type="java.lang.String">java:/DefaultJMSProvider</config-property>

    <max-pool-size>115</max-pool-size>

    <security-domain-and-application>JmsXARealm</security-domain-and-application>

  </tx-connection-factory>

One problem that might be related is that in DB the DLQ has about 210000 entries.

Any help will be higly appreciated.

Best Regards!

Irfan

[irfanmasood_015]

By removing the failed message from DLQ and Stoping both nodes and then starting them one by one resolved the issue, still not sure what was exactly the issue, funny JBoss