2 Replies Latest reply on Jul 16, 2011 11:29 PM by Muhammad Irfan Masood

    JMS Error in cluster

    Muhammad Irfan Masood Newbie

      Hi Dears,

       

      I have start receiving below error in server.log file suddenly, this happens while the application tries to send a JMS message to queue

       

      2011-07-10 10:24:19,042 ERROR [org.jboss.resource.adapter.jms.JmsSessionFactoryImpl] could not create session

      org.jboss.resource.JBossResourceException: Unexpected throwable while trying to create a connection: null; - nested throwable: (java.lang.IllegalArgumentException: factory is null)

          at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:246)

          at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:534)

          at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:444)

          at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:312)

          at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)

          at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:887)

          at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.allocateConnection(JmsSessionFactoryImpl.java:369)

          at org.jboss.resource.adapter.jms.JmsSessionFactoryImpl.createQueueSession(JmsSessionFactoryImpl.java:137)

      .........................

      .............................

       

      The part of log also contains

       

      javax.jms.JMSException: Error creating the dlq connection: null

          at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:171)

          at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:237)

          at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:164)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerCreate(JMSContainerInvoker.java:542)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker.startService(JMSContainerInvoker.java:764)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl.onException(JMSContainerInvoker.java:1267)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker$1.run(JMSContainerInvoker.java:776)

      2011-07-10 06:45:14,155 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Reconnected to JMS provider

      2011-07-10 06:45:14,155 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider

      2011-07-10 06:45:14,155 WARN  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] JMS provider failure detected:

      javax.jms.JMSException: Error creating the dlq connection: null

          at org.jboss.ejb.plugins.jms.DLQHandler.createService(DLQHandler.java:171)

          at org.jboss.system.ServiceMBeanSupport.jbossInternalCreate(ServiceMBeanSupport.java:237)

          at org.jboss.system.ServiceMBeanSupport.create(ServiceMBeanSupport.java:164)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker.innerCreate(JMSContainerInvoker.java:542)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker.startService(JMSContainerInvoker.java:764)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker$ExceptionListenerImpl.onException(JMSContainerInvoker.java:1267)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker$1.run(JMSContainerInvoker.java:776)

      2011-07-10 06:45:14,155 INFO  [org.jboss.ejb.plugins.jms.JMSContainerInvoker] Trying to reconnect to JMS provider

       

      This all happened when we restarted the Jboss nodes running in cluster

       

      BEFORE restartting the nodes below error was printing heavily in log file

       

      2011-07-10 02:18:04,850 ERROR [org.jboss.ejb.plugins.LogInterceptor] TransactionRolledbackLocalException in method: public abstract void javax.jms.MessageListener.onMessage(javax.jms.Message), causedBy:

      javax.ejb.EJBException: Failed to acquire the pool semaphore, strictTimeout=5000

          at org.jboss.ejb.plugins.AbstractInstancePool.get(AbstractInstancePool.java:132)

          at org.jboss.ejb.plugins.MessageDrivenInstanceInterceptor.invoke(MessageDrivenInstanceInterceptor.java:48)

          at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:84)

          at org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java:315)

          at org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java:148)

          at org.jboss.ejb.plugins.RunAsSecurityInterceptor.invoke(RunAsSecurityInterceptor.java:90)

          at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java:191)

          at org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java:122)

          at org.jboss.ejb.MessageDrivenContainer.internalInvoke(MessageDrivenContainer.java:372)

          at org.jboss.ejb.Container.invoke(Container.java:723)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker.invoke(JMSContainerInvoker.java:914)

          at org.jboss.ejb.plugins.jms.JMSContainerInvoker$MessageListenerImpl.onMessage(JMSContainerInvoker.java:1208)

          at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:276)

          at org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:871)

          at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:159)

          at org.jboss.mq.SpySession.run(SpySession.java:347)

          at org.jboss.jms.asf.StdServerSession.run0(StdServerSession.java:200)

          at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:180)

          at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:748)

          at java.lang.Thread.run(Thread.java:534)

       

      I will really gratefull for help!

       

      Best Regards!

       

      Irfan

        • 1. Re: JMS Error in cluster
          Muhammad Irfan Masood Newbie

          I also see below warning in the logs many times

           

          2011-07-12 10:47:49,744 WARN  [org.jboss.ha.framework.interfaces.HAPartition.DefaultPartition] javax.naming.NameNotFoundException: XAConnectionFactory not bound

           

          This warning is printed after jmsXA is deployed as shown below

           

          2011-07-12 10:47:49,166 INFO  [org.jboss.resource.adapter.jms.JmsManagedConnectionFactory.JmsXA] Bound connection factory for resource adapter for ConnectionManager 'jboss.jca:service=TxCM,name=JmsXA to JNDI name 'java:/JmsXA'

           

          But before the MDBs are deployed.

           

          I am using JBoss 3.2.5 with JDK 1.4.2.

           

          In hajndi-jms-ds.xml file factory is configured as shown below:

           

          <tx-connection-factory>

              <jndi-name>JmsXA</jndi-name>

              <xa-transaction/>

              <adapter-display-name>JMS Adapter</adapter-display-name>

              <config-property name="SessionDefaultType" type="java.lang.String">javax.jms.Topic</config-property>

              <config-property name="JmsProviderAdapterJNDI" type="java.lang.String">java:/DefaultJMSProvider</config-property>

              <max-pool-size>115</max-pool-size>

              <security-domain-and-application>JmsXARealm</security-domain-and-application>

            </tx-connection-factory>

           

           

          One problem that might be related is that in DB the DLQ has about 210000 entries.

           

          Any help will be higly appreciated.

           

          Best Regards!

           

          Irfan

          • 2. Re: JMS Error in cluster
            Muhammad Irfan Masood Newbie

            By removing the failed message from DLQ and Stoping both nodes and then starting them one by one resolved the issue, still not sure what was exactly the issue, funny JBoss