2 Replies Latest reply on Aug 29, 2008 9:48 PM by vimalm

    Sonic JCA Resource Adapter Wait for Available Session Ended

    vimalm

      Hi

      We are using JBOSS 4.0.4GA along with Sonic JCA Resource adapter. I have a MDB deployed on JBOSS which listens on a Topic on Sonic ESB. Under heavy load on this topic, my MDB appears to turn deaf after processing 10000 or so messages on this topic. I see in the server.log messages like one shown below:

      2006-12-22 02:43:48,433 DEBUG [Sonic Resource Adapter for JCA] SonicServerSessionPool.getServerSession() SessionMessageProcessor (34) 1: TopicSubscriber msg handler, $CC$$TS$wait for available session
      


      2006-12-22 02:43:48,525 DEBUG [Sonic Resource Adapter for JCA] SonicServerSessionPool.getServerSession() SessionMessageProcessor (27) 1: TopicSubscriber msg handler, $CC$$TS$ wait for available session ended
      


      This phenomenan is fairly reproducible. I wrote a stand along java code, which will publish about 21000 messages on the topic by reading stuff from a file. This program takes about 5 minutes to dump these 21000 messages on the topic.

      Structure of this client program is fairly standard and simple. Given below in case you are interested to have a look.

      InitialContext ctx = new InitialContext(properties);
      ConnectionFactory connectionFactory = (ConnectionFactory) client.lookup(ctx, "ConnectionFactory");
      javax.jms.Connection connection = connectionFactory.createConnection();
      javax.jms.Session session = connection.createSession(false,Session.AUTO_ACKNOWLEDGE);
      Topic tp = session.createTopic("TopicName");
      MessageProducer messageProducer = session.createProducer(tp);
      BufferedReader in = new BufferedReader(new FileReader("data.txt"));
      String str = null;
      while((str = in.readLine()) !=null) {
       TextMessage message = session.createTextMessage(str);
       messageProducer.send(message, DeliveryMode.PERSISTENT, 4, 5400000);
       System.out.println("Published message");
      }
      // Do stuff to close BufferedReader, MessageProducer, Session and Connection and in that order
      


      Now my MDB takes about 300 ms on an average to process the message. This involves updating about 10 tables on an Oracle database using a local-tx datasource.

      Here is the ejb-jar.xml for the MDB
      <enterprise-beans>
       <message-driven>
       <ejb-name>SomeMDB</ejb-name>
       <ejb-class>mypackage.SomeMDB</ejb-class>
       <messaging-type>javax.jms.MessageListener</messaging-type>
       <transaction-type>Bean</transaction-type>
       <activation-config>
       <activation-config-property>
       <activation-config-property-name>destinationType</activation-config-property-name>
       <activation-config-property-value>javax.jms.Topic</activation-config-property-value>
       </activation-config-property>
       <activation-config-property>
       <activation-config-property-name>destination</activation-config-property-name>
       <activation-config-property-value>[[sharedsub]]TopicName</activation-config-property-value>
       </activation-config-property>
       <activation-config-property>
       <activation-config-property-name>subscriptionDurability</activation-config-property-name>
       <activation-config-property-value>Durable</activation-config-property-value>
       </activation-config-property>
       <activation-config-property>
       <activation-config-property-name>subscriptionName</activation-config-property-name>
       <activation-config-property-value>SomeMDBDurableSub</activation-config-property-value>
       </activation-config-property>
       <activation-config-property>
       <activation-config-property-name>clientID</activation-config-property-name>
       <activation-config-property-value>SomeMDB</activation-config-property-value>
       </activation-config-property>
       <activation-config-property>
       <activation-config-property-name>reconnectTimeout</activation-config-property-name>
       <activation-config-property-value>5</activation-config-property-value>
       </activation-config-property>
       </activation-config>
       </message-driven>
       </enterprise-beans>
      
       <!-- Assembly Descriptor -->
       <assembly-descriptor>
       <container-transaction>
       <method>
       <ejb-name>SomeMDB</ejb-name>
       <method-name>*</method-name>
       </method>
       <trans-attribute>NotSupported</trans-attribute>
       </container-transaction>
      
       </assembly-descriptor>
      


      Here is the code for MDB
      public void onMessage(Message message) {
       TextMessage txtMess = (TextMessage)message;
       Connection con = null;
       try {
       String xml = txtMess.getText();
       log.debug("SomeMDB|onMessage|Message Driven Bean Got Message: " + xml);
       InitialContext ctx = new InitialContext();
       DataSource ds = (DataSource) ctx.lookup("java:jdbc/MyDataSource");
       con = ds.getConnection();
       con.setAutoCommit(false);
       dp = new DataProcessor(con);
       dp.process(xml); // Process message to take about 150 ms
       con.commit();
       } catch(Throwable e) {
       log.error("Caught Throwable " + e.getMessage() , e);
       try {
       con.rollback();
       } catch(Throwable t) {
       log.error("CAn't rollback" , t);
       }
       } finally {
       try {
       con.close();
       } catch(SQLException e1) {
       log.error("Can't close connection " , e1);
       }
       }
      }
      


      The echo message in SomeMDB stops appearing the log after processing about 10000 messages.

      I have tried a lot of permutations. Some of them being:

      1. Instead of using Bean managed transaction with notsupported transactiontype, use a CM transaction with Required transaction type. That causes another type of problem because Sonic JCA adapter appears to start a XA transaction and I was using the local-tx for oracle. I got a series of about 5 transaction time out. Further messages did not get processed.
      2. I tried a CM Transaction with Required Transaction Type with Oracle XA transaction which dropped messages too. But I did not get any more transaction time out. The error message was the same as shown near the top.
      3. I started off with my client doing a producer.send(textmessage) and my MDB not being durable which did not work too.

      Here is what works and is very weird.
      In my publisher code, I add a delay of 300 ms between the messages produced by doing a Thread.sleep(300). My MDB now seems to respond well and consume all the message.


      Per SonicMQ documentation, if the publisher is publishing PERSISTENT messages and MDB is durable that will gurantee message delivery. But it does not appear so. I crashed my container after my MDB stopped listening and restarted it in the hope that it will pick up the message. But my MDB did not pick up the message.

      Durable subscription works and I have tested it. So there should not be a problem with activation config.

      We probably can't ask the publisher in a production like environment to add this delay between the messages so that we can consume them because they run several production jobs sequentially and by adding this delay there entire job processing time will be increased by 1.5 hours or so.

      Does any one have run into this problem? Any help will be appreciated.

      I have searched into the forum and have read the issue with concurrent.jar etc. That should not be a problem because we are using such a newer version of JBoss.

      Any thoughts will be appreciated. I am going to try to add the Jboss MQ trace level logging to see if that gives me any more explanation.

      Happy new year.
      Vimal