0 Replies Latest reply on Oct 29, 2010 4:35 AM by burmanm

    Transactions, JCA/JTA, JMS and delivery problems

    burmanm

      Hi,

       

      Background is that we have JBoss Messaging's storage in the SQLServer 2008R2. This works just fine, however we want to ensure no messages are lost if there's network blackout between the ESB and the JMS queues. Network blackout removes the backend from the JMS queues, so nothing should happen in the meanwhile. So normal jms-providers won't work, since if we read something from the queue and can't put it to the next queue, the message is lost. DLQ is unavailable as well in that case. jUDDI and ESB's message table runs in the same SQLServer.

       

      What I've tried is to configure everything as jms-jca-providers and datasources as xa-datasource, and setting JMS busses with transacted="true". What we should end up with is transacted services and hopefully secured delivery, so no messages would be lost and the process would continue when network connection is restored.

       

      However, when network connection is lost, so is jUDDI and JMS. So no DLQ or JMS queues can be found anymore. Fine, but I can't even test this far. What happens when I stick enough messages in to the ESB is that most of the messages are just unable to be delivered. Out of 40k messages 4k are delivered successfully through the ESB's services. That's 90% loss. Arf. First messages go through just fine, but then the errors start. Neither the SQLServer nor the ESB server are overloaded, both have plenty of CPU juice still left (for some reason, using the JCA adapters causes slowdowns, instead of proper 100% CPU utilization, I'm left at somewhere 60%). All the jms-listeners have something like 16 threads in my case.

       

      2010-10-27 18:43:58,439 ERROR [org.jboss.soa.esb.client.MessageMulticaster] (WorkManager(2)-52) Failed to deliver message to Service '\
      Validation:PreRoutingValidationService'.  Delivering message to Dead Letter Channel.^M

       

      I start to see these, so .. for some reason message couldn't be delivered to the queue. In the beginning, at the least the DLQ is still working:

       

      2010-10-27 18:43:58,495 INFO  [org.jboss.soa.esb.client.ServiceInvoker] (WorkManager(2)-86) Delivering message [header: [  RelatesTo: \
      jms:correlationID#038808b0-7c62-43bf-9965-5c683bd6bc53 ]] to RDLVRQ.^M

       

      However, not for long:

       

      2010-10-27 18:44:43,545 ERROR [fi.logica.bis.heracles.esb.actions.exception.ExceptionHandlerAction] (WorkManager(2)-11) Delivering exc\
      eption message failed.^M
      org.jboss.soa.esb.listeners.message.MissingServiceException: Registry details for service [JBossESB-Internal:DeadLetterService] could \
      not be determined from the registry.^M

       

      [28.10.2010 11:08:47] Michael Burman: 2010-10-27 18:44:43,545 WARN  [org.jboss.soa.esb.listeners.message.ActionProcessingPipeline] (WorkManager(2)-75) Failed to send response failure to DLQ service^M

       

      So it can't anymore find the DeadLetterService? And from there, it won't recover anymore. Rest of the server logs are just full of:

       

      2010-10-27 18:44:43,557 ERROR [org.jboss.resource.adapter.jms.inflow.JmsServerSession] (WorkManager(2)-46) Unexpected error delivering\
      message delegator->JBossMessage[21105749221985308]:NON-PERSISTENT, deliveryId=37416^M
      org.jboss.soa.esb.listeners.jca.JcaGatewayException: Unexpected exception caught while processing JCA message^M

       

      All this in a situation, where both the JMS and jUDDI are still responding. What's happening here? How can I get reliable messaging at high rate? Priority #1 is delivering the message, I can even accept occasional duplicates. But losing something is not an option.

       

      Should DLQ be transacted="true" also? These errors are quite common after DLQ has been populated:

       

      17:09:00,011 WARN  [loggerI18N] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource< 131075, 30, 28, 494545535110152975398515810149545258529999565055505658535550494553511015297539851581014954525852999956505550565853555050 >

       

      But even cleaning up the queues causes this in the starting process of JBossAS:

       

      10:45:45,560 WARN  [loggerI18N] [com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror] [com.arjuna.ats.internal.jta.resources.arjunacore.rollbackxaerror] XAResourceRecord.rollback - xa error XAException.XAER_NOTA