6 Replies Latest reply on Mar 23, 2010 10:40 PM by koten ou

    Help!Messages are Lost!

    koten ou Newbie
      Hello everyone!

      Recently I joined a new project about JBOSS-ESB,which using JBOSS-Messaging for exchaning Messages.
      But yesterday ,A problem was found that messages lost when the jboss service was shutdowned normally.

      First of all, could you please tell me whether there is probability in Message-Lost when jboss service be normally shutdowned?
      If YES, could you please tell me there is any way to prevent it?
      If NO,I think it could be some mistakes in my project.Let me show some detail to you.If no enough, I will give more after your reply.

      1.I use the jbossesb-server-4.7 for my project which using JBOSS-Messaging (version-1.4 , I think).
      2.My servers' structure is ,two servers(A1,A2) send messages to their local queue,two servers(B1,B2) listen to the queues both in A1 and A2 ,and send messages to C1.
        Like that
      3.For example , A1 and A2 both send 10000 messages in theirs local queues , B1 and B2 get the messages from A1 and A2 and send them to C1.
      If no servers JBOSS-Services are shutdowned , it is correct that the amount of messages is 20000 in C1.
      If I shutdown B1's JBOSS service normally in processing messages (in windows-services-gui), the amount of message in C1 is less than 20000.

      4.jboss-esb.xml of B1:
      <?xml version="1.0"?>
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd">
        <!-- for A1 -->
        <jms-provider connection-factory="ConnectionFactory"
         jndi-pkg-prefix="org.jboss.naming:org.jnp.interfaces" name="detection_a1">
         <jms-bus busid="detection_a1">
           dest-name="queue/a1" dest-type="QUEUE"/>
        <!-- for A2 -->
        <jms-provider connection-factory="ConnectionFactory"
         jndi-pkg-prefix="org.jboss.naming:org.jnp.interfaces" name="detection_a2">
         <jms-bus busid="detection_a2">
           dest-name="queue/a2" dest-type="QUEUE"/>
        <service category="Srv_Detection" description="service of B1"
         invmScope="GLOBAL" name="detection">
         <property name="maxThreads" value="3" />
          <!-- for A1 -->
          <jms-listener busidref="detection_a1" is-gateway="true" name="det_lis_a1"/>
          <!-- for A2 -->
          <jms-listener busidref="detection_a2" is-gateway="true" name="det_lis_a2"/>
         <actions mep="OneWay">
          <action class="org.jboss.soa.esb.actions.routing.JMSRouter" name="Routing">
           <property name="jndi-URL" value=""/>
           <property name="unwrap" value="true"/>
           <property name="jndiName" value="queue/analysis_log_que"/>

      5.log of B1 in shutdown

      21:43:41,767 INFO  [Server] Shutting down the server, blockingShutdown: false
      21:43:41,767 INFO  [Server] Server exit(0) called
      21:43:41,767 INFO  [Server] Runtime shutdown hook called, forceHalt: true
      21:43:41,767 INFO  [Server] JBoss SHUTDOWN: Undeploying all packages
      21:43:41,798 INFO  [TomcatDeployer] undeploy, ctxPath=/jbpm-console, warUrl=.../tmp/deploy/tmp4205643525614609068jbpm-console-exp.war/
      21:43:41,845 INFO  [JBpmCallback] destroy() invoked
      21:43:42,001 INFO  [JobExecutorThread] JbpmJobExector: leaves cyberspace
      21:43:42,001 INFO  [LockMonitorThread] JbpmJobExector:LockMonitorThread@ got interrupted
      21:43:42,001 INFO  [LockMonitorThread] JbpmJobExector:LockMonitorThread@ leaves cyberspace
      21:43:42,001 INFO  [SessionFactoryImpl] closing
      21:43:42,001 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=JbpmDS' from JNDI name 'java:JbpmDS'
      21:43:42,064 INFO  [H2Database] Database standalone closed clean
      21:43:42,064 INFO  [QueueService] Queue[/queue/CallbackQueue] stopped
      21:43:42,079 WARN  [ServiceMessageCounterLifecycleResource] Calling cleanup on existing service message counters for identity ID-2
      21:43:42,689 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=JBossESBDS' from JNDI name 'java:JBossESBDS'
      21:43:42,845 INFO  [HypersonicDatabase] Database standalone closed clean
      21:43:42,845 INFO  [QueueService] Queue[/queue/OperationsResultCollectorQueue] stopped
      21:43:42,845 INFO  [QueueService] Queue[/queue/OperationsCollectorQueue] stopped
      21:43:42,845 INFO  [QueueService] Queue[/queue/DataCollectorQueue] stopped
      21:43:42,845 INFO  [QueueService] Queue[/queue/DeadMessageQueue] stopped
      21:43:42,845 WARN  [ServiceMessageCounterLifecycleResource] Calling cleanup on existing service message counters for identity ID-1
      21:43:42,907 INFO  [QuartzScheduler] Scheduler ESBScheduler:jbossesb.esb_$_NON_CLUSTERED shutting down.
      21:43:42,907 INFO  [QuartzScheduler] Scheduler ESBScheduler:jbossesb.esb_$_NON_CLUSTERED paused.
      21:43:42,907 INFO  [QuartzScheduler] Scheduler ESBScheduler:jbossesb.esb_$_NON_CLUSTERED shutdown complete.
      21:43:42,907 WARN  [ServiceMessageCounterLifecycleResource] Calling cleanup on existing service message counters for identity ID-0
      21:43:42,939 INFO  [TomcatDeployer] undeploy, ctxPath=/jmx-console, warUrl=.../deploy/jmx-console.war/
      21:43:42,939 INFO  [TomcatDeployer] undeploy, ctxPath=/contract, warUrl=.../tmp/deploy/tmp6746294844412296109contract-exp.war/
      21:43:42,954 INFO  [TomcatDeployer] undeploy, ctxPath=/admin-console, warUrl=.../deploy/admin-console.war/
      21:43:42,954 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' from JNDI name 'java:JmsXA'
      21:43:42,970 INFO  [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@13ccf137 undeployed
      21:43:42,970 INFO  [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@74e551a4 undeployed
      21:43:42,970 INFO  [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@4045548 undeployed
      21:43:42,970 INFO  [QueueService] Queue[/queue/DLQ] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/ExpiryQueue] stopped
      21:43:42,970 INFO  [TopicService] Topic[/topic/testTopic] stopped
      21:43:42,970 INFO  [TopicService] Topic[/topic/securedTopic] stopped
      21:43:42,970 INFO  [TopicService] Topic[/topic/testDurableTopic] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/testQueue] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/A] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/B] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/C] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/D] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/ex] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/PrivateDLQ] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/PrivateExpiryQueue] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/QueueWithOwnDLQAndExpiryQueue] stopped
      21:43:42,970 INFO  [TopicService] Topic[/topic/TopicWithOwnDLQAndExpiryQueue] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/QueueWithOwnRedeliveryDelay] stopped
      21:43:42,970 INFO  [TopicService] Topic[/topic/TopicWithOwnRedeliveryDelay] stopped
      21:43:42,970 INFO  [QueueService] Queue[/queue/testDistributedQueue] stopped
      21:43:42,970 INFO  [TopicService] Topic[/topic/testDistributedTopic] stopped
      21:43:42,970 INFO  [ServerPeer] ServerPeer[0] is Stopping. NOTE! Stopping the server peer cleanly will NOT cause failover to occur
      21:43:43,017 INFO  [ServerPeer] JMS ServerPeer[0] stopped
      21:43:43,017 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' from JNDI name 'java:DefaultDS'
      21:43:43,157 INFO  [HypersonicDatabase] Database standalone closed clean
      21:43:43,157 INFO  [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
      21:43:43,157 INFO  [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
      21:43:43,157 INFO  [SimpleThreadPool] There are still 66 worker threads active. See javadoc runInThread(Runnable) for a possible explanation
      21:43:43,157 INFO  [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
      21:43:43,173 INFO  [JuddiRMIService] Unbinding juddi services
      21:43:43,173 INFO  [Registry] Stopping jUDDI registry...
      21:43:43,173 INFO  [Registry] Shutting down SubscriptionNotifier
      21:43:43,173 INFO  [Registry] jUDDI shutdown completed.
      21:43:43,173 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=juddiDB' from JNDI name 'java:juddiDB'
      21:43:43,189 INFO  [MailService] Mail service 'java:/Mail' removed from JNDI
      21:43:43,204 INFO  [TomcatDeployer] undeploy, ctxPath=/web-console, warUrl=.../deploy/management/console-mgr.sar/web-console.war/
      21:43:43,220 INFO  [Http11Protocol] Coyote HTTP/1.1を http- で一時停止します
      21:43:43,220 INFO  [AjpProtocol] Pausing Coyote AJP/1.3 on ajp-
      21:43:44,220 INFO  [StandardService] サービス jboss.web を停止します
      21:43:44,235 INFO  [Http11Protocol] Coyote HTTP/1.1を http- で停止します
      21:43:44,235 INFO  [AjpProtocol] Stopping Coyote AJP/1.3 on ajp-
      21:43:44,235 INFO  [TomcatDeployer] undeploy, ctxPath=/, warUrl=.../deploy/jboss-web.deployer/ROOT.war/
      21:43:44,235 INFO  [TomcatDeployer] undeploy, ctxPath=/invoker, warUrl=.../deploy/http-invoker.sar/invoker.war/
      21:43:44,235 INFO  [TomcatDeployer] undeploy, ctxPath=/jbossws, warUrl=.../deploy/jbossws.sar/jbossws-context.war/
      21:43:44,454 INFO  [HypersonicDatabase] Database standalone closed clean
      21:43:44,564 INFO  [InVMTemporaryTransport] Stopping reaper thread
      21:43:46,189 INFO  [TransactionManagerService] Stopping recovery manager
      21:43:46,251 INFO  [Server] Shutdown complete
      Shutdown complete
      Halting VM
      Shutdown JBossESB Server 4.7 service [2010-03-16 21:43:47]

      thank you .I'm expecting for to your reply.
        • 1. Re: Help!Messages are Lost!
          Hans Wolffenbuttel Expert



          Two things you might want to consider:

          - are you persisting messages? If not then there is no way JBoss is able to send them after shutdown, because the messages are all inmemory.

          - have you considered transactional messaging with a transactional connection factory? This way when a transaction cant be completed you dont loose your message.





          • 2. Re: Help!Messages are Lost!
            koten ou Newbie

            Hi Hans


            Thanks for your reply.


            I think I did nothing for transactional ConnectionFactory.I will attempt to do  something about this.


            But,when I send messages just with A1 ,alone, it seems that no message loss in processing, maybe it is accidental.


            By the way , I want to know that ,if it was transactional configured, the way  I stop JBOSS service or stop depolyed services affect to the message loss ?

            For example, I stop the service on the console-webpage or windows gui,or it be shutdown by some reasons unknow.



            • 3. Re: Help!Messages are Lost!
              Hans Wolffenbuttel Expert



              Sending messages is 'never' a problem when JBoss is not shut down as you have already aknowledged. The problem resides in the handeling of the message in case of a scheduled or sudden shutdown. When shutdown properly JBossESB tries to send all accepted messages (as far as I know) before shutting down. When a sudden shutdown takes place it is important to know where your message should be next time JBoss is rebooted. If  they are in a persisted queue, JBoss can resend those messages from that queue. I don't know what will happen if you don't persist the  queued messages.





              • 4. Re: Help!Messages are Lost!
                koten ou Newbie

                Hi hans


                Thank you for your reply and I think it helpful for me to deal with my problem.


                Now I'm trying using transactional ConnectionFactory (XAConnectionFactory) to prevent recieved messages from losting when I shut down the JBoss-server.

                But now it is not well performing for that it seems unable to listen to a remote queue.(I'm looking for several ways to solve it ,but it still unvailable for remote queue)

                If you know something in cofiguring it to a remote queue, do you mind give me your success experience?


                On the other hand, as a JBOSS-Fresher,could you please give some advice in message-persisting?



                thank you



                • 5. Re: Help!Messages are Lost!
                  Hans Wolffenbuttel Expert



                  For your remote queueing problem you could look at the folowing thread http://community.jboss.org/message/531055#531055 where this is dicussed in more detail. Also the link for the discussion mentioned there is http://community.jboss.org/message/533165#533165 . Which has at the moment no replies yet. I have no experience in the remote JMS queue usage yet, so I can't elaborate any futher.


                  For persisting messages you can look at the quickstarts and at http://community.jboss.org/wiki/CombiningESBJPAHibernateJTAandSpring#cf how to persist with Hibernate and JPA. For persisting queued messages you can easily change the messaging datasource from HSQL to a database of your preference. Be advised you do not use inVM parameter in your queue definition because then your messages won't be persisted.





                  • 6. Re: Help!Messages are Lost!
                    koten ou Newbie

                    Hi hans


                    Thank you for your  valuable advice.

                    I have find out that it is InVM parameter which makes the message lost,and I think it helpful about the URL you gave to me.


                    Thanks again