Help!Messages are Lost!

[okotenn]

Hello everyone!

Recently I joined a new project about JBOSS-ESB,which using JBOSS-Messaging for exchaning Messages.
But yesterday ,A problem was found that messages lost when the jboss service was shutdowned normally.

First of all, could you please tell me whether there is probability in Message-Lost when jboss service be normally shutdowned?
If YES, could you please tell me there is any way to prevent it?
If NO,I think it could be some mistakes in my project.Let me show some detail to you.If no enough, I will give more after your reply.

1.I use the jbossesb-server-4.7 for my project which using JBOSS-Messaging (version-1.4 , I think).
2.My servers' structure is ,two servers(A1,A2) send messages to their local queue,two servers(B1,B2) listen to the queues both in A1 and A2 ,and send messages to C1.
  Like that
 
3.For example , A1 and A2 both send 10000 messages in theirs local queues , B1 and B2 get the messages from A1 and A2 and send them to C1.
If no servers JBOSS-Services are shutdowned , it is correct that the amount of messages is 20000 in C1.
If I shutdown B1's JBOSS service normally in processing messages (in windows-services-gui), the amount of message in C1 is less than 20000.

4.jboss-esb.xml of B1:
<?xml version="1.0"?>
<jbossesb
xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.1.xsd">
<providers>
  <!-- for A1 -->
  <jms-provider connection-factory="ConnectionFactory"
   jndi-URL="jnp://192.168.20.41:1099"
   jndi-context-factory="org.jnp.interfaces.NamingContextFactory"
   jndi-pkg-prefix="org.jboss.naming:org.jnp.interfaces" name="detection_a1">
   <jms-bus busid="detection_a1">
    <jms-message-filter
     dest-name="queue/a1" dest-type="QUEUE"/>
   </jms-bus>
  </jms-provider>
  <!-- for A2 -->
  <jms-provider connection-factory="ConnectionFactory"
   jndi-URL="jnp://192.168.20.42:1099"
   jndi-context-factory="org.jnp.interfaces.NamingContextFactory"
   jndi-pkg-prefix="org.jboss.naming:org.jnp.interfaces" name="detection_a2">
   <jms-bus busid="detection_a2">
    <jms-message-filter
     dest-name="queue/a2" dest-type="QUEUE"/>
   </jms-bus>
  </jms-provider>
</providers>
<services>
  <service category="Srv_Detection" description="service of B1"
   invmScope="GLOBAL" name="detection">
   <property name="maxThreads" value="3" />
   <listeners>
    <!-- for A1 -->
    <jms-listener busidref="detection_a1" is-gateway="true" name="det_lis_a1"/>
    <!-- for A2 -->
    <jms-listener busidref="detection_a2" is-gateway="true" name="det_lis_a2"/>
   </listeners>
   <actions mep="OneWay">
    <action class="org.jboss.soa.esb.actions.routing.JMSRouter" name="Routing">
     <property name="jndi-URL" value="192.168.40.61:1099"/>
     <property name="unwrap" value="true"/>
     <property name="jndiName" value="queue/analysis_log_que"/>
    </action>
   </actions>
  </service>
</services>
</jbossesb>

5.log of B1 in shutdown

21:43:41,767 INFO  [Server] Shutting down the server, blockingShutdown: false
21:43:41,767 INFO  [Server] Server exit(0) called
21:43:41,767 INFO  [Server] Runtime shutdown hook called, forceHalt: true
21:43:41,767 INFO  [Server] JBoss SHUTDOWN: Undeploying all packages
21:43:41,798 INFO  [TomcatDeployer] undeploy, ctxPath=/jbpm-console, warUrl=.../tmp/deploy/tmp4205643525614609068jbpm-console-exp.war/
21:43:41,845 INFO  [JBpmCallback] destroy() invoked
21:43:42,001 INFO  [JobExecutorThread] JbpmJobExector:192.168.100.51:1 leaves cyberspace
21:43:42,001 INFO  [LockMonitorThread] JbpmJobExector:LockMonitorThread@192.168.100.51 got interrupted
21:43:42,001 INFO  [LockMonitorThread] JbpmJobExector:LockMonitorThread@192.168.100.51 leaves cyberspace
21:43:42,001 INFO  [SessionFactoryImpl] closing
21:43:42,001 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=JbpmDS' from JNDI name 'java:JbpmDS'
21:43:42,064 INFO  [H2Database] Database standalone closed clean
21:43:42,064 INFO  [QueueService] Queue[/queue/CallbackQueue] stopped
21:43:42,079 WARN  [ServiceMessageCounterLifecycleResource] Calling cleanup on existing service message counters for identity ID-2
21:43:42,689 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=JBossESBDS' from JNDI name 'java:JBossESBDS'
21:43:42,845 INFO  [HypersonicDatabase] Database standalone closed clean
21:43:42,845 INFO  [QueueService] Queue[/queue/OperationsResultCollectorQueue] stopped
21:43:42,845 INFO  [QueueService] Queue[/queue/OperationsCollectorQueue] stopped
21:43:42,845 INFO  [QueueService] Queue[/queue/DataCollectorQueue] stopped
21:43:42,845 INFO  [QueueService] Queue[/queue/DeadMessageQueue] stopped
21:43:42,845 WARN  [ServiceMessageCounterLifecycleResource] Calling cleanup on existing service message counters for identity ID-1
21:43:42,907 INFO  [QuartzScheduler] Scheduler ESBScheduler:jbossesb.esb_$_NON_CLUSTERED shutting down.
21:43:42,907 INFO  [QuartzScheduler] Scheduler ESBScheduler:jbossesb.esb_$_NON_CLUSTERED paused.
21:43:42,907 INFO  [QuartzScheduler] Scheduler ESBScheduler:jbossesb.esb_$_NON_CLUSTERED shutdown complete.
21:43:42,907 WARN  [ServiceMessageCounterLifecycleResource] Calling cleanup on existing service message counters for identity ID-0
21:43:42,939 INFO  [TomcatDeployer] undeploy, ctxPath=/jmx-console, warUrl=.../deploy/jmx-console.war/
21:43:42,939 INFO  [TomcatDeployer] undeploy, ctxPath=/contract, warUrl=.../tmp/deploy/tmp6746294844412296109contract-exp.war/
21:43:42,954 INFO  [TomcatDeployer] undeploy, ctxPath=/admin-console, warUrl=.../deploy/admin-console.war/
21:43:42,954 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=ConnectionFactoryBinding,name=JmsXA' from JNDI name 'java:JmsXA'
21:43:42,970 INFO  [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@13ccf137 undeployed
21:43:42,970 INFO  [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@74e551a4 undeployed
21:43:42,970 INFO  [ConnectionFactory] org.jboss.jms.server.connectionfactory.ConnectionFactory@4045548 undeployed
21:43:42,970 INFO  [QueueService] Queue[/queue/DLQ] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/ExpiryQueue] stopped
21:43:42,970 INFO  [TopicService] Topic[/topic/testTopic] stopped
21:43:42,970 INFO  [TopicService] Topic[/topic/securedTopic] stopped
21:43:42,970 INFO  [TopicService] Topic[/topic/testDurableTopic] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/testQueue] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/A] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/B] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/C] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/D] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/ex] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/PrivateDLQ] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/PrivateExpiryQueue] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/QueueWithOwnDLQAndExpiryQueue] stopped
21:43:42,970 INFO  [TopicService] Topic[/topic/TopicWithOwnDLQAndExpiryQueue] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/QueueWithOwnRedeliveryDelay] stopped
21:43:42,970 INFO  [TopicService] Topic[/topic/TopicWithOwnRedeliveryDelay] stopped
21:43:42,970 INFO  [QueueService] Queue[/queue/testDistributedQueue] stopped
21:43:42,970 INFO  [TopicService] Topic[/topic/testDistributedTopic] stopped
21:43:42,970 INFO  [ServerPeer] ServerPeer[0] is Stopping. NOTE! Stopping the server peer cleanly will NOT cause failover to occur
21:43:43,017 INFO  [ServerPeer] JMS ServerPeer[0] stopped
21:43:43,017 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=DefaultDS' from JNDI name 'java:DefaultDS'
21:43:43,157 INFO  [HypersonicDatabase] Database standalone closed clean
21:43:43,157 INFO  [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
21:43:43,157 INFO  [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
21:43:43,157 INFO  [SimpleThreadPool] There are still 66 worker threads active. See javadoc runInThread(Runnable) for a possible explanation
21:43:43,157 INFO  [QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
21:43:43,173 INFO  [JuddiRMIService] Unbinding juddi services
21:43:43,173 INFO  [Registry] Stopping jUDDI registry...
21:43:43,173 INFO  [Registry] Shutting down SubscriptionNotifier
21:43:43,173 INFO  [Registry] jUDDI shutdown completed.
21:43:43,173 INFO  [ConnectionFactoryBindingService] Unbound ConnectionManager 'jboss.jca:service=DataSourceBinding,name=juddiDB' from JNDI name 'java:juddiDB'
21:43:43,189 INFO  [MailService] Mail service 'java:/Mail' removed from JNDI
21:43:43,204 INFO  [TomcatDeployer] undeploy, ctxPath=/web-console, warUrl=.../deploy/management/console-mgr.sar/web-console.war/
21:43:43,220 INFO  [Http11Protocol] Coyote HTTP/1.1を http-0.0.0.0-8080 で一時停止します
21:43:43,220 INFO  [AjpProtocol] Pausing Coyote AJP/1.3 on ajp-0.0.0.0-8009
21:43:44,220 INFO  [StandardService] サービス jboss.web を停止します
21:43:44,235 INFO  [Http11Protocol] Coyote HTTP/1.1を http-0.0.0.0-8080 で停止します
21:43:44,235 INFO  [AjpProtocol] Stopping Coyote AJP/1.3 on ajp-0.0.0.0-8009
21:43:44,235 INFO  [TomcatDeployer] undeploy, ctxPath=/, warUrl=.../deploy/jboss-web.deployer/ROOT.war/
21:43:44,235 INFO  [TomcatDeployer] undeploy, ctxPath=/invoker, warUrl=.../deploy/http-invoker.sar/invoker.war/
21:43:44,235 INFO  [TomcatDeployer] undeploy, ctxPath=/jbossws, warUrl=.../deploy/jbossws.sar/jbossws-context.war/
21:43:44,454 INFO  [HypersonicDatabase] Database standalone closed clean
21:43:44,564 INFO  [InVMTemporaryTransport] Stopping reaper thread
21:43:46,189 INFO  [TransactionManagerService] Stopping recovery manager
21:43:46,251 INFO  [Server] Shutdown complete
Shutdown complete
Halting VM
Shutdown JBossESB Server 4.7 service [2010-03-16 21:43:47]


thank you .I'm expecting for to your reply.

[h.wolffenbuttel]

Hi,

Two things you might want to consider:

- are you persisting messages? If not then there is no way JBoss is able to send them after shutdown, because the messages are all inmemory.

- have you considered transactional messaging with a transactional connection factory? This way when a transaction cant be completed you dont loose your message.

Regards,

Hans

[okotenn]

Hi Hans

Thanks for your reply.

I think I did nothing for transactional ConnectionFactory.I will attempt to do  something about this.

But,when I send messages just with A1 ,alone, it seems that no message loss in processing, maybe it is accidental.

By the way , I want to know that ,if it was transactional configured, the way  I stop JBOSS service or stop depolyed services affect to the message loss ?

For example, I stop the service on the console-webpage or windows gui,or it be shutdown by some reasons unknow.

regards

[h.wolffenbuttel]

Hi,

Sending messages is 'never' a problem when JBoss is not shut down as you have already aknowledged. The problem resides in the handeling of the message in case of a scheduled or sudden shutdown. When shutdown properly JBossESB tries to send all accepted messages (as far as I know) before shutting down. When a sudden shutdown takes place it is important to know where your message should be next time JBoss is rebooted. If  they are in a persisted queue, JBoss can resend those messages from that queue. I don't know what will happen if you don't persist the  queued messages.

Regards,

Hans

[okotenn]

Hi hans

Thank you for your reply and I think it helpful for me to deal with my problem.

Now I'm trying using transactional ConnectionFactory (XAConnectionFactory) to prevent recieved messages from losting when I shut down the JBoss-server.

But now it is not well performing for that it seems unable to listen to a remote queue.(I'm looking for several ways to solve it ,but it still unvailable for remote queue)

If you know something in cofiguring it to a remote queue, do you mind give me your success experience?

On the other hand, as a JBOSS-Fresher,could you please give some advice in message-persisting?

thank you

regards

[h.wolffenbuttel]

Hi,

For your remote queueing problem you could look at the folowing thread http://community.jboss.org/message/531055#531055 where this is dicussed in more detail. Also the link for the discussion mentioned there is http://community.jboss.org/message/533165#533165 . Which has at the moment no replies yet. I have no experience in the remote JMS queue usage yet, so I can't elaborate any futher.

For persisting messages you can look at the quickstarts and at http://community.jboss.org/wiki/CombiningESBJPAHibernateJTAandSpring#cf how to persist with Hibernate and JPA. For persisting queued messages you can easily change the messaging datasource from HSQL to a database of your preference. Be advised you do not use inVM parameter in your queue definition because then your messages won't be persisted.

Regards,

Hans

[okotenn]

Hi hans

Thank you for your  valuable advice.

I have find out that it is InVM parameter which makes the message lost,and I think it helpful about the URL you gave to me.

Thanks again