Management removing all messages on shutdown

[clebert.suconic]

I just found a bug recently introduced. Looking at the SVN Logs it was introduced by the recent work on management.

If you cleanly shutdown the server, Shutdown hooks are undeploying the queues, and all the messages are being deleted on a regular shutdown.

I have locally changed the journal to throw an exception if delete was called, and this is the place it is happening:

java.lang.Exception: trace
 at org.jboss.messaging.core.journal.impl.JournalImpl.appendDeleteRecordTransactional(JournalImpl.java:611)
 at org.jboss.messaging.core.persistence.impl.journal.JournalStorageManager.storeDeleteTransactional(JournalStorageManager.java:231)
 at org.jboss.messaging.core.transaction.impl.TransactionImpl.addAcknowledgement(TransactionImpl.java:149)
 at org.jboss.messaging.core.server.impl.QueueImpl.deleteAllReferences(QueueImpl.java:481)
 at org.jboss.messaging.core.management.impl.MessagingServerControl.destroyQueue(MessagingServerControl.java:362)
 at org.jboss.messaging.jms.server.impl.JMSServerManagerImpl.destroyQueue(JMSServerManagerImpl.java:164)
 at org.jboss.messaging.jms.server.impl.JMSServerDeployer.undeploy(JMSServerDeployer.java:221)
 at org.jboss.messaging.core.deployers.impl.XmlDeployer.stop(XmlDeployer.java:230)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:585)
 at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:56)
 at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:110)
 at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:97)
 at org.jboss.kernel.plugins.dependency.LifecycleAction.uninstallActionInternal(LifecycleAction.java:249)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.uninstallAction(KernelControllerContextAction.java:169)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.uninstallAction(KernelControllerContextAction.java:47)
 at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleUninstallAction(SimpleControllerContextAction.java:79)
 at org.jboss.dependency.plugins.action.AccessControllerContextAction.uninstall(AccessControllerContextAction.java:131)
 at org.jboss.dependency.plugins.AbstractControllerContextActions.uninstall(AbstractControllerContextActions.java:58)
 at org.jboss.dependency.plugins.AbstractControllerContext.uninstall(AbstractControllerContext.java:333)
 at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:1323)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:1009)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:986)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:986)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:986)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:986)
 at org.jboss.dependency.plugins.AbstractController.uninstallContext(AbstractController.java:936)
 at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:463)
 at org.jboss.dependency.plugins.AbstractController.uninstall(AbstractController.java:426)
 at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeployBean(AbstractKernelDeployer.java:384)
 at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeployBeans(AbstractKernelDeployer.java:363)
 at org.jboss.kernel.plugins.deployment.AbstractKernelDeployer.undeploy(AbstractKernelDeployer.java:162)
 at org.jboss.messaging.microcontainer.JBMBootstrapServer.undeploy(JBMBootstrapServer.java:127)
 at org.jboss.messaging.microcontainer.JBMBootstrapServer.shutDown(JBMBootstrapServer.java:199)
 at org.jboss.messaging.microcontainer.JBMBootstrapServer$Shutdown.run(JBMBootstrapServer.java:218)

We need to distinguish a real drop queue to a shutdown event. On the latest you're not supposed to delete messages.

[clebert.suconic]

https://jira.jboss.org/jira/browse/JBMESSAGING-1413

[clebert.suconic]

We just need the shutdown hook on JBMBootstrapServer somehow stopping deployments instead of undeploying them. That would give us the differentiation we need:

 public void shutDown()
 {
 log.info("Shutting down");
 ListIterator<KernelDeployment> iterator = deployments.listIterator(deployments.size());
 while (iterator.hasPrevious())
 {
 KernelDeployment deployment = (KernelDeployment) iterator.previous();
 try {undeploy(deployment);} catch (Throwable ignored){}
 }
 }

[timfox]

Just don't delete the queue!

[timfox]

I don't think you need to change anything related to the shutdown hook, it's just a simple bug in JMSServerDeployer.

The undeploy method should just be removing the objects from jndi - not deleting them!

[clebert.suconic]

Problem is,

If the user is really undeploying a Queue, the messages should go away, while if you just shutdown the server that doesn't mean undeploying. (just stop)

(Especially with the journal schema, where reclaiming would break.)

[clebert.suconic]

My suggestion was a simple change on the shutdown hook:

 public void shutDown()
 {
 log.info("Shutting down");
 ListIterator<KernelDeployment> iterator = deployments.listIterator(deployments.size());
 while (iterator.hasPrevious())
 {
 KernelDeployment deployment = (KernelDeployment) iterator.previous();
 someHowInvokeStop(deployment);
 }
 }

And besides that, we should also change the method stop on JMSServerDeployer, as that is calling undeploy.

I just don't want to take the risk of messages hanging on the journal when you don't have a queue any more.

[timfox]

Problem is,

If the user is really undeploying a Queue, the messages should go away,

No, deploying just means removing from JNDI, it does not mean deleting messages!

[clebert.suconic]

"timfox" wrote:

Problem is,

If the user is really undeploying a Queue, the messages should go away,

No, deploying just means removing from JNDI, it does not mean deleting messages!

Ok.. what you do when the user really undeploy the queue then? You just leave the messages on the journal?

(Well... yes is not really an option here!)

[timfox]

Ok.. what you do when the user really undeploy the queue then? You just leave the messages on the journal?

(Well... yes is not really an option here!)

The answer is yes. Why is yes not an option? That's the way we did in 1.x and that's the way we do it in 2.0.

Deploy means "I do not want the queue in jndi" that's all. It *does not* and never has meant "delete queue". If you want to delete the queue and all it's messages you use the delete queue method.

It's pretty straightforward.

[clebert.suconic]

Anyway... Whatever you do... the requirement I see on Journal is:


I - If the user removes the queue
-> Message must be removed, or else you would break reclaiming, and it would be impossible reloading the server

II - If the user shutdown the server
-> Messages should be kept on the journal


As long as we respect these requirements this could be implemented anyway you want. My suggestion was just what I saw on the code, and what would be easier implementing.

[clebert.suconic]

The answer is yes. Why is yes not an option? That's the way we did in 1.x and that's the way we do it in 2.0.

As I said on the first post: Reclaiming would break... The linked list effect. That was an option on Databases, and that's not an option on the circular journal file system.

[clebert.suconic]

If you want to delete the queue and all it's messages you use the delete queue method.

As long as you ensure this is done when the Queue is deleted.. that's fine.


My understanding is.. the user has the option of remove/delete the queue by simply removing the XML.

If that's not the case.. than it's fine.. but I panic with the possibility of the user removing a queue with pending messages and leaving them stuck forever on the journal.

[timfox]

As I said on the first post: Reclaiming would break... The linked list effect. That was an option on Databases, and that's not an option on the circular journal file system.

Why would reclaiming break? It shouldn't make any difference to reclaiming.

Can you explain yourself a bit more?

[clebert.suconic]

Just to conclude the thread... and as we talked today over #jbossmessaging@freenode, I thought undeploy could be translated as deleting the queue to the user, and I thought that would end up with Messages stuck on the journal.


As this is not the case this was just my misunderstanding.