journal

[ataylor]

fyi, I've intermittently been seeing this exception:

java.lang.NullPointerException
 at org.jboss.messaging.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:662)
 at org.jboss.messaging.core.server.impl.MessagingServerImpl.start(MessagingServerImpl.java:327)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)
 at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)
 at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:241)
 at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:109)
 at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:70)
 at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221)
 at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)
 at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)
 at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
 at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
 at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
 at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1598)
 at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1062)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:774)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)
 at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:121)
 at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:51)
 at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
 at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
 at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1178)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098)
 at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
 at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1598)
 at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1062)
 at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
 at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
 at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
 at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781)
 at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:545)
 at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:304)
 at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:205)
 at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:405)
 at org.jboss.Main.boot(Main.java:209)
 at org.jboss.Main$1.run(Main.java:547)
 at java.lang.Thread.run(Thread.java:619)

[timfox]

(11:02:21) AndyTaylor: jbossfox: it happened a couple of times when i ctrl-c the AS and restarted
(11:02:26) AndyTaylor: jbossfox: ok
(11:02:33) AndyTaylor: jbossfox: its the line
(11:02:42) AndyTaylor: jbossfox: MessageReference ref = queue.reroute(record.message, null);
(11:03:16) jbossfox: AndyTaylor: ok so queue is null

[timfox]

My guess is that since we're not syncing after adding a binding, then if the server crashes the binding might not be committed, but the messages was synced, so on restart the queue won't be found.

Solution: we need to sync after adding or deleting bindings.

We should add explicit add/update/delete record methods which take a sync param.

[clebert.suconic]

One thing that I realized (after our meeting today):

I believe this will be something else. Maybe messages being persisted for NonPersistentQueues? (as we discussed during the meeting)... and we need to investigate that possibility what would be a bug.



The bindingsJournal already syncs on every nonTransactional Add/update/delete:

bindingsJournal = new JournalImpl(1024 * 1024, 2, true, true, bindingsFF, "jbm-bindings", "bindings", 1, -1);

this is true for synNonTransactional, and true for syncTransactional. (we just sync every call on the BindingsJournal).

[clebert.suconic]

You are running tests that are deploying and undeploying queues.. right?

Maybe the queue was undeployed right before shutdown with messages still sitting on the journal, and Maybe you Ctrl-C while the deletes were being made.

When the server was restart, maybe there was messages still on the journal, while the queue was already gone... and the server wouldn't start because of that.

If this is the case we need to ignore these messages (providing warnings on the log). or maybe provide deletes in a two way on the Bindings Journal. An update on mark-to-delete followed by another delete when the messages were gone.

[timfox]

Not sure what you mean by "undeploy".

Deleting a queue, deletes the messages in the queue too.

[clebert.suconic]

"timfox" wrote:
deletes the messages in the queue too.

Yes, that's the point.


What if the server is crashed/stopped *while* the messages are being deleted. How restart would behave?

[timfox]

Ok so would need to swap the following lines of code:

storageManager.deleteQueueBinding(queue.getPersistenceID());

queue.deleteAllReferences();

[clebert.suconic]

What about do something like:

storageManager.prepareDeleteQueueBinding(queue.getPersistenceID());
queue.deleteAllReferences();
storageManager.deleteQueueBinding(queue.getPersistenceID());

prepareDeleteQueueBinding would add an updateRecord on the BindingsJournal.

When restarting, we would get PrepareDeleteBinding record and resume deleting records.


That could be done after Beta. (we would add a JIRA). For now we would be good at swapping the lines.

[clebert.suconic]

That could be done after Beta. (we would add a JIRA). For now we would be good at swapping the lines.

Or.. maybe we "just do it" (C)


It doesn't seem very difficult to implement it.

[timfox]

I don't see there's any need to add a prepare record or attempt to do some kind of 2PC between the journals, swapping the lines should do fine.

[clebert.suconic]

swapping the lines should do fine.

yes, but the *only* problem I see is the Queue not being deleted, unless someone called delete again.

Say...

- the user undeploy the queue, what will cause the delete on Journal.
- Server is restarted.
- The queue will still be there, with some messages.
- Later the user will redeploy the queue.
- An exception will probably happen.. and the user will wonder why the queue wasn't deleted.

[timfox]

I don't think that is a big deal.

If the queue is still there they can call delete again. The important thing is there is no error on startup.

[clebert.suconic]

"timfox" wrote:
I don't think that is a big deal.

if that's not a problem, then ok...

I have already done the switch.


Andy: let me know if you still see this problem (after an update)

I have also added a JIRA for creating a test for this:

https://jira.jboss.org/jira/browse/JBMESSAGING-1519