13 Replies Latest reply on Feb 26, 2009 12:36 PM by Clebert Suconic

    journal

    Andy Taylor Master

      fyi, I've intermittently been seeing this exception:

      java.lang.NullPointerException
       at org.jboss.messaging.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:662)
       at org.jboss.messaging.core.server.impl.MessagingServerImpl.start(MessagingServerImpl.java:327)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at org.jboss.reflect.plugins.introspection.ReflectionUtils.invoke(ReflectionUtils.java:59)
       at org.jboss.reflect.plugins.introspection.ReflectMethodInfoImpl.invoke(ReflectMethodInfoImpl.java:150)
       at org.jboss.joinpoint.plugins.BasicMethodJoinPoint.dispatch(BasicMethodJoinPoint.java:66)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction$JoinpointDispatchWrapper.execute(KernelControllerContextAction.java:241)
       at org.jboss.kernel.plugins.dependency.ExecutionWrapper.execute(ExecutionWrapper.java:47)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchExecutionWrapper(KernelControllerContextAction.java:109)
       at org.jboss.kernel.plugins.dependency.KernelControllerContextAction.dispatchJoinPoint(KernelControllerContextAction.java:70)
       at org.jboss.kernel.plugins.dependency.LifecycleAction.installActionInternal(LifecycleAction.java:221)
       at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:54)
       at org.jboss.kernel.plugins.dependency.InstallsAwareAction.installAction(InstallsAwareAction.java:42)
       at org.jboss.dependency.plugins.action.SimpleControllerContextAction.simpleInstallAction(SimpleControllerContextAction.java:62)
       at org.jboss.dependency.plugins.action.AccessControllerContextAction.install(AccessControllerContextAction.java:71)
       at org.jboss.dependency.plugins.AbstractControllerContextActions.install(AbstractControllerContextActions.java:51)
       at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1598)
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1062)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:774)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:540)
       at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:121)
       at org.jboss.deployers.vfs.deployer.kernel.BeanMetaDataDeployer.deploy(BeanMetaDataDeployer.java:51)
       at org.jboss.deployers.spi.deployer.helpers.AbstractSimpleRealDeployer.internalDeploy(AbstractSimpleRealDeployer.java:62)
       at org.jboss.deployers.spi.deployer.helpers.AbstractRealDeployer.deploy(AbstractRealDeployer.java:50)
       at org.jboss.deployers.plugins.deployers.DeployerWrapper.deploy(DeployerWrapper.java:171)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doDeploy(DeployersImpl.java:1439)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1157)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.doInstallParentFirst(DeployersImpl.java:1178)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.install(DeployersImpl.java:1098)
       at org.jboss.dependency.plugins.AbstractControllerContext.install(AbstractControllerContext.java:348)
       at org.jboss.dependency.plugins.AbstractController.install(AbstractController.java:1598)
       at org.jboss.dependency.plugins.AbstractController.incrementState(AbstractController.java:934)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:1062)
       at org.jboss.dependency.plugins.AbstractController.resolveContexts(AbstractController.java:984)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:822)
       at org.jboss.dependency.plugins.AbstractController.change(AbstractController.java:553)
       at org.jboss.deployers.plugins.deployers.DeployersImpl.process(DeployersImpl.java:781)
       at org.jboss.deployers.plugins.main.MainDeployerImpl.process(MainDeployerImpl.java:545)
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.loadProfile(ProfileServiceBootstrap.java:304)
       at org.jboss.system.server.profileservice.ProfileServiceBootstrap.start(ProfileServiceBootstrap.java:205)
       at org.jboss.bootstrap.AbstractServerImpl.start(AbstractServerImpl.java:405)
       at org.jboss.Main.boot(Main.java:209)
       at org.jboss.Main$1.run(Main.java:547)
       at java.lang.Thread.run(Thread.java:619)
      


        • 1. Re: journal
          Tim Fox Master

           

          (11:02:21) AndyTaylor: jbossfox: it happened a couple of times when i ctrl-c the AS and restarted
          (11:02:26) AndyTaylor: jbossfox: ok
          (11:02:33) AndyTaylor: jbossfox: its the line
          (11:02:42) AndyTaylor: jbossfox: MessageReference ref = queue.reroute(record.message, null);
          (11:03:16) jbossfox: AndyTaylor: ok so queue is null
          


          • 2. Re: journal
            Tim Fox Master

            My guess is that since we're not syncing after adding a binding, then if the server crashes the binding might not be committed, but the messages was synced, so on restart the queue won't be found.

            Solution: we need to sync after adding or deleting bindings.

            We should add explicit add/update/delete record methods which take a sync param.

            • 3. Re: journal
              Clebert Suconic Master

              One thing that I realized (after our meeting today):

              I believe this will be something else. Maybe messages being persisted for NonPersistentQueues? (as we discussed during the meeting)... and we need to investigate that possibility what would be a bug.



              The bindingsJournal already syncs on every nonTransactional Add/update/delete:


              bindingsJournal = new JournalImpl(1024 * 1024, 2, true, true, bindingsFF, "jbm-bindings", "bindings", 1, -1);



              this is true for synNonTransactional, and true for syncTransactional. (we just sync every call on the BindingsJournal).

              • 4. Re: journal
                Clebert Suconic Master


                You are running tests that are deploying and undeploying queues.. right?

                Maybe the queue was undeployed right before shutdown with messages still sitting on the journal, and Maybe you Ctrl-C while the deletes were being made.

                When the server was restart, maybe there was messages still on the journal, while the queue was already gone... and the server wouldn't start because of that.

                If this is the case we need to ignore these messages (providing warnings on the log). or maybe provide deletes in a two way on the Bindings Journal. An update on mark-to-delete followed by another delete when the messages were gone.


                • 5. Re: journal
                  Tim Fox Master

                  Not sure what you mean by "undeploy".

                  Deleting a queue, deletes the messages in the queue too.

                  • 6. Re: journal
                    Clebert Suconic Master

                     

                    "timfox" wrote:
                    deletes the messages in the queue too.


                    Yes, that's the point.


                    What if the server is crashed/stopped *while* the messages are being deleted. How restart would behave?

                    • 7. Re: journal
                      Tim Fox Master

                      Ok so would need to swap the following lines of code:

                      storageManager.deleteQueueBinding(queue.getPersistenceID());
                      
                      queue.deleteAllReferences();
                      


                      • 8. Re: journal
                        Clebert Suconic Master

                        What about do something like:


                        storageManager.prepareDeleteQueueBinding(queue.getPersistenceID());
                        queue.deleteAllReferences();
                        storageManager.deleteQueueBinding(queue.getPersistenceID());
                        


                        prepareDeleteQueueBinding would add an updateRecord on the BindingsJournal.

                        When restarting, we would get PrepareDeleteBinding record and resume deleting records.


                        That could be done after Beta. (we would add a JIRA). For now we would be good at swapping the lines.

                        • 9. Re: journal
                          Clebert Suconic Master

                           

                          "clebert.suconic@jboss.com" wrote:

                          That could be done after Beta. (we would add a JIRA). For now we would be good at swapping the lines.


                          Or.. maybe we "just do it" (C)


                          It doesn't seem very difficult to implement it.

                          • 10. Re: journal
                            Tim Fox Master

                            I don't see there's any need to add a prepare record or attempt to do some kind of 2PC between the journals, swapping the lines should do fine.

                            • 11. Re: journal
                              Clebert Suconic Master

                               

                              swapping the lines should do fine.


                              yes, but the *only* problem I see is the Queue not being deleted, unless someone called delete again.

                              Say...

                              - the user undeploy the queue, what will cause the delete on Journal.
                              - Server is restarted.
                              - The queue will still be there, with some messages.
                              - Later the user will redeploy the queue.
                              - An exception will probably happen.. and the user will wonder why the queue wasn't deleted.

                              • 12. Re: journal
                                Tim Fox Master

                                I don't think that is a big deal.

                                If the queue is still there they can call delete again. The important thing is there is no error on startup.

                                • 13. Re: journal
                                  Clebert Suconic Master

                                   

                                  "timfox" wrote:
                                  I don't think that is a big deal.


                                  if that's not a problem, then ok...

                                  I have already done the switch.


                                  Andy: let me know if you still see this problem (after an update)

                                  I have also added a JIRA for creating a test for this:

                                  https://jira.jboss.org/jira/browse/JBMESSAGING-1519