9 Replies Latest reply on Sep 22, 2003 8:50 AM by Adrian Brock

    Problems with Messages after shutdown on Jboss 3.2.2RC3

    Christian Neuroth Newbie

      Hi!

      I am currently testing JMS on JBoss and I am facing the following scenario:

      When I send messages to a queue and try to receive them - there is no problem.

      If I shutdown the server and start it again, there are already some error messages during the boot process (attached).

      Furthermore, the MQ services don't start (Queue not bound...).

      There were 5000 messages in the queue and I am using the standard Hypersonic database.

      Why does the server not restart with the messages kept in the queue?

      Thanks

      Christian


      2003-09-22 10:52:39,602 ERROR [org.jboss.mq.pm.jdbc2.PersistenceManager] Starting failed
      org.jboss.mq.SpyJMSException: Could not resolve uncommited transactions. Message recovery may not be accurate; - nested throwable: (org.jboss.util.NestedSQLException: Could not create connection; - nested throwable: (java.sql.SQLException: Connection is broken: Connection refused); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Connection is broken: Connection refused)))
      at org.jboss.mq.pm.jdbc2.PersistenceManager.resolveAllUncommitedTXs(PersistenceManager.java:295)
      at org.jboss.mq.pm.jdbc2.PersistenceManager.startService(PersistenceManager.java:1315)
      at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:192)
      at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
      at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:976)
      at $Proxy12.start(Unknown Source)
      at org.jboss.system.ServiceController.start(ServiceController.java:394)
      at org.jboss.system.ServiceController.start(ServiceController.java:411)
      at org.jboss.system.ServiceController.start(ServiceController.java:411)
      at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)
      at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
      at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
      at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
      at $Proxy4.start(Unknown Source)
      at org.jboss.deployment.SARDeployer.start(SARDeployer.java:226)
      at org.jboss.deployment.MainDeployer.start(MainDeployer.java:824)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:632)
      at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:605)
      at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:324)

      ....

      10:52:45,909 ERROR [URLDeploymentScanner] MBeanException: Exception in MBean operation 'checkIncompleteDeployments()'
      Cause: Incomplete Deployment listing:
      Packages waiting for a deployer:

      Incompletely deployed packages:

      MBeans waiting for classes:

      MBeans waiting for other MBeans:
      [ObjectName: jboss.mq:service=InvocationLayer,type=HTTP
      state: CREATED
      I Depend On: jboss.mq:service=Invoker
      jboss.web:service=WebServer

      Depends On Me: , ObjectName: jboss.mq:service=InvocationLayer,type=JVM
      state: CREATED
      I Depend On: jboss.mq:service=Invoker

      Depends On Me: , ObjectName: jboss.mq:service=InvocationLayer,type=UIL2
      state: CREATED
      I Depend On: jboss.mq:service=Invoker

      Depends On Me: , ObjectName: jboss.mq:service=DestinationManager
      state: CREATED
      I Depend On: jboss.mq:service=MessageCache
      jboss.mq:service=PersistenceManager
      jboss.mq:service=StateManager

      Depends On Me: jboss.mq.destination:service=Topic,name=testTopic
      jboss.mq.destination:service=Topic,name=securedTopic
      jboss.mq.destination:service=Topic,name=testDurableTopic
      jboss.mq.destination:service=Queue,name=testQueue
      jboss.mq.destination:service=Queue,name=A
      jboss.mq.destination:service=Queue,name=B
      jboss.mq.destination:service=Queue,name=C
      jboss.mq.destination:service=Queue,name=D
      jboss.mq.destination:service=Queue,name=ex
      jboss.mq:service=SecurityManager
      jboss.mq.destination:service=Queue,name=DLQ
      , ObjectName: jboss.mq:service=PersistenceManager
      state: FAILED
      I Depend On: jboss.jca:service=LocalTxCM,name=DefaultDS

      Depends On Me: jboss.mq:service=DestinationManager
      org.jboss.mq.SpyJMSException: Could not resolve uncommited transactions. Message recovery may not be accurate; - nested throwable: (org.jboss.util.NestedSQLException: Could not create connection; - nested throwable: (java.sql.SQLException: Connection is broken: Connection refused); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Connection is broken: Connection refused))), ObjectName: jboss.mq.destination:service=Topic,name=testTopic
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager
      jboss.mq:service=SecurityManager

      Depends On Me: , ObjectName: jboss.mq.destination:service=Topic,name=securedTopic
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager
      jboss.mq:service=SecurityManager

      Depends On Me: , ObjectName: jboss.mq.destination:service=Topic,name=testDurableTopic
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager
      jboss.mq:service=SecurityManager

      Depends On Me: , ObjectName: jboss.mq.destination:service=Queue,name=testQueue
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager
      jboss.mq:service=SecurityManager

      Depends On Me: , ObjectName: jboss.mq.destination:service=Queue,name=A
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager

      Depends On Me: , ObjectName: jboss.mq.destination:service=Queue,name=B
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager

      Depends On Me: , ObjectName: jboss.mq.destination:service=Queue,name=C
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager

      Depends On Me: , ObjectName: jboss.mq.destination:service=Queue,name=D
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager

      Depends On Me: , ObjectName: jboss.mq.destination:service=Queue,name=ex
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager

      Depends On Me: , ObjectName: jboss.mq:service=Invoker
      state: CREATED
      I Depend On: jboss.mq:service=TracingInterceptor

      Depends On Me: jboss.mq:service=InvocationLayer,type=HTTP
      jboss.mq:service=InvocationLayer,type=JVM
      jboss.mq:service=InvocationLayer,type=UIL2
      jboss.mq:service=InvocationLayer,type=OIL2
      jboss.mq:service=InvocationLayer,type=OIL
      jboss.mq:service=InvocationLayer,type=RMI
      jboss.mq:service=InvocationLayer,type=UIL
      , ObjectName: jboss.mq:service=TracingInterceptor
      state: CREATED
      I Depend On: jboss.mq:service=SecurityManager

      Depends On Me: jboss.mq:service=Invoker
      , ObjectName: jboss.mq:service=SecurityManager
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager

      Depends On Me: jboss.mq.destination:service=Topic,name=testTopic
      jboss.mq.destination:service=Topic,name=securedTopic
      jboss.mq.destination:service=Topic,name=testDurableTopic
      jboss.mq.destination:service=Queue,name=testQueue
      jboss.mq:service=TracingInterceptor
      jboss.mq.destination:service=Queue,name=DLQ
      , ObjectName: jboss.mq.destination:service=Queue,name=DLQ
      state: CREATED
      I Depend On: jboss.mq:service=DestinationManager
      jboss.mq:service=SecurityManager

      Depends On Me: , ObjectName: jboss.mq:service=InvocationLayer,type=OIL2
      state: CREATED
      I Depend On: jboss.mq:service=Invoker

      Depends On Me: , ObjectName: jboss.mq:service=InvocationLayer,type=OIL
      state: CREATED
      I Depend On: jboss.mq:service=Invoker

      Depends On Me: , ObjectName: jboss.mq:service=InvocationLayer,type=RMI
      state: CREATED
      I Depend On: jboss.mq:service=Invoker

      Depends On Me: , ObjectName: jboss.mq:service=InvocationLayer,type=UIL
      state: CREATED
      I Depend On: jboss.mq:service=Invoker

      Depends On Me: ]
      10:52:45,913 INFO [URLDeploymentScanner] Started
      10:52:45,985 INFO [MainDeployer] Deployed package: file:/home/jboss/jboss-3.2.2RC3/server/default/conf/jboss-service.xml
      10:52:45,988 INFO [Server] JBoss (MX MicroKernel) [3.2.2RC3 (build: CVSTag=JBoss_3_2_2_RC3 date=200308241317)] Started in 34s:205ms

        • 1. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
          Stephane Nicoll Master

          Your are probably using a transactional JMS session without calling commit on the session.

          Regards,

          Stephane

          • 2. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
            Christian Neuroth Newbie

            Hi!

            No, the send of the messages is commited... I found another exception that occurs during the startup of the server.

            Probably, I missed to explain the sequence of the events:

            1.) I sent 5000 messages to the JMS queue
            2.) Shutdown the server
            3.) Restart...

            During the restart, the problems occur.

            Is my problem probably connected to this topic created my Rod Macpherson?

            http://www.mail-archive.com/jboss-user@lists.sourceforge.net/msg32107.html//www.mail-archive.com/jboss-user@lists.sourceforge.net/msg32107.html

            Thanks

            Christian


            org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (java.sql.SQLException: Connection is broken: Connection refused)
            at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:153)
            at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnection(InternalManagedConnectionPool.java:372)
            at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:165)
            at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:445)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:426)
            at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:331)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:509)
            at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:839)
            at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:102)
            at org.jboss.mq.pm.jdbc2.PersistenceManager.getConnection(PersistenceManager.java:1225)
            at org.jboss.mq.pm.jdbc2.PersistenceManager.resolveAllUncommitedTXs(PersistenceManager.java:214)
            at org.jboss.mq.pm.jdbc2.PersistenceManager.startService(PersistenceManager.java:1315)
            at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:192)
            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
            at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:976)
            at $Proxy12.start(Unknown Source)
            at org.jboss.system.ServiceController.start(ServiceController.java:394)
            at org.jboss.system.ServiceController.start(ServiceController.java:411)
            at org.jboss.system.ServiceController.start(ServiceController.java:411)
            at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
            at $Proxy4.start(Unknown Source)
            at org.jboss.deployment.SARDeployer.start(SARDeployer.java:226)
            at org.jboss.deployment.MainDeployer.start(MainDeployer.java:824)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:632)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:605)
            at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
            at $Proxy6.deploy(Unknown Source)
            at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:302)
            at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:476)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:201)
            at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:274)
            at org.jboss.system.ServiceMBeanSupport.start(ServiceMBeanSupport.java:192)
            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
            at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:976)
            at $Proxy0.start(Unknown Source)
            at org.jboss.system.ServiceController.start(ServiceController.java:394)
            at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
            at $Proxy4.start(Unknown Source)
            at org.jboss.deployment.SARDeployer.start(SARDeployer.java:226)
            at org.jboss.deployment.MainDeployer.start(MainDeployer.java:824)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:632)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:605)
            at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:589)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:324)
            at org.jboss.mx.capability.ReflectedMBeanDispatcher.invoke(ReflectedMBeanDispatcher.java:284)
            at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:546)
            at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:177)
            at $Proxy5.deploy(Unknown Source)
            at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:384)
            at org.jboss.system.server.ServerImpl.start(ServerImpl.java:291)
            at org.jboss.Main.boot(Main.java:150)
            at org.jboss.Main$1.run(Main.java:388)
            at java.lang.Thread.run(Thread.java:534)
            Caused by: java.sql.SQLException: Connection is broken: Connection refused
            at org.hsqldb.Trace.getError(Unknown Source)
            at org.hsqldb.Trace.error(Unknown Source)
            at org.hsqldb.jdbcConnection.reconnectHSQL(Unknown Source)
            at org.hsqldb.jdbcConnection.openHSQL(Unknown Source)
            at org.hsqldb.jdbcConnection.(Unknown Source)
            at org.hsqldb.jdbcDriver.connect(Unknown Source)
            at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:143)

            • 3. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
              Stephane Nicoll Master

              The 'Queue not bound' stuff is because JMS failed to start. Try creatuing a non transactionnal JMS session and remove the calls to commit / rollback to see if you still have the problem

              (just to check whether it's coming from your transactionnal session or not). What connection factory are you using?

              Regards,

              Stephane

              • 4. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
                Christian Neuroth Newbie

                Hi!

                Thanks for the help, Lord! ;-)

                No, even without a transaction - MQ is not started, the same exceptions occur. :(

                The different is, that the default.script file (Hypersonic) grows up to 27 MB....

                Any ideas?

                Christian

                • 5. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
                  Stephane Nicoll Master

                  The 27Mo is your messages ;-) and HSQL seems to be broken :/

                  If you are testing stuff remove the data directory (rm -rf data) and restart JBoss (which will restart ok this time) Please note that removing the data directory will imply the lost of your messages!

                  Then relauch your tests whith

                  createSession(false, Session.AUTO_ACKNOWLEDGE)

                  Regards,

                  Stephane

                  • 6. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
                    Christian Neuroth Newbie

                    thank you for your help, stephane!

                    yes! that is obvious...

                    the problem is we really need a performance of 5000 messages sent in a single transaction.

                    i thought hypersonic would be able to support this...
                    :(

                    and when the server is started, and there are messages left in the queue... they should not be lost!

                    thanks

                    christian

                    • 7. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
                      Stephane Nicoll Master

                      It seems that your messages are not committed properly. I had the problem a few months ago (message lost on restart). If you are inside a J2EE component, you don't need to specify a JMS transactionnel session, the global transaction context will carry that for you.

                      Regards,

                      Stephane

                      • 8. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
                        Stephane Nicoll Master

                        Regarding my last post, I thing this it. Have a look to the stacktrace:

                        at org.jboss.mq.pm.jdbc2.PersistenceManager.resolveAllUncommitedTXs(PersistenceManager.java:295)

                        The method's name is quite explicit, don't you think?

                        Regards,

                        Stephane

                        • 9. Re: Problems with Messages after shutdown on Jboss 3.2.2RC3
                          Adrian Brock Master

                          jboss-3.2.2RC4 includes a change to the configuration
                          for hsqldb to use "create cached table"
                          This reverses the assumption (everything is flushed
                          to disk with a memory cache rather than memory first with
                          period savepoints).

                          You should seriously consider your use of hsqldb if
                          you can't afford to lose a message/downtime.
                          hsqldb is in the same rank as MS access. Ok as a desktop
                          database and testing but not an enterprise db.

                          Regards,
                          Adrian