6 Replies Latest reply on Apr 6, 2006 2:09 AM by George Taylor

    out of memory error loading large queue

    George Taylor Newbie

      I am doing a test of jbossMQ to see if it will be suitable for use in a messaging platform.

      When i load a queue up with over 100k messages and stop the jboss server (simulated server death) I restart the server and jboss gets an out of memory error loading up the queue. (The queue is persisted to postgres)

      Is there a way of making the server not load all of the records into memory on startup.

      My server will no longer load the queues with lots of messages in them.

      Also, the next test that i will need to test is having 400 queue receivers running, would this be a problem in jbossMQ, or is there a limit to the number of receivers i can have. (openJms failed at 100)

      thanks.

        • 2. Re: out of memory error loading large queue
          George Taylor Newbie

          Scott, thanks for the links, I went through them and made the changes, however the documented settings don't seem to work on the build that i have.

          I am using jboss 4.0.3SP1

          my configuration file for postgres is

          <?xml version="1.0" encoding="UTF-8"?>
          <!-- $Id: postgres-jdbc2-service.xml,v 1.8.2.3 2005/08/13 20:47:52 adrian Exp $ -->
          <server>
           <mbean code="org.jboss.mq.server.jmx.DestinationManager" name="jboss.mq:service=DestinationManager">
           <depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache</depends>
           <depends optional-attribute-name="PersistenceManager">jboss.mq:service=PersistenceManager</depends>
           <depends optional-attribute-name="StateManager">jboss.mq:service=StateManager</depends>
           </mbean>
           <mbean code="org.jboss.mq.server.MessageCache"
           name="jboss.mq:service=MessageCache">
           <attribute name="HighMemoryMark">50</attribute>
           <attribute name="MaxMemoryMark">60</attribute>
           <attribute name="CacheStore">jboss.mq:service=PersistenceManager</attribute>
           </mbean>
           <mbean code="org.jboss.mq.pm.jdbc2.PersistenceManager"
           name="jboss.mq:service=PersistenceManager">
           <depends optional-attribute-name="ConnectionManager">jboss.jca:service=DataSourceBinding,name=PostgresDS</depends>
           <attribute name="SqlProperties">
           BLOB_TYPE=BYTES_BLOB
           INSERT_TX = INSERT INTO JMS_TRANSACTIONS (TXID) values(?)
           INSERT_MESSAGE = INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
           SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM JMS_TRANSACTIONS
           SELECT_MAX_TX = SELECT MAX(TXID) FROM (SELECT MAX(TXID) AS TXID FROM JMS_TRANSACTIONS UNION SELECT MAX(TXID) AS TXID FROM JMS_MESSAGES)
           DELETE_ALL_TX = DELETE FROM JMS_TRANSACTIONS
           SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE DESTINATION=?
           SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
           MARK_MESSAGE = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
           UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
           UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
           UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
           UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
           UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
           UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
           DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTIONS) AND TXOP=?
           DELETE_TX = DELETE FROM JMS_TRANSACTIONS WHERE TXID = ?
           DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXID=? AND TXOP=?
           DELETE_TEMPORARY_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXOP='T'
           DELETE_MESSAGE = DELETE FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
           CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGES (MESSAGEID INTEGER NOT NULL, DESTINATION VARCHAR(150) NOT NULL, TXID INTEGER, TXOP CHAR(1), MESSAGEBLOB BYTEA, PRIMARY KEY (MESSAGEID, DESTINATION))
           CREATE_IDX_MESSAGE_TXOP_TXID = CREATE INDEX JMS_MESSAGES_TXOP_TXID ON JMS_MESSAGES (TXOP, TXID)
           CREATE_IDX_MESSAGE_DESTINATION = CREATE INDEX JMS_MESSAGES_DESTINATION ON JMS_MESSAGES (DESTINATION)
           CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTIONS ( TXID INTEGER, PRIMARY KEY (TXID) )
           CREATE_TABLES_ON_STARTUP = TRUE
           </attribute>
           <attribute name="RecoverMessagesChunk">1</attribute>
           </mbean>
          
          </server>
          


          when i run with this version, of which the
          <attribute name="RecoverMessagesChunk">1</attribute>
          is the only new line from the configuration, i get the following error on startup.

          2006-02-27 07:20:21,439 DEBUG [org.jboss.system.ServiceCreator] Created bean: jboss.mq:service=MessageCache
          2006-02-27 07:20:21,439 DEBUG [org.jboss.system.ServiceConfigurator] HighMemoryMark set to 50 in jboss.mq:service=MessageCache
          2006-02-27 07:20:21,481 DEBUG [org.jboss.system.ServiceConfigurator] MaxMemoryMark set to 60 in jboss.mq:service=MessageCache
          2006-02-27 07:20:21,483 DEBUG [org.jboss.system.ServiceConfigurator] CacheStore set to jboss.mq:service=PersistenceManager in jboss.mq:service=MessageCache
          2006-02-27 07:20:21,484 DEBUG [org.jboss.system.ServiceCreator] About to create bean: jboss.mq:service=PersistenceManager with code: org.jboss.mq.pm.jdbc2.PersistenceManager
          2006-02-27 07:20:21,535 DEBUG [org.jboss.system.ServiceCreator] Created bean: jboss.mq:service=PersistenceManager
          2006-02-27 07:20:21,535 DEBUG [org.jboss.system.ServiceController] recording that jboss.mq:service=PersistenceManager depends on jboss.jca:service=DataSourceBinding,name=PostgresDS
          2006-02-27 07:20:21,535 DEBUG [org.jboss.system.ServiceConfigurator] considering ConnectionManager with object name jboss.jca:service=DataSourceBinding,name=PostgresDS
          2006-02-27 07:20:21,536 DEBUG [org.jboss.system.ServiceConfigurator] SqlProperties set to BLOB_TYPE=BYTES_BLOB
           INSERT_TX = INSERT INTO JMS_TRANSACTIONS (TXID) values(?)
           INSERT_MESSAGE = INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
           SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM JMS_TRANSACTIONS
           SELECT_MAX_TX = SELECT MAX(TXID) FROM (SELECT MAX(TXID) AS TXID FROM JMS_TRANSACTIONS UNION SELECT MAX(TXID) AS TXID FROM JMS_MESSAGES)
           DELETE_ALL_TX = DELETE FROM JMS_TRANSACTIONS
           SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE DESTINATION=?
           SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
           MARK_MESSAGE = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
           UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
           UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
           UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
           UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
           UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
           UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
           DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTIONS) AND TXOP=?
           DELETE_TX = DELETE FROM JMS_TRANSACTIONS WHERE TXID = ?
           DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXID=? AND TXOP=?
           DELETE_TEMPORARY_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXOP='T'
           DELETE_MESSAGE = DELETE FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
           CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGES (MESSAGEID INTEGER NOT NULL, DESTINATION VARCHAR(150) NOT NULL, TXID INTEGER, TXOP CHAR(1), MESSAGEBLOB BYTEA, PRIMARY KEY (MESSAGEID, DESTINATION))
           CREATE_IDX_MESSAGE_TXOP_TXID = CREATE INDEX JMS_MESSAGES_TXOP_TXID ON JMS_MESSAGES (TXOP, TXID)
           CREATE_IDX_MESSAGE_DESTINATION = CREATE INDEX JMS_MESSAGES_DESTINATION ON JMS_MESSAGES (DESTINATION)
           CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTIONS ( TXID INTEGER, PRIMARY KEY (TXID) )
           CREATE_TABLES_ON_STARTUP = TRUE in jboss.mq:service=PersistenceManager
          2006-02-27 07:20:21,543 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.parentTraceEnabled=true
          2006-02-27 07:20:21,543 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.nestedTraceEnabled=false
          2006-02-27 07:20:21,543 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.detectDuplicateNesting=true
          2006-02-27 07:20:21,539 INFO [org.jboss.system.ServiceConfigurator] Problem configuring service jboss.mq:service=PersistenceManager
          org.jboss.deployment.DeploymentException: No Attribute found with name: RecoverMessagesChunk
           at org.jboss.system.ServiceConfigurator.configure(ServiceConfigurator.java:300)
           at org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:442)
           at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:153)
           at org.jboss.system.ServiceController.install(ServiceController.java:215)
           at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:585)
           at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
           at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
           at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
           at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
           at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
           at $Proxy4.install(Unknown Source)
           at org.jboss.deployment.SARDeployer.create(SARDeployer.java:232)
           at org.jboss.deployment.MainDeployer.create(MainDeployer.java:935)
           at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:789)
           at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:753)
           at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:585)
           at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
           at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
           at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:118)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
           at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
           at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
           at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
           at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
           at $Proxy6.deploy(Unknown Source)
           at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:319)
           at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:507)
           at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:192)
           at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:265)
           at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:274)
           at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:230)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:585)
           at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
           at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
           at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
           at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
           at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:943)
           at $Proxy0.start(Unknown Source)
           at org.jboss.system.ServiceController.start(ServiceController.java:428)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:585)
           at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
           at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
           at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
           at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
           at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:943)
           at $Proxy0.start(Unknown Source)
           at org.jboss.system.ServiceController.start(ServiceController.java:428)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:585)
           at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
           at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
           at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
           at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
           at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
           at $Proxy4.start(Unknown Source)
           at org.jboss.deployment.SARDeployer.start(SARDeployer.java:285)
           at org.jboss.deployment.MainDeployer.start(MainDeployer.java:989)
           at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:790)
           at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:753)
           at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:737)
           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
           at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
           at java.lang.reflect.Method.invoke(Method.java:585)
           at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
           at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
           at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:118)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
           at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
           at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
           at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
           at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
           at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
           at $Proxy5.deploy(Unknown Source)
           at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:453)
           at org.jboss.system.server.ServerImpl.start(ServerImpl.java:330)
           at org.jboss.Main.boot(Main.java:187)
           at org.jboss.Main$1.run(Main.java:438)
           at java.lang.Thread.run(Thread.java:595)
          2006-02-27 07:20:21,548 DEBUG [org.jboss.system.ServiceController] Creating service jboss.mq:service=DestinationManager
          2006-02-27 07:20:21,548 DEBUG [org.jboss.system.ServiceController] waiting in create of jboss.mq:service=DestinationManager waiting on jboss.mq:service=MessageCache
          2006-02-27 07:20:21,548 DEBUG [org.jboss.system.ServiceController] Creating service jboss.mq:service=MessageCache
          2006-02-27 07:20:21,549 DEBUG [org.jboss.mq.server.MessageCache] Creating jboss.mq:service=MessageCache
          2006-02-27 07:20:21,549 DEBUG [org.jboss.mq.server.MessageCache] Created jboss.mq:service=MessageCache
          2006-02-27 07:20:21,729 DEBUG [org.jboss.system.ServiceController] Creating dependent components for: jboss.mq:service=MessageCache dependents are: [ObjectName: jboss.mq:service=DestinationManager
           State: CONFIGURED
           I Depend On:
           jboss.mq:service=MessageCache
           jboss.mq:service=PersistenceManager
           jboss.mq:service=StateManager
           Depends On Me:
           jboss.mq.destination:service=Queue,name=Q_IN_GEORGE
           jboss.mq.destination:service=Queue,name=Q_INBOX_GEORGE
           jboss.mq.destination:service=Queue,name=Q_OUT_GEORGE
           jboss.mq.destination:service=Queue,name=Q_ARCHIVE_GEORGE
           jboss.mq:service=SecurityManager
           jboss.mq.destination:service=Queue,name=DLQ
          ]
          


          and hence it won't start.

          I have the following amount of records in the tables
          jms_message_log 197137
          jms_reference_log 0 (not sure why this is 0)

          Because of the timelines i am under, I am haveing to get others in my team to investigate other jms providers as well. JMS is the core of message transmission on the system we are developing, and it will have a very high load (200-300 connections, 1million+ messages an hour). If you can give me any advise on how to get around these issues it would be much appreciated.

          Also, do you recommend that i change to jboss 4.0.4RC1, and where would i find details on the additional settings for 4.0.4 ?

          • 3. Re: out of memory error loading large queue
            Scott Stark Master

            As stated in the wiki page, RecoverMessagesChunk is only available in 4.0.4/3.2.8 and higher. 4.0.4RC1 also has the change.

            • 4. Re: out of memory error loading large queue
              George Taylor Newbie

              thanks for that, I will try that out.

              • 5. Re: out of memory error loading large queue
                kokil jain Newbie

                Hi,

                I was curiously to find out if upgrade to newer version worked for you. I have just looking into Jboss MQ and will appreciate it you could reply.

                Thanks.

                • 6. Re: out of memory error loading large queue
                  George Taylor Newbie

                  unfortunatly not, I tried alot of messageing services to try and get the load levels that we wanted, and i couldn't get jboss to handle the loads correctly.

                  Our load levels are in the order of

                  300 Queue Receivers
                  150 Queue Writers
                  400 messages a second read and write.

                  I know jboss has now fully released jboss messaging now, but unfortunatly we could not wait for a stable release to continue our development cycle and have had to go for a commercial product as all the open source jms servers could not handle the concurrent load and restart with 1 million messages enqueued.