6 Replies Latest reply on Apr 6, 2006 2:09 AM by taggat

    out of memory error loading large queue

      I am doing a test of jbossMQ to see if it will be suitable for use in a messaging platform.

      When i load a queue up with over 100k messages and stop the jboss server (simulated server death) I restart the server and jboss gets an out of memory error loading up the queue. (The queue is persisted to postgres)

      Is there a way of making the server not load all of the records into memory on startup.

      My server will no longer load the queues with lots of messages in them.

      Also, the next test that i will need to test is having 400 queue receivers running, would this be a problem in jbossMQ, or is there a limit to the number of receivers i can have. (openJms failed at 100)

      thanks.

        • 1. Re: out of memory error loading large queue
          starksm64
          • 2. Re: out of memory error loading large queue

            Scott, thanks for the links, I went through them and made the changes, however the documented settings don't seem to work on the build that i have.

            I am using jboss 4.0.3SP1

            my configuration file for postgres is

            <?xml version="1.0" encoding="UTF-8"?>
            <!-- $Id: postgres-jdbc2-service.xml,v 1.8.2.3 2005/08/13 20:47:52 adrian Exp $ -->
            <server>
             <mbean code="org.jboss.mq.server.jmx.DestinationManager" name="jboss.mq:service=DestinationManager">
             <depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache</depends>
             <depends optional-attribute-name="PersistenceManager">jboss.mq:service=PersistenceManager</depends>
             <depends optional-attribute-name="StateManager">jboss.mq:service=StateManager</depends>
             </mbean>
             <mbean code="org.jboss.mq.server.MessageCache"
             name="jboss.mq:service=MessageCache">
             <attribute name="HighMemoryMark">50</attribute>
             <attribute name="MaxMemoryMark">60</attribute>
             <attribute name="CacheStore">jboss.mq:service=PersistenceManager</attribute>
             </mbean>
             <mbean code="org.jboss.mq.pm.jdbc2.PersistenceManager"
             name="jboss.mq:service=PersistenceManager">
             <depends optional-attribute-name="ConnectionManager">jboss.jca:service=DataSourceBinding,name=PostgresDS</depends>
             <attribute name="SqlProperties">
             BLOB_TYPE=BYTES_BLOB
             INSERT_TX = INSERT INTO JMS_TRANSACTIONS (TXID) values(?)
             INSERT_MESSAGE = INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
             SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM JMS_TRANSACTIONS
             SELECT_MAX_TX = SELECT MAX(TXID) FROM (SELECT MAX(TXID) AS TXID FROM JMS_TRANSACTIONS UNION SELECT MAX(TXID) AS TXID FROM JMS_MESSAGES)
             DELETE_ALL_TX = DELETE FROM JMS_TRANSACTIONS
             SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE DESTINATION=?
             SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
             MARK_MESSAGE = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
             UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
             UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
             UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
             UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
             UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
             UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
             DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTIONS) AND TXOP=?
             DELETE_TX = DELETE FROM JMS_TRANSACTIONS WHERE TXID = ?
             DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXID=? AND TXOP=?
             DELETE_TEMPORARY_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXOP='T'
             DELETE_MESSAGE = DELETE FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
             CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGES (MESSAGEID INTEGER NOT NULL, DESTINATION VARCHAR(150) NOT NULL, TXID INTEGER, TXOP CHAR(1), MESSAGEBLOB BYTEA, PRIMARY KEY (MESSAGEID, DESTINATION))
             CREATE_IDX_MESSAGE_TXOP_TXID = CREATE INDEX JMS_MESSAGES_TXOP_TXID ON JMS_MESSAGES (TXOP, TXID)
             CREATE_IDX_MESSAGE_DESTINATION = CREATE INDEX JMS_MESSAGES_DESTINATION ON JMS_MESSAGES (DESTINATION)
             CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTIONS ( TXID INTEGER, PRIMARY KEY (TXID) )
             CREATE_TABLES_ON_STARTUP = TRUE
             </attribute>
             <attribute name="RecoverMessagesChunk">1</attribute>
             </mbean>
            
            </server>
            


            when i run with this version, of which the
            <attribute name="RecoverMessagesChunk">1</attribute>
            is the only new line from the configuration, i get the following error on startup.

            2006-02-27 07:20:21,439 DEBUG [org.jboss.system.ServiceCreator] Created bean: jboss.mq:service=MessageCache
            2006-02-27 07:20:21,439 DEBUG [org.jboss.system.ServiceConfigurator] HighMemoryMark set to 50 in jboss.mq:service=MessageCache
            2006-02-27 07:20:21,481 DEBUG [org.jboss.system.ServiceConfigurator] MaxMemoryMark set to 60 in jboss.mq:service=MessageCache
            2006-02-27 07:20:21,483 DEBUG [org.jboss.system.ServiceConfigurator] CacheStore set to jboss.mq:service=PersistenceManager in jboss.mq:service=MessageCache
            2006-02-27 07:20:21,484 DEBUG [org.jboss.system.ServiceCreator] About to create bean: jboss.mq:service=PersistenceManager with code: org.jboss.mq.pm.jdbc2.PersistenceManager
            2006-02-27 07:20:21,535 DEBUG [org.jboss.system.ServiceCreator] Created bean: jboss.mq:service=PersistenceManager
            2006-02-27 07:20:21,535 DEBUG [org.jboss.system.ServiceController] recording that jboss.mq:service=PersistenceManager depends on jboss.jca:service=DataSourceBinding,name=PostgresDS
            2006-02-27 07:20:21,535 DEBUG [org.jboss.system.ServiceConfigurator] considering ConnectionManager with object name jboss.jca:service=DataSourceBinding,name=PostgresDS
            2006-02-27 07:20:21,536 DEBUG [org.jboss.system.ServiceConfigurator] SqlProperties set to BLOB_TYPE=BYTES_BLOB
             INSERT_TX = INSERT INTO JMS_TRANSACTIONS (TXID) values(?)
             INSERT_MESSAGE = INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
             SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM JMS_TRANSACTIONS
             SELECT_MAX_TX = SELECT MAX(TXID) FROM (SELECT MAX(TXID) AS TXID FROM JMS_TRANSACTIONS UNION SELECT MAX(TXID) AS TXID FROM JMS_MESSAGES)
             DELETE_ALL_TX = DELETE FROM JMS_TRANSACTIONS
             SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE DESTINATION=?
             SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
             MARK_MESSAGE = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
             UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
             UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
             UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
             UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
             UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
             UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
             DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTIONS) AND TXOP=?
             DELETE_TX = DELETE FROM JMS_TRANSACTIONS WHERE TXID = ?
             DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXID=? AND TXOP=?
             DELETE_TEMPORARY_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXOP='T'
             DELETE_MESSAGE = DELETE FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
             CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGES (MESSAGEID INTEGER NOT NULL, DESTINATION VARCHAR(150) NOT NULL, TXID INTEGER, TXOP CHAR(1), MESSAGEBLOB BYTEA, PRIMARY KEY (MESSAGEID, DESTINATION))
             CREATE_IDX_MESSAGE_TXOP_TXID = CREATE INDEX JMS_MESSAGES_TXOP_TXID ON JMS_MESSAGES (TXOP, TXID)
             CREATE_IDX_MESSAGE_DESTINATION = CREATE INDEX JMS_MESSAGES_DESTINATION ON JMS_MESSAGES (DESTINATION)
             CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTIONS ( TXID INTEGER, PRIMARY KEY (TXID) )
             CREATE_TABLES_ON_STARTUP = TRUE in jboss.mq:service=PersistenceManager
            2006-02-27 07:20:21,543 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.parentTraceEnabled=true
            2006-02-27 07:20:21,543 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.nestedTraceEnabled=false
            2006-02-27 07:20:21,543 DEBUG [org.jboss.util.NestedThrowable] org.jboss.util.NestedThrowable.detectDuplicateNesting=true
            2006-02-27 07:20:21,539 INFO [org.jboss.system.ServiceConfigurator] Problem configuring service jboss.mq:service=PersistenceManager
            org.jboss.deployment.DeploymentException: No Attribute found with name: RecoverMessagesChunk
             at org.jboss.system.ServiceConfigurator.configure(ServiceConfigurator.java:300)
             at org.jboss.system.ServiceConfigurator.internalInstall(ServiceConfigurator.java:442)
             at org.jboss.system.ServiceConfigurator.install(ServiceConfigurator.java:153)
             at org.jboss.system.ServiceController.install(ServiceController.java:215)
             at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:585)
             at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
             at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
             at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
             at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
             at $Proxy4.install(Unknown Source)
             at org.jboss.deployment.SARDeployer.create(SARDeployer.java:232)
             at org.jboss.deployment.MainDeployer.create(MainDeployer.java:935)
             at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:789)
             at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:753)
             at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:585)
             at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
             at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
             at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:118)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
             at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
             at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
             at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
             at $Proxy6.deploy(Unknown Source)
             at org.jboss.deployment.scanner.URLDeploymentScanner.deploy(URLDeploymentScanner.java:319)
             at org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:507)
             at org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:192)
             at org.jboss.deployment.scanner.AbstractDeploymentScanner.startService(AbstractDeploymentScanner.java:265)
             at org.jboss.system.ServiceMBeanSupport.jbossInternalStart(ServiceMBeanSupport.java:274)
             at org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:230)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:585)
             at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
             at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
             at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
             at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:943)
             at $Proxy0.start(Unknown Source)
             at org.jboss.system.ServiceController.start(ServiceController.java:428)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:585)
             at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
             at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
             at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
             at org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:943)
             at $Proxy0.start(Unknown Source)
             at org.jboss.system.ServiceController.start(ServiceController.java:428)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:585)
             at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
             at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:72)
             at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
             at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
             at $Proxy4.start(Unknown Source)
             at org.jboss.deployment.SARDeployer.start(SARDeployer.java:285)
             at org.jboss.deployment.MainDeployer.start(MainDeployer.java:989)
             at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:790)
             at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:753)
             at org.jboss.deployment.MainDeployer.deploy(MainDeployer.java:737)
             at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
             at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
             at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
             at java.lang.reflect.Method.invoke(Method.java:585)
             at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:141)
             at org.jboss.mx.server.Invocation.dispatch(Invocation.java:80)
             at org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:118)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
             at org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:127)
             at org.jboss.mx.server.Invocation.invoke(Invocation.java:74)
             at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:245)
             at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:644)
             at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:176)
             at $Proxy5.deploy(Unknown Source)
             at org.jboss.system.server.ServerImpl.doStart(ServerImpl.java:453)
             at org.jboss.system.server.ServerImpl.start(ServerImpl.java:330)
             at org.jboss.Main.boot(Main.java:187)
             at org.jboss.Main$1.run(Main.java:438)
             at java.lang.Thread.run(Thread.java:595)
            2006-02-27 07:20:21,548 DEBUG [org.jboss.system.ServiceController] Creating service jboss.mq:service=DestinationManager
            2006-02-27 07:20:21,548 DEBUG [org.jboss.system.ServiceController] waiting in create of jboss.mq:service=DestinationManager waiting on jboss.mq:service=MessageCache
            2006-02-27 07:20:21,548 DEBUG [org.jboss.system.ServiceController] Creating service jboss.mq:service=MessageCache
            2006-02-27 07:20:21,549 DEBUG [org.jboss.mq.server.MessageCache] Creating jboss.mq:service=MessageCache
            2006-02-27 07:20:21,549 DEBUG [org.jboss.mq.server.MessageCache] Created jboss.mq:service=MessageCache
            2006-02-27 07:20:21,729 DEBUG [org.jboss.system.ServiceController] Creating dependent components for: jboss.mq:service=MessageCache dependents are: [ObjectName: jboss.mq:service=DestinationManager
             State: CONFIGURED
             I Depend On:
             jboss.mq:service=MessageCache
             jboss.mq:service=PersistenceManager
             jboss.mq:service=StateManager
             Depends On Me:
             jboss.mq.destination:service=Queue,name=Q_IN_GEORGE
             jboss.mq.destination:service=Queue,name=Q_INBOX_GEORGE
             jboss.mq.destination:service=Queue,name=Q_OUT_GEORGE
             jboss.mq.destination:service=Queue,name=Q_ARCHIVE_GEORGE
             jboss.mq:service=SecurityManager
             jboss.mq.destination:service=Queue,name=DLQ
            ]
            


            and hence it won't start.

            I have the following amount of records in the tables
            jms_message_log 197137
            jms_reference_log 0 (not sure why this is 0)

            Because of the timelines i am under, I am haveing to get others in my team to investigate other jms providers as well. JMS is the core of message transmission on the system we are developing, and it will have a very high load (200-300 connections, 1million+ messages an hour). If you can give me any advise on how to get around these issues it would be much appreciated.

            Also, do you recommend that i change to jboss 4.0.4RC1, and where would i find details on the additional settings for 4.0.4 ?

            • 3. Re: out of memory error loading large queue
              starksm64

              As stated in the wiki page, RecoverMessagesChunk is only available in 4.0.4/3.2.8 and higher. 4.0.4RC1 also has the change.

              • 4. Re: out of memory error loading large queue

                thanks for that, I will try that out.

                • 5. Re: out of memory error loading large queue
                  jaink

                  Hi,

                  I was curiously to find out if upgrade to newer version worked for you. I have just looking into Jboss MQ and will appreciate it you could reply.

                  Thanks.

                  • 6. Re: out of memory error loading large queue

                    unfortunatly not, I tried alot of messageing services to try and get the load levels that we wanted, and i couldn't get jboss to handle the loads correctly.

                    Our load levels are in the order of

                    300 Queue Receivers
                    150 Queue Writers
                    400 messages a second read and write.

                    I know jboss has now fully released jboss messaging now, but unfortunatly we could not wait for a stable release to continue our development cycle and have had to go for a commercial product as all the open source jms servers could not handle the concurrent load and restart with 1 million messages enqueued.