13 Replies Latest reply on Apr 11, 2004 10:30 AM by adrian.brock

    During startup server tries to load all messages in a JMS qu

    amresh

      We are using JBoss 3.2.3 with Sybase as the JMS Store

      If we restart JBoss with a significant number of messages (for eg above 15000) on any of the JMS queues. It takes very long time to start up.

      Logs suggest that the server is trying to load the JMS queue (one with a very high queue depth). At one point the the startup took a very long time and resulted in an out of memory exception. The only way we could get the server to start was by increasing the heap size from 256 to 512 (the other option was to delete/drain the queue).

      Is there configuration to configure the server to not load the entire queue during startup?

      Rgds,

      Amresh

        • 1. Re: During startup server tries to load all messages in a JM

          Thanks Stephane

          Any idea when is JBoss 3.2.4 expected to be rolled out?

          Also is this fix on Adrian's MUST HAVE list or NICE TO HAVE list.

          Rgds,

          Amresh

          • 2. Re: During startup server tries to load all messages in a JM
            amresh

            Thanks Stephane

            Any idea when is JBoss 3.2.4 expected to be rolled out?

            Also is this fix on Adrian's MUST HAVE list or NICE TO HAVE list.

            Rgds,

            Amresh

            • 3. Re: During startup server tries to load all messages in a JM

              It is not on my list for 3.2.4 (but I would like to have this feature).

              The fundamental problem is that the list of messages in JBossMQ's queue/subscriptions
              are entirely within memory (the messages are not).
              This is compounded by the fact that to construct those datastructures at a restart
              it must at reload all messages into memory (no matter how temporarily).

              Can you provide the example config for sybase?

              Regards,
              Adrian

              • 4. Re: During startup server tries to load all messages in a JM
                amresh

                Adrian thanks for your reply.

                It would be good to know your ideas on this. If possible I would like try adding this feature myself.

                When you say sybase config to you mean the sybase-jdbc2-service.xml. If yes here it is:



                <?xml version="1.0" encoding="UTF-8"?>
                
                <!-- $Id: sybase-jdbc2-service.xml,v 1.1.2.1 2003/12/10 18:15:59 DAmresh Exp $ -->
                
                <server>
                
                 <!-- ==================================================================== -->
                 <!-- Persistence and caching using Sybase -->
                 <!-- IMPORTANT: Remove hsqldb-jdbc2-service.xml -->
                 <!-- ==================================================================== -->
                
                 <!--
                 | The destination manager is the core service within JBossMQ
                 -->
                 <mbean code="org.jboss.mq.server.jmx.DestinationManager" name="jboss.mq:service=DestinationManager">
                 <depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache</depends>
                 <depends optional-attribute-name="PersistenceManager">jboss.mq:service=PersistenceManager</depends>
                 <depends optional-attribute-name="StateManager">jboss.mq:service=StateManager</depends>
                 </mbean>
                
                 <!--
                 | The MessageCache decides where to put JBossMQ message that
                 | are sitting around waiting to be consumed by a client.
                 |
                 | The memory marks are in Megabytes. Once the JVM memory usage hits
                 | the high memory mark, the old messages in the cache will start getting
                 | stored in the DataDirectory. As memory usage gets closer to the
                 | Max memory mark, the amount of message kept in the memory cache aproaches 0.
                 -->
                 <mbean code="org.jboss.mq.server.MessageCache"
                 name="jboss.mq:service=MessageCache">
                 <attribute name="HighMemoryMark">50</attribute>
                 <attribute name="MaxMemoryMark">60</attribute>
                 <attribute name="CacheStore">jboss.mq:service=PersistenceManager</attribute>
                 </mbean>
                
                 <!-- The PersistenceManager is used to store messages to disk. -->
                 <!--
                 | The jdbc2 PersistenceManager is the new improved JDBC implementation.
                 | This implementation allows you to control how messages are stored in
                 | the database.
                 |
                 | This jdbc2 PM configuration has been tested against Sybase
                
                 <mbean code="org.jboss.mq.pm.jdbc2.PersistenceManager"
                 name="jboss.mq:service=PersistenceManager">
                 <depends optional-attribute-name="ConnectionManager">jboss.jca:service=LocalTxCM,name=OracleDS</depends>
                 <attribute name="SqlProperties">
                 BLOB_TYPE=BINARYSTREAM_BLOB
                 INSERT_TX = INSERT INTO JMS_TRANSACTIONS (TXID) values(?)
                 INSERT_MESSAGE = INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
                 SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM JMS_TRANSACTIONS
                 SELECT_MAX_TX = SELECT MAX(TXID) FROM JMS_MESSAGES
                 SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE DESTINATION=?
                 SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
                 MARK_MESSAGE = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
                 UPDATE_MESSAGE = UPDATE JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
                 UPDATE_MARKED_MESSAGES = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
                 UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
                 DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM JMS_TRANSACTIONS) AND TXOP=?
                 DELETE_TX = DELETE FROM JMS_TRANSACTIONS WHERE TXID = ?
                 DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXID=? AND TXOP=?
                 DELETE_MESSAGE = DELETE FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
                 CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGES ( MESSAGEID INTEGER NOT NULL, \
                 DESTINATION VARCHAR(255) NOT NULL, TXID INTEGER, TXOP CHAR(1), \
                 MESSAGEBLOB BLOB, PRIMARY KEY (MESSAGEID, DESTINATION) )
                 CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTIONS ( TXID INTEGER )
                 CREATE_TABLES_ON_STARTUP = TRUE
                 </attribute>
                 </mbean>
                 -->
                
                 <!-- This is FME PersistenceManager, it is a copy of jdbc2 PersistenceManager with a changed start
                 up behaviour. Code for the class can be found in fme repository of CVS-->
                 <mbean code="com.aaa.bbb.jboss.mq.pm.jdbc2.PersistenceManager"
                 name="jboss.mq:service=PersistenceManager">
                 <depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache</depends>
                 <depends optional-attribute-name="ConnectionManager">jboss.jca:service=LocalTxCM,name=jdbc/JMSLocalTxDataSource</depends>
                 <attribute name="SqlProperties">
                 BLOB_TYPE=BYTES_BLOB
                 INSERT_TX = INSERT INTO dbo.JMS_TRANSACTIONS (TXID) values(?)
                 INSERT_MESSAGE = INSERT INTO dbo.JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
                 SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM dbo.JMS_TRANSACTIONS
                 SELECT_MAX_TX = SELECT MAX(TXID) FROM dbo.JMS_MESSAGES
                 SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM dbo.JMS_MESSAGES WHERE DESTINATION=?
                 SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM dbo.JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
                 MARK_MESSAGE = UPDATE dbo.JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
                 UPDATE_MESSAGE = UPDATE dbo.JMS_MESSAGES SET MESSAGEBLOB=? WHERE MESSAGEID=? AND DESTINATION=?
                 UPDATE_MARKED_MESSAGES = UPDATE dbo.JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=?
                 UPDATE_MARKED_MESSAGES_WITH_TX = UPDATE dbo.JMS_MESSAGES SET TXID=?, TXOP=? WHERE TXOP=? AND TXID=?
                 DELETE_MARKED_MESSAGES_WITH_TX = DELETE FROM dbo.JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM dbo.JMS_TRANSACTIONS) AND TXOP=?
                 DELETE_TX = DELETE FROM dbo.JMS_TRANSACTIONS WHERE TXID = ?
                 DELETE_MARKED_MESSAGES = DELETE FROM dbo.JMS_MESSAGES WHERE TXID=? AND TXOP=?
                 DELETE_MESSAGE = DELETE FROM dbo.JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
                 CREATE_MESSAGE_TABLE = CREATE TABLE dbo.JMS_MESSAGES ( MESSAGEID INTEGER NOT NULL, \
                 DESTINATION VARCHAR(255) NOT NULL, TXID INTEGER NULL, TXOP CHAR(1), \
                 MESSAGEBLOB IMAGE, PRIMARY KEY (MESSAGEID, DESTINATION) )
                 CREATE_TX_TABLE = CREATE TABLE dbo.JMS_TRANSACTIONS ( TXID INTEGER )
                 SELECT_MARKED_MESSAGES_WITH_TX = SELECT MESSAGEID, MESSAGEBLOB, TXID, TXOP FROM dbo.JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM dbo.JMS_TRANSACTIONS) AND TXOP=?
                 INSERT_MARKED_MESSAGES_WITH_TX = INSERT INTO dbo.SAVED_JMS_MESSAGES SELECT * FROM dbo.JMS_MESSAGES WHERE TXID IN (SELECT TXID FROM dbo.JMS_TRANSACTIONS) AND TXOP=?
                 </attribute>
                 </mbean>
                </server>


                And here is the datasource config:

                <local-tx-datasource>
                 <jndi-name>jdbc/JMSLocalTxDataSource</jndi-name>
                 <connection-url>jdbc:sybase:Tds:dev.gb.xxx.com:7300</connection-url>
                 <driver-class>com.sybase.jdbc2.jdbc.SybDriver</driver-class>
                 <user-name>aaa</user-name>
                 <password>bbb</password>
                 <min-pool-size>5</min-pool-size>
                 <max-pool-size>15</max-pool-size>
                 </local-tx-datasource>




                Rgds,

                Amresh

                • 5. Re: During startup server tries to load all messages in a JM

                  I am having the same problems. The issue is very significant because my application processes millions of messages per day and it has to use Queues to feed the Data warehousing system. I would be gladd to help in correcting this problem. Can you guys give me information on which module has the problem.

                  • 6. Re: During startup server tries to load all messages in a JM
                    genman


                    The JMS/JBDC persistence managers have a method called "restoreQueue":

                    src/main/org/jboss/mq/pm/jdbc2/PersistenceManager.java

                    This method reads all the messages back into memory. Theoretically, even if you didn't have enough memory to read them all in, they be "softened" and JBoss would start, albeit a bit slowly.

                    Adrian had the idea of including special MessageReference markers in the BasicQueue that would force incremental loading from disk when the Queue empties:
                    http://jboss.org/index.html?module=bb&op=viewtopic&t=46186

                    Perhaps an "easy" fix would be to have "restoreQueue" return messages that are "pre-softened". This would work better in cases you have a few large messages, not millions of smaller messages.

                    • 7. Re: During startup server tries to load all messages in a JM

                      Yes, the presoftened message idea ties in with my other requirement
                      to not serialize/deserialize message bodies.

                      http://www.jboss.org/index.html?module=bb&op=viewtopic&t=46187

                      All you actually need is the message properties at recovery, not the whole message.

                      Regards,
                      Adrian

                      • 8. Re: During startup server tries to load all messages in a JM

                        Ok I don't think that this is the problem. At least to me the problem is that the basicQueue attempts to cash every single message. This is a big problem since messages can build up over time to millions adding up to GB of data. Today, as it stands, if the the MDB side shuts down for more than one hour, the messages sitting in the queue will build up to over 400,000. Even if it is just the header, it is not good enough. There needs to be a more effective memory managament such as setting limitations to how much data can be loaded into the basicqueue at once. Let me look into this a little more to see what can be done.

                        thanks,
                        ME

                        • 9. Re: During startup server tries to load all messages in a JM

                          There already is such a constraint, e.g.

                          <attribute name="MaxDepth">10000</attribute>
                          


                          • 10. Re: During startup server tries to load all messages in a JM

                            What file is this in?

                            • 12. Re: During startup server tries to load all messages in a JM

                              I checked the code for the MaxDepth attribute in the BasicQueue code and it will throw an exception if the queue is at the maxDepth. Here is the functionality that I think would be correct:


                              1.) Accept messages in memory and presistant store up to maxDepth.

                              2.) Any messages above maxDepth should only be placed in presisitant store with a special marker that indicates that the message needs to be requeued. The basic queuing FIFO functionality needs to be retain by placing all subsequent messages in the store untill there are no additional marked messages and there is space in memory.

                              3.) We need an addtional thread in the BasicQueue that reads marked messages from the presistant queue and moves them into memory space and the messages in memory are consumed.

                              This would give us protection against over use of memory and will still retain the speed of using memory.

                              Manuel

                              • 13. Re: During startup server tries to load all messages in a JM

                                Agreed, see my post linked to by Genman above.

                                Regards,
                                Adrian