11 Replies Latest reply on Jul 24, 2002 1:12 PM by lindsey

    Problem reading from large queues

    lindsey

      I've been running some tests using a Postgres-backed queue for queue sizes exceeding 500 megs. Everything seems to go into the queue fine. As memory fills up, cache files are being written to move messages out of ram. To force writing to the file system by the cache for testing, I set the low and high mark for caching to 64 megs. The problem occurs when I try to read the messages back out. I get the following exception that indicates it can't find one of the cache files:

      org.jboss.mq.SpyJMSException: Could not load message from secondary storage:
      at org.jboss.mq.pm.file.CacheStore.loadFromStorage(CacheStore.java:54)
      at org.jboss.mq.server.MessageCache.loadFromStorage(MessageCache.java:245)
      at org.jboss.mq.server.MessageReference.makeHard(MessageReference.java:207)
      at org.jboss.mq.server.MessageReference.getMessage(MessageReference.java:93)
      at org.jboss.mq.server.BasicQueue.setupMessageAcknowledgement(BasicQueue.java:341)
      at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:206)
      at org.jboss.mq.server.JMSQueue.receive(JMSQueue.java:117)
      at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:207)
      at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager.java:668)
      at org.jboss.mq.server.JMSServerInterceptorSupport.receive(JMSServerInterceptorSupport.java:225)
      at org.jboss.mq.security.ServerSecurityInterceptor.receive(ServerSecurityInterceptor.java:96)
      at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:422)
      at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:227)
      at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:279)
      at java.lang.Thread.run(Thread.java:536)
      linked exception is:
      java.io.FileNotFoundException: /home/aaron/jboss/dist/jboss-3.0.0_tomcat-4.0.3/server/default/tmp/jbossmq/Message-0 (No such file or directory)
      at java.io.FileInputStream.open(Native Method)
      at java.io.FileInputStream.(FileInputStream.java:103)
      at org.jboss.mq.pm.file.CacheStore.loadFromStorage(CacheStore.java:47)
      at org.jboss.mq.server.MessageCache.loadFromStorage(MessageCache.java:245)
      at org.jboss.mq.server.MessageReference.makeHard(MessageReference.java:207)
      at org.jboss.mq.server.MessageReference.getMessage(MessageReference.java:93)
      at org.jboss.mq.server.BasicQueue.setupMessageAcknowledgement(BasicQueue.java:341)
      at org.jboss.mq.server.BasicQueue.receive(BasicQueue.java:206)
      at org.jboss.mq.server.JMSQueue.receive(JMSQueue.java:117)
      at org.jboss.mq.server.ClientConsumer.receive(ClientConsumer.java:207)
      at org.jboss.mq.server.JMSDestinationManager.receive(JMSDestinationManager.java:668)
      at org.jboss.mq.server.JMSServerInterceptorSupport.receive(JMSServerInterceptorSupport.java:225)
      at org.jboss.mq.security.ServerSecurityInterceptor.receive(ServerSecurityInterceptor.java:96)
      at org.jboss.mq.server.TracingInterceptor.receive(TracingInterceptor.java:422)
      at org.jboss.mq.server.JMSServerInvoker.receive(JMSServerInvoker.java:227)
      at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:279)
      at java.lang.Thread.run(Thread.java:536)


      I verified that Message-0 does not exist in the cache directory (the message files start at Message-1). Has anyone experienced anything similar? Are there any workarounds for this?

      Here are the details of my system:
      RH Linux 7.2
      JDK-1.4 (1.3.1 exhibits the same behavior)
      Postgres-7.1.3
      JBoss-3.0.0 (3.0.1RC1 exhibits the same behavior)

      Thanks,
      Aaron

        • 1. Re: Problem reading from large queues
          hchirino

          Hi.. the MessageCache should be using the db for storing messages that it swaps out of memory.

          is the CacheStore attribute of your MessageCache set as forllows? :


          500
          600
          jboss.mq:service=PersistenceManager



          • 2. Re: Problem reading from large queues
            lindsey

            Thanks for the response. I came to a similar conclusion on my own, but with one problem. I can't get the system to run with the configuration as you've shown. The PersistenceManager doesn't start because of a circular dependence (I think that's the problem, at least). The MessageCache depends on the PersistenceManager as its CacheStore. The PersistenceManager depends on the MessageCache. Basically nothing in the JMS system comes up when I use that configuration.
            The only way I found to get around this, was to create another MessgageCache and CacheStore called DummyMessageCache and DummyCacheStore respectively. I then had to create another jdbc2.PersistenceManager to be used as the CacheStore. Here's the setup:


            500
            600
            <depends optional-attribute-name="CacheStore">jboss.mq:service=CacheStore



            500
            600
            <depends optional-attribute-name="CacheStore">jboss.mq:service=DummyCacheStore



            tmp/jbossmq



            <depends optional-attribute-name="MessageCache">jboss.mq:service=DummyMessageCache
            <!-- standard jdbc2 PM setup below here -->



            <depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache
            <!-- Normal jdbc2 PM configuration below here -->



            It's pretty kludgey, but that's the only way I could resolve the dependence problem. If there's a better way to do it, I'd be glad to hear it. Also, would creating a second instance of the jdbc2 PM, as I'm doing, cause any problems? I'm running into errors where it looks like the PM is trying to store messages that the CacheStore has already taken care of.

            Aaron




            • 3. Re: Problem reading from large queues
              lindsey

              Thanks for the response. I came to a similar conclusion on my own, but with one problem. I can't get the system to run with the configuration as you've shown. The PersistenceManager doesn't start because of a circular dependence (I think that's the problem, at least). The MessageCache depends on the PersistenceManager as its CacheStore. The PersistenceManager depends on the MessageCache. Basically nothing in the JMS system comes up when I use that configuration.
              The only way I found to get around this, was to create another MessgageCache and CacheStore called DummyMessageCache and DummyCacheStore respectively. I then had to create another jdbc2.PersistenceManager to be used as the CacheStore. Here's the setup:


              500
              600
              <depends optional-attribute-name="CacheStore">jboss.mq:service=CacheStore



              500
              600
              <depends optional-attribute-name="CacheStore">jboss.mq:service=DummyCacheStore



              tmp/jbossmq



              <depends optional-attribute-name="MessageCache">jboss.mq:service=DummyMessageCache
              <!-- standard jdbc2 PM setup below here -->



              <depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache
              <!-- Normal jdbc2 PM configuration below here -->



              It's pretty kludgey, but that's the only way I could resolve the dependence problem. If there's a better way to do it, I'd be glad to hear it. Also, would creating a second instance of the jdbc2 PM, as I'm doing, cause any problems? I'm running into errors where it looks like the PM is trying to store messages that the CacheStore has already taken care of.

              Aaron




              • 4. Re: Problem reading from large queues
                lindsey

                Thanks for the response. I came to a similar conclusion on my own, but with one problem. I can't get the system to run with the configuration as you've shown. The PersistenceManager doesn't start because of a circular dependence (I think that's the problem, at least). The MessageCache depends on the PersistenceManager as its CacheStore. The PersistenceManager depends on the MessageCache. Basically nothing in the JMS system comes up when I use that configuration.
                The only way I found to get around this, was to create another MessgageCache and CacheStore called DummyMessageCache and DummyCacheStore respectively. I then had to create another jdbc2.PersistenceManager to be used as the CacheStore. Here's the setup:


                500
                600
                <depends optional-attribute-name="CacheStore">jboss.mq:service=CacheStore



                500
                600
                <depends optional-attribute-name="CacheStore">jboss.mq:service=DummyCacheStore



                tmp/jbossmq



                <depends optional-attribute-name="MessageCache">jboss.mq:service=DummyMessageCache
                <!-- standard jdbc2 PM setup below here -->



                <depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache
                <!-- Normal jdbc2 PM configuration below here -->



                It's pretty kludgey, but that's the only way I could resolve the dependence problem. If there's a better way to do it, I'd be glad to hear it. Also, would creating a second instance of the jdbc2 PM, as I'm doing, cause any problems? I'm running into errors where it looks like the PM is trying to store messages that the CacheStore has already taken care of.

                Aaron



                • 5. Re: Problem reading from large queues
                  lindsey

                  Thanks for the response. I came to a similar conclusion on my own, but with one problem. I can't get the system to run with the configuration as you've shown. The PersistenceManager doesn't start because of a circular dependence (I think that's the problem, at least). The MessageCache depends on the PersistenceManager as its CacheStore. The PersistenceManager depends on the MessageCache. Basically nothing in the JMS system comes up when I use that configuration.
                  The only way I found to get around this, was to create another MessgageCache and CacheStore called DummyMessageCache and DummyCacheStore respectively. I then had to create another jdbc2.PersistenceManager to be used as the CacheStore. Here's the setup:


                  500
                  600
                  <depends optional-attribute-name="CacheStore">jboss.mq:service=CacheStore



                  500
                  600
                  <depends optional-attribute-name="CacheStore">jboss.mq:service=DummyCacheStore



                  tmp/jbossmq



                  <depends optional-attribute-name="MessageCache">jboss.mq:service=DummyMessageCache
                  <!-- standard jdbc2 PM setup below here -->



                  <depends optional-attribute-name="MessageCache">jboss.mq:service=MessageCache
                  <!-- Normal jdbc2 PM configuration below here -->



                  It's pretty kludgey, but that's the only way I could resolve the dependence problem. If there's a better way to do it, I'd be glad to hear it. Also, would creating a second instance of the jdbc2 PM, as I'm doing, cause any problems? I'm running into errors where it looks like the PM is trying to store messages that the CacheStore has already taken care of.

                  Aaron



                  • 6. Re: Problem reading from large queues
                    hchirino

                    Notice the difference between mine and yours:


                    500
                    600
                    jboss.mq:service=PersistenceManager



                    My CacheStore attribute uses a regular tag instead of a tag. That lets the deployment to succeed.

                    Regards,
                    Hiram

                    • 7. Re: Problem reading from large queues
                      lindsey

                      Hiram,
                      Thanks for the clarification. Everything starts up fine with your configuration. I'm still having problems, though. Everything works great until the cache finally comes into play. As I add messages, I've verified that PersistenceManager.add() is being called for each message and each is being put into the database. After a few messages, it's time for the cache to kick in. Before add() is called for the new message, saveToStorage() is called and the message is saved to the database. I've verified that at the end of saveToStorage() messageRef.isStored is set to true. Then, removeFromStorage() is immediately called, which just returns since it's a persistent message (I've verified that it just returns). Finally, add() is called, but now isStored is set to false, so it attempts to store the message again with the same messageid and chokes:

                      org.jboss.mq.SpyJMSException: Could not store message: 5
                      at org.jboss.mq.pm.jdbc2.PersistenceManager.add(PersistenceManager.java:660)
                      at org.jboss.mq.server.PersistentQueue.addMessage(PersistentQueue.java:41)
                      at org.jboss.mq.server.JMSQueue.addMessage(JMSQueue.java:111)
                      at org.jboss.mq.server.JMSDestinationManager.addMessage(JMSDestinationManager.java:397)
                      at org.jboss.mq.server.JMSDestinationManager.addMessage(JMSDestinationManager.java:375)
                      at org.jboss.mq.server.JMSServerInterceptorSupport.addMessage(JMSServerInterceptorSupport.java:135)
                      at org.jboss.mq.security.ServerSecurityInterceptor.addMessage(ServerSecurityInterceptor.java:155)
                      at org.jboss.mq.server.TracingInterceptor.addMessage(TracingInterceptor.java:206)
                      at org.jboss.mq.server.JMSServerInvoker.addMessage(JMSServerInvoker.java:137)
                      at org.jboss.mq.il.oil.OILServerILService$Client.run(OILServerILService.java:233)
                      at java.lang.Thread.run(Thread.java:536)
                      linked exception is:
                      java.sql.SQLException: ERROR: Cannot insert a duplicate key into unique index jms_messages_pkey

                      Any ideas as to what might be causing this?

                      Thanks,

                      Aaron

                      • 8. Re: Problem reading from large queues
                        lindsey

                        Okay, I think I have the problem isolated. What I'm seeing is an exception when a message is saved to storage in the cache before it is added through the persistence manager. The JMSQueue.addMessage() method adds the message to the cache (getting back a MessageReference) and then passes this to PersistentQueue.addMessage(). In my test case, the message I'm adding to the queue is stored to the DB by the cache immediately upon being added to the cache by JMSQueue. Then, when PersistentQueue.addMessage() is called, the isStored flag is blown away with the invalidate() call, and the PersistenceManager ends up trying to store the message again, causing an error.
                        My proposed solution would be to get rid of the invalidate() call within PersistentQueue.addMessage(). This also forces a change within the jdbc2 PersistenceManager.add() method to pass a non-null connection to markMessage. This seems to make things work as expected.
                        I have a few concerns. Are there side effects to removing the invalidate() call? Is the problem I'm seeing really a problem or could it be a configuration issue on my side? Is this fixed in a later version of JBoss than 3.0.0?
                        Anyone have any comments/suggestions?

                        Aaron

                        • 9. Re: Problem reading from large queues
                          hchirino

                          I can't remember if that was fixed post 3.0.0, try running it at least against 3.0.1. Better if if you running it against the CVS head.

                          Regards,
                          Hiram

                          • 10. Re: Problem reading from large queues
                            hchirino

                            P.S. Could you post the SqlProperties that you use when you run against Postgres??

                            Thanks,
                            Hiram

                            • 11. Re: Problem reading from large queues
                              lindsey

                              Hiram,
                              I tried to run a test against CVS head, but I couldn't get jboss to startup properly with the default configuration after compiling. I did run the test against 3.0.1RC1 and got the same results. Doing diffs between 3.0.0 and 3.0.1RC1 doesn't show any changes (below messaging/src/main/org/jboss/mq). Since I couldn't run the test for head, I diffed the 3.0.1RC1 code vs. CVS head, and there didn't look to be any significant changes between those two either (significant meaning that it looks like it might affect this problem). It looks like it's a real bug to me.

                              Here is the SqlProperties attribute that I am using for Postgres:

                              BLOB_TYPE=BYTES_BLOB
                              INSERT_TX = INSERT INTO JMS_TRANSACTIONS (TXID) values(?)
                              INSERT_MESSAGE = INSERT INTO JMS_MESSAGES (MESSAGEID, DESTINATION, MESSAGEBLOB, TXID, TXOP) VALUES(?,?,?,?,?)
                              SELECT_ALL_UNCOMMITED_TXS = SELECT TXID FROM JMS_TRANSACTIONS
                              SELECT_MAX_TX = SELECT MAX(TXID) FROM JMS_MESSAGES
                              SELECT_MESSAGES_IN_DEST = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES
                              WHERE DESTINATION=?
                              SELECT_MESSAGE = SELECT MESSAGEID, MESSAGEBLOB FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
                              MARK_MESSAGE = UPDATE JMS_MESSAGES SET TXID=?, TXOP=? WHERE MESSAGEID=? AND DESTINATION=?
                              DELETE_ALL_MESSAGE_WITH_TX = DELETE FROM JMS_MESSAGES WHERE TXID=?
                              DELETE_TX = DELETE FROM JMS_TRANSACTIONS WHERE TXID = ?
                              DELETE_MARKED_MESSAGES = DELETE FROM JMS_MESSAGES WHERE TXID=? AND TXOP=?
                              DELETE_MESSAGE = DELETE FROM JMS_MESSAGES WHERE MESSAGEID=? AND DESTINATION=?
                              CREATE_MESSAGE_TABLE = CREATE TABLE JMS_MESSAGES ( MESSAGEID INTEGER NOT NULL, \
                              DESTINATION VARCHAR(50) NOT NULL, TXID INTEGER, TXOP CHAR(1), \
                              MESSAGEBLOB BYTEA, PRIMARY KEY (MESSAGEID, DESTINATION) )
                              CREATE_TX_TABLE = CREATE TABLE JMS_TRANSACTIONS ( TXID INTEGER )
                              CREATE_TABLES_ON_STARTUP=false

                              Let me know what you think. Thanks.

                              Aaron