3 Replies Latest reply on Jul 11, 2012 5:52 PM by Noa Drach

    NPE while queue is paging

    Noa Drach Novice

      Hello All,

       

      It seems that my system got a OOME while it was paging - so i guess i will need to adjust my paging definitions.

      I write 'seems' because the error happened during the night and this is the earliest error I see before the NPE filled up 10 log files

       

      10/07/2012 22:26:52,176 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-27 (HornetQ-server-HornetQServerImpl::serverUUID=87b2a5f6-cac7-11e1-8220-005056b044a0-1685884)) Caught unexpected Throwable: java.lang.OutOfMemoryError
                at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.6.0_16]
                at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99) [rt.jar:1.6.0_16]
                at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) [rt.jar:1.6.0_16]
                at org.hornetq.core.journal.impl.NIOSequentialFileFactory.allocateDirectBuffer(NIOSequentialFileFactory.java:130) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.persistence.impl.journal.JournalStorageManager.allocateDirectBuffer(JournalStorageManager.java:1655) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.impl.PageImpl.read(PageImpl.java:119) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getPageCache(PageCursorProviderImpl.java:189) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getPageCache(PageCursorProviderImpl.java:140) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.internalGetNext(PageSubscriptionImpl.java:339) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.access$1500(PageSubscriptionImpl.java:65) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.moveNext(PageSubscriptionImpl.java:1177) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1134) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.hasNext(PageSubscriptionImpl.java:1289) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1797) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2454) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_16]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_16]
                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_16]
      

       

      I see that it happened while working on page 18 - anf those are my paging definitions:

       

                          <max-size-bytes>104857600</max-size-bytes>
                          <page-size-bytes>52428800</page-size-bytes>
                          <address-full-policy>PAGE</address-full-policy>
      
      

      but after i got the error above the system keeps throwing the following error

       

      10/07/2012 22:26:52,316 ERROR [org.hornetq.core.server.impl.QueueImpl] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=87b2a5f6-cac7-11e1-8220-005056b044a0-1685884)) Failed to deliver: java.lang.RuntimeException
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1138) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.hasNext(PageSubscriptionImpl.java:1289) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1797) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2454) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_16]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_16]
                at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_16]
      Caused by: java.lang.NullPointerException
                at org.hornetq.core.paging.cursor.impl.PageCacheImpl.getNumberOfMessages(PageCacheImpl.java:110) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.internalGetNext(PageSubscriptionImpl.java:341) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.access$1500(PageSubscriptionImpl.java:65) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.moveNext(PageSubscriptionImpl.java:1177) [hornetq-core-2.2.13.Final.jar:]
                at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1134) [hornetq-core-2.2.13.Final.jar:]
                ... 8 more
      
      

       

      is it a defect? I would assume you need to check if the messages member exists before you check for it's length - but I don't know enough of this code and maybe if the PageSubscriptionImpl it is not reasonable that the messages member is null

       

      thanks,

      Noa

        • 1. Re: NPE while queue is paging
          Noa Drach Novice

          BTW, I put the paging definitions in one entry for all the queues:

          <address-settings>
                              <address-setting match="#">
                                  <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                                  <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                                  <redelivery-delay>0</redelivery-delay>
                                  <max-size-bytes>104857600</max-size-bytes>
                                  <page-size-bytes>52428800</page-size-bytes>
                                  <address-full-policy>PAGE</address-full-policy>
                                  <message-counter-history-day-limit>10</message-counter-history-day-limit>
                              </address-setting>
                          </address-settings>
          
          

           

           

          is it a good practice? I see in the documenatation the following:

           

           

          Messages are stored per address on the file system. Each address has an individual folder where messages are stored in multiple files (page files). Each file will contain messages up to a max configured size (page-size-bytes). When reading page-files all messages on the page-file are read, routed and the file is deleted as soon as the messages are recovered.

           

          does this mean I need to do the address setting for each queue I am using?

          • 3. Re: NPE while queue is paging
            Noa Drach Novice

            some clarification - i reduced the max size bytes to 50MB and the page file to 12MB - and the error reoccured