3 Replies Latest reply on Jul 11, 2012 5:52 PM by newway

NPE while queue is paging

newway Jul 11, 2012 12:43 AM

Hello All,

It seems that my system got a OOME while it was paging - so i guess i will need to adjust my paging definitions.

I write 'seems' because the error happened during the night and this is the earliest error I see before the NPE filled up 10 log files

10/07/2012 22:26:52,176 ERROR [org.hornetq.utils.OrderedExecutorFactory] (Thread-27 (HornetQ-server-HornetQServerImpl::serverUUID=87b2a5f6-cac7-11e1-8220-005056b044a0-1685884)) Caught unexpected Throwable: java.lang.OutOfMemoryError
          at sun.misc.Unsafe.allocateMemory(Native Method) [rt.jar:1.6.0_16]
          at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99) [rt.jar:1.6.0_16]
          at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) [rt.jar:1.6.0_16]
          at org.hornetq.core.journal.impl.NIOSequentialFileFactory.allocateDirectBuffer(NIOSequentialFileFactory.java:130) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.persistence.impl.journal.JournalStorageManager.allocateDirectBuffer(JournalStorageManager.java:1655) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.impl.PageImpl.read(PageImpl.java:119) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getPageCache(PageCursorProviderImpl.java:189) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getPageCache(PageCursorProviderImpl.java:140) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.internalGetNext(PageSubscriptionImpl.java:339) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.access$1500(PageSubscriptionImpl.java:65) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.moveNext(PageSubscriptionImpl.java:1177) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1134) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.hasNext(PageSubscriptionImpl.java:1289) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1797) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2454) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_16]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_16]
          at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_16]

I see that it happened while working on page 18 - anf those are my paging definitions:

                    <max-size-bytes>104857600</max-size-bytes>
                    <page-size-bytes>52428800</page-size-bytes>
                    <address-full-policy>PAGE</address-full-policy>

but after i got the error above the system keeps throwing the following error

10/07/2012 22:26:52,316 ERROR [org.hornetq.core.server.impl.QueueImpl] (Thread-15 (HornetQ-server-HornetQServerImpl::serverUUID=87b2a5f6-cac7-11e1-8220-005056b044a0-1685884)) Failed to deliver: java.lang.RuntimeException
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1138) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.hasNext(PageSubscriptionImpl.java:1289) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.server.impl.QueueImpl.deliver(QueueImpl.java:1797) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.server.impl.QueueImpl.access$1100(QueueImpl.java:77) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.server.impl.QueueImpl$DeliverRunner.run(QueueImpl.java:2454) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [hornetq-core-2.2.13.Final.jar:]
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_16]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_16]
          at java.lang.Thread.run(Thread.java:619) [rt.jar:1.6.0_16]
Caused by: java.lang.NullPointerException
          at org.hornetq.core.paging.cursor.impl.PageCacheImpl.getNumberOfMessages(PageCacheImpl.java:110) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.internalGetNext(PageSubscriptionImpl.java:341) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.access$1500(PageSubscriptionImpl.java:65) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.moveNext(PageSubscriptionImpl.java:1177) [hornetq-core-2.2.13.Final.jar:]
          at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1134) [hornetq-core-2.2.13.Final.jar:]
          ... 8 more

is it a defect? I would assume you need to check if the messages member exists before you check for it's length - but I don't know enough of this code and maybe if the PageSubscriptionImpl it is not reasonable that the messages member is null

thanks,

Noa

1. Re: NPE while queue is paging

newway Jul 11, 2012 12:53 AM (in response to newway)

BTW, I put the paging definitions in one entry for all the queues:

<address-settings>
                    <address-setting match="#">
                        <dead-letter-address>jms.queue.DLQ</dead-letter-address>
                        <expiry-address>jms.queue.ExpiryQueue</expiry-address>
                        <redelivery-delay>0</redelivery-delay>
                        <max-size-bytes>104857600</max-size-bytes>
                        <page-size-bytes>52428800</page-size-bytes>
                        <address-full-policy>PAGE</address-full-policy>
                        <message-counter-history-day-limit>10</message-counter-history-day-limit>
                    </address-setting>
                </address-settings>

is it a good practice? I see in the documenatation the following:

Messages are stored per address on the file system. Each address has an individual folder where messages are stored in multiple files (page files). Each file will contain messages up to a max configured size (page-size-bytes). When reading page-files all messages on the page-file are read, routed and the file is deleted as soon as the messages are recovered.

does this mean I need to do the address setting for each queue I am using?

2. Re: NPE while queue is paging

newway Jul 11, 2012 4:43 PM (in response to newway)

Opened JIRA - https://issues.jboss.org/browse/AS7-5154
Actions
3. Re: NPE while queue is paging

newway Jul 11, 2012 5:52 PM (in response to newway)

some clarification - i reduced the max size bytes to 50MB and the page file to 12MB - and the error reoccured
Actions

Go to original post