Couldn't locate page transaction and OutOfMemoryError: Java heap space after upgrade to 2.1.14

[janvandeklok]

Hello,

We run Jboss 6.0.1 and have just migrated hornetQ from version 2.2.5.  to 2.2.14 because we had a strange problem with losing messages from queue's (at least thats what we think what was happening). (see my previous discussion).

We just replaced the hornet*.jar files in jboss with their 2.2.14 equivalents. This should be OK according one of the HornetQ developers.

The jboss runs with the following memory settings set "JAVA_OPTS=-Xms512M -Xmx1536M -XX:MaxPermSize=256M"

These settings have not been changed for over a year now and have always proved to be sufficient.

We have at startup 1,55 GB  of data in 1599 hornet-data-xxxxx.hq files.

We have in total just over 700 queues defined.

In the paging directory there just 1 page for 1 queue (so practicaly nothing is paged)

When we start The jboss server  (this is the first start after the upgrade) we get the following error:

10:54:19,451 INFO  [FileLockNodeManager] Live Server Obtained live lock

10:54:49,780 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036418044, ignoring message on position PagePositionImpl [pageNr=1, messageNr=0, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036418466, ignoring message on position PagePositionImpl [pageNr=1, messageNr=1, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036435382, ignoring message on position PagePositionImpl [pageNr=1, messageNr=2, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036436482, ignoring message on position PagePositionImpl [pageNr=1, messageNr=3, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036441911, ignoring message on position PagePositionImpl [pageNr=1, messageNr=4, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036517094, ignoring message on position PagePositionImpl [pageNr=1, messageNr=5, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036518185, ignoring message on position PagePositionImpl [pageNr=1, messageNr=6, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036555950, ignoring message on position PagePositionImpl [pageNr=1, messageNr=7, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036555954, ignoring message on position PagePositionImpl [pageNr=1, messageNr=8, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,790 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556039, ignoring message on position PagePositionImpl [pageNr=1, messageNr=9, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556066, ignoring message on position PagePositionImpl [pageNr=1, messageNr=10, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556095, ignoring message on position PagePositionImpl [pageNr=1, messageNr=11, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556124, ignoring message on position PagePositionImpl [pageNr=1, messageNr=12, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556167, ignoring message on position PagePositionImpl [pageNr=1, messageNr=13, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556234, ignoring message on position PagePositionImpl [pageNr=1, messageNr=14, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556726, ignoring message on position PagePositionImpl [pageNr=1, messageNr=15, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556773, ignoring message on position PagePositionImpl [pageNr=1, messageNr=16, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556854, ignoring message on position PagePositionImpl [pageNr=1, messageNr=17, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036556903, ignoring message on position PagePositionImpl [pageNr=1, messageNr=18, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036557038, ignoring message on position PagePositionImpl [pageNr=1, messageNr=19, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036557483, ignoring message on position PagePositionImpl [pageNr=1, messageNr=20, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,800 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036557556, ignoring message on position PagePositionImpl [pageNr=1, messageNr=21, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036557673, ignoring message on position PagePositionImpl [pageNr=1, messageNr=22, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036557742, ignoring message on position PagePositionImpl [pageNr=1, messageNr=23, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036557805, ignoring message on position PagePositionImpl [pageNr=1, messageNr=24, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036574592, ignoring message on position PagePositionImpl [pageNr=1, messageNr=25, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036574622, ignoring message on position PagePositionImpl [pageNr=1, messageNr=26, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036575735, ignoring message on position PagePositionImpl [pageNr=1, messageNr=27, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036576778, ignoring message on position PagePositionImpl [pageNr=1, messageNr=28, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036586153, ignoring message on position PagePositionImpl [pageNr=1, messageNr=29, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036592050, ignoring message on position PagePositionImpl [pageNr=1, messageNr=30, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036592800, ignoring message on position PagePositionImpl [pageNr=1, messageNr=31, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036592863, ignoring message on position PagePositionImpl [pageNr=1, messageNr=32, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,810 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15036592969, ignoring message on position PagePositionImpl [pageNr=1, messageNr=33, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15037653786, ignoring message on position PagePositionImpl [pageNr=1, messageNr=34, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15037653794, ignoring message on position PagePositionImpl [pageNr=1, messageNr=35, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15037674517, ignoring message on position PagePositionImpl [pageNr=1, messageNr=36, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15044957735, ignoring message on position PagePositionImpl [pageNr=1, messageNr=37, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15044970604, ignoring message on position PagePositionImpl [pageNr=1, messageNr=38, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15044972011, ignoring message on position PagePositionImpl [pageNr=1, messageNr=39, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15045581674, ignoring message on position PagePositionImpl [pageNr=1, messageNr=40, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15046818591, ignoring message on position PagePositionImpl [pageNr=1, messageNr=41, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15047460845, ignoring message on position PagePositionImpl [pageNr=1, messageNr=42, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,820 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15047460849, ignoring message on position PagePositionImpl [pageNr=1, messageNr=43, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,830 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15047540676, ignoring message on position PagePositionImpl [pageNr=1, messageNr=44, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

10:54:49,830 WARN  [PageSubscriptionImpl] Couldn't locate page transaction 15050082253, ignoring message on position PagePositionImpl [pageNr=1, messageNr=45, recordID=0] on address=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi queue=jms.queue.studielink.delivery.STLK.to.IBG.group.1.kvi

11:03:23,591 ERROR [OrderedExecutorFactory] Caught unexpected Throwable: java.lang.OutOfMemoryError: Java heap space

    at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleStringInternal(ChannelBufferWrapper.java:83) [:]

    at org.hornetq.core.buffers.impl.ChannelBufferWrapper.readSimpleString(ChannelBufferWrapper.java:77) [:]

    at org.hornetq.utils.TypedProperties$StringValue.<init>(TypedProperties.java:1044) [:]

    at org.hornetq.utils.TypedProperties.decode(TypedProperties.java:547) [:]

    at org.hornetq.core.message.impl.MessageImpl.decodeHeadersAndProperties(MessageImpl.java:255) [:]

    at org.hornetq.core.message.impl.MessageImpl.decode(MessageImpl.java:935) [:]

    at org.hornetq.core.message.impl.MessageImpl.decode(MessageImpl.java:479) [:]

    at org.hornetq.core.paging.impl.PagedMessageImpl.decode(PagedMessageImpl.java:126) [:]

    at org.hornetq.core.paging.impl.PageImpl.read(PageImpl.java:147) [:]

    at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getPageCache(PageCursorProviderImpl.java:189) [:]

    at org.hornetq.core.paging.cursor.impl.PageCursorProviderImpl.getPageCache(PageCursorProviderImpl.java:140) [:]

    at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.internalGetNext(PageSubscriptionImpl.java:339) [:]

    at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl.access$1500(PageSubscriptionImpl.java:65) [:]

    at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.moveNext(PageSubscriptionImpl.java:1177) [:]

    at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.next(PageSubscriptionImpl.java:1134) [:]

    at org.hornetq.core.paging.cursor.impl.PageSubscriptionImpl$CursorIterator.hasNext(PageSubscriptionImpl.java:1289) [:]

    at org.hornetq.core.server.impl.QueueImpl$5.run(QueueImpl.java:1249) [:]

    at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100) [:]

    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [:1.6.0_37]

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [:1.6.0_37]

    at java.lang.Thread.run(Thread.java:662) [:1.6.0_37]

We did keep the hornet data that was created by the previous hornetversion 2.1.5!

We have a printout of the journals (> 5 gb ) but I don''t know what to look for for errors in this journal.

What we do see is that due to a delivery issue there have been logged a massive amount of rollbacks for 1 certain message due to an faulty  external receiver. This makes the journals that big. I fear that hornetq tries to read all the journalfiles into memory at once. If this is the case I would like to know whether this is really nessecary. I assume that Hornetq uses the journals to recreate the state of the in-memory parts or the queue's as it was left off when the server got down. Why should the memory usage grow that much when the journal actually just repeats the same action over and over again.

Here's a snippit of these repeating rollback logging :

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889945]

operation@UpdateTX;txID=15052352289,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352289

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889946]

operation@UpdateTX;txID=15052352290,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352290

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889947]

operation@UpdateTX;txID=15052352291,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352291

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889948]

operation@UpdateTX;txID=15052352292,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352292

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889949]

operation@UpdateTX;txID=15052352293,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352293

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889950]

operation@UpdateTX;txID=15052352294,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352294

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889951]

operation@UpdateTX;txID=15052352295,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

My question is : why does this happen??

-  Is it a bug in the 2.2.14 version?

-  Did we have to do some kind of manual conversion of the hornetdata?

-  Can the 2.2.14 version handle these large amounts of journal data?

- How do we get this thing running again without loosing messages ????  (my main concern !!! since we have to do this also in a production   environment where we can't just discard the existing hornetdata upon a conversion)

Question: Are the journalpages freed in memory iimidiatly after they have been processed?? (in that case we should not have this problem) 

Question: Can someone give me a list of userRecordType codes and their meanings???????

Any help is appreciated.

Regards,  Jan van de Klok

[clebert.suconic]

It seems you are hitting a bug that was fixed later than 2.2.14+. Can you test it with the latest version?

[janvandeklok]

Hello Clebert,

   We're not allowed to use beta's or Release candidates for our production environment. I don't see any final version later than 2.2.14.

I nevertheless tried to run our jboss 6.0.1 with all its hornet jars replaced by jars from the 2.3 CR version but ran into Classnot found errors etc. We cant change the jboss version right now.

Can this fix be implemented in a a 2.2.15  bug fix version?? 

Regards Jan van de Klok

[clebert.suconic]

You can go to 2.2.latest.. that's not a beta.

A way to get this is from maven, or if you do a checkout on Branch_2_2_EAP and build it yourself from a latest tag.

[janvandeklok]

Clebert,

  despite the title of the discussion, we are already running 2.2.14 !!     (The 2.1.14 in the title was a typo, but I don't know how to change the title).

As stated in the start of the discussion we run 2.2.5 in production and are now migrating to 2.2.14.

This is where it all happend. So 2.2.14 cant handle the huge journals'!!

[clebert.suconic]

That's why you need to configure paging...

And there was a bug on paging, at 2.2.14 that was fixed on a later version that's being delivered to EAP, and it's being fixed on 2.3.0.next.

you could get 2.2.23 from maven and this paging issue will be fixed.

[clebert.suconic]

BTW: if you use transactions, paging is not infinite.. you still need the journal to load messages.

We are considering a new implementation that will let it be infinite even with transactions.

The fact that you have many can't load page -transactions tells me you are hitting the bug I fixed on 2.2.14.

I will try to help you any way I can.. but you will need an upgrade first.

Can you replicate this on a test env?

[janvandeklok]

We are already using paging!

[janvandeklok]

I will upgrade to 2.2.23 first.

I'm out for a 3 week holiday the day after tomorrow, so i'm no sure I get this all done by tomorrow.

Jan

[janvandeklok]

Couldn't find 2.2.23 version in the nexus repository.  downloaded the highest available version 2.2.21  from the nexus repository and installed it on our test server.

It still has problems loading the huge amount of journal data. I have to increase the amount of memory of the jvm to finish loading the journals.

Does this new version make smaller journals (more efficient journalling of continues failing deliveries) than the 2.2.5 version??

[clebert.suconic]

2.2.5 had a bug on the journal...

But a Journal is not supposed to get huge...  and if you start to page and send every message transactionally (one transaction per message), you will have on record per message no matter what on the journal. So, be careful with transactions and paging!

A Journal is always supposed to fit the memory. It's written for performance and it's supposed to be a safeguard for crashes as a way to recover the memory.

[janvandeklok]

Clebert,

this is the reason why the journals are so huge:

Here's a snippit of these repeating rollback logging :

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889945]

operation@UpdateTX;txID=15052352289,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352289

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889946]

operation@UpdateTX;txID=15052352290,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352290

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889947]

operation@UpdateTX;txID=15052352291,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352291

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889948]

operation@UpdateTX;txID=15052352292,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352292

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889949]

operation@UpdateTX;txID=15052352293,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352293

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889950]

operation@UpdateTX;txID=15052352294,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

operation@Rollback;txID=15052352294

operation@Update;recordID=15047461285;userRecordType=34;isUpdate=true;DeliveryCountUpdateEncoding [queueID=3829, count=4889951]

operation@UpdateTX;txID=15052352295,recordID=15047461285;userRecordType=33;isUpdate=true;ACK;QueueEncoding [queueID=3829]

There was 1 failing delivery address which cause the system to rolback the same mesage over and over again due to a programmatic flaw (I hate the bug word :-)

look at the count : count=4889951] 

I assume that all this roll back logging info does not need to be loaded into memory which it saeems to do. These massive amount of logghing statements seem to cause the problem!!!

Can you confirm that?

Its not that we have a lot of message around. Just a few with 1 failing constantly to deliver causing the journals to fill up (over 1600 journal pages )

That's what i'm trying to explain all the time!!

[clebert.suconic]

You could have DLA (Dead Letter Address) setup, and this wouldn't happen (it would be sent to the Dead Letter Queue through the Dead Letter Address).

There was some change around the Transaction for this... I believe the transaction is never committed or rolled back. I believe the newer versions will treat this better.