JBossDeveloper

>> Are you starting the test with a clean directory?

I could answer that question myself as your test is validating sequences.. so you can only be doing it with a cleaned journal / data directory.

I still have the other two questions:

- How Long

- Are you sure you are using the updated version

I can't replicate the ones you posted Friday your afternoon.

As a matter of fact... I just woke up and came to look at the test... everything is fine after 10 Hours run. and that's about the third time I put the test to run.

If you could please double check the version you posted (configs.. etc) against your latest post

One important information for me: What OS version are you using?

Can you also describe what kind of hardware you're testing on please? Maybe we have a failure that won't happen on a XYZ combination.

Thanks Ronny,

I did a lot of tests on trying to replicate it.. and I didn't have much success. I may try a different environment with more hardware power... allthough the one I have now is pretty much similar to yours, only difference is I'm using Fedora. But the kernel version and CPU power is similar to yours.. (that's why I asked what env you had).

I also added a stress test that will play all the possible combinations with adds, updates.. tx.. etc and starting compacting, to make sure there isn't any tricky situation I didn't handle. It's passing

I also, changed an internal writing operation on compacting, to avoid any possible races on the OS with writing. I don't have any indication the problem is here... but the change won't hurt anyway.

I will leave the test running here again.. and I will see how it behaves overnight. If you could do the same... (please make sure you do a ./build.sh clean after the update... just to be safe)

Thanks for the help

Clebert

a good news is I could replicate the error.. and I found a bug on readJournal that is used on compactor. That could lead to errors on compacting.

I just committed this fix and I am putting your test to run again. Lets see.

I'm 5 hours on this test.. so far so good.

I am leaving the test running overnight on three different computers.

- My Laptop (limited disk)

- I have a workstation for tests at home.. 4G, SCSI disk... 8 cores .. so a medium box.

- one instance of my performance lab (a powerful box with RAID disks... and a much faster CPU)

Lets see how it goes overnight.

Those are definitely gone:

java.nio.BufferUnderflowException
        at java.nio.Buffer.nextGetIndex(Buffer.java:474)
        at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:205)
        at org.hornetq.core.journal.impl.JournalImpl.readJournalFile(JournalImpl.java:501)
        at org.hornetq.core.journal.impl.JournalImpl.compact(JournalImpl.java:1536)
        at org.hornetq.core.journal.impl.JournalImpl$5.run(JournalImpl.java:2334)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)
[Thread-3 (group:HornetQ-server-threads1623247029-1609377764)] 19:35:00,768 WARNING [org.hornetq.core.postoffice.impl.DuplicateIDCacheImpl]  Error on deleting duplicate cache
java.lang.IllegalStateException: Cannot find add info 68065214
        at org.hornetq.core.journal.impl.JournalImpl.appendDeleteRecord(JournalImpl.java:918)
        at org.hornetq.core.persistence.impl.journal.JournalStorageManager.deleteDuplicateID(JournalStorageManager.java:542)
        at org.hornetq.core.postoffice.impl.DuplicateIDCacheImpl.addToCacheInMemory(DuplicateIDCacheImpl.java:177)
        at org.hornetq.core.postoffice.impl.DuplicateIDCacheImpl.access$000(DuplicateIDCacheImpl.java:42)
        at org.hornetq.core.postoffice.impl.DuplicateIDCacheImpl$AddDuplicateIDOperation.process(DuplicateIDCacheImpl.java:218)
        at org.hornetq.core.postoffice.impl.DuplicateIDCacheImpl$AddDuplicateIDOperation.afterCommit(DuplicateIDCacheImpl.java:238)
        at org.hornetq.core.transaction.impl.TransactionImpl.afterCommit(TransactionImpl.java:436)
        at org.hornetq.core.transaction.impl.TransactionImpl$2.done(TransactionImpl.java:275)
        at org.hornetq.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:207)
        at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:96)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:619)

I couldn't yet replicate those ones, but they might be related:

[pool-4-thread-1] 19:10:43,030 WARNING [org.hornetq.core.journal.impl.JournalCompactor]  Couldn't find addRecord information for record 64312022 during compacting
[pool-4-thread-1] 19:10:43,030 WARNING [org.hornetq.core.journal.impl.JournalCompactor]  Couldn't find addRecord information for record 64312023 during compacting
[pool-4-thread-1] 19:10:43,030 WARNING [org.hornetq.core.journal.impl.JournalCompactor]  Couldn't find addRecord information for record 64312024 during compacting
[pool-4-thread-1] 19:10:43,031 WARNING [org.hornetq.core.journal.impl.JournalCompactor]  Couldn't find addRecord information for record 64312025 during compacting

Also: you will need to update the client libraries on your next tests also.

I'm still expecting those at your tests:

[pool-4-thread-1] 19:10:43,030 WARNING [org.hornetq.core.journal.impl.JournalCompactor]  Couldn't find addRecord information for record 64312022 during compacting
[pool-4-thread-1] 19:10:43,030 WARNING [org.hornetq.core.journal.impl.JournalCompactor]  Couldn't find addRecord information for record 64312023 during compacting
[pool-4-thread-1] 19:10:43,030 WARNING [org.hornetq.core.journal.impl.JournalCompactor]  Couldn't find addRecord information for record 64312024 during compacting

As I could replicate it. However it took me 10 hours last time.

I'm debugging it now.. have added some logging to my copy and lets see if I can get any info. I'm sure I will fix it.. but it could take me a another day or two as this takes time to be replicated. (I feel like a scientist putting his mice to run and take results.. it just takes time).

But I'm still accepting help if you have any other tests that will replicate it easier :-)

BTW: those *seem* harmless so far. But I want to understand it and fix them.