Missing nodes after concurrent inserts (ModeShape 3.8.0)

[nl]

Hi ModeShapers,

my problem as follows: Our application is creating plenty of documents concurrently in a jdbc-based cache store. But even there are no exception during processing some documents (or nodes) are not available afterwards (ItemNotFoundException).

I can reproduce this in a very simple test case (see attached): 5000 nt:file nodes are created with 100 threads using a Threadpool. Each thread is using its own session. After calling session.save() this test logs the file node's identifier. After processing all documents it checks the existence of each identifier. I'd expect that each identifier is known but is not.

At first I thought that this is related to MODE-2280 but it does not seems to be.

It feels like a bug. Any idea?

Thanks, Niels

[hchiorean]

Running the attached project locally I'm unable to reproduce the problem that you're reporting - i.e. the test runs fine.

Also, you're not using a *real* transaction manager, meaning Infinispan will default to the DummyTransactionManager. The ISPN docs state the following:

DummyTransactionManagerLookup : This provides with a dummy transaction manager which should only be used for testing.  Being a dummy, this is not recommended for production use a it has some severe limitations to do with concurrent transactions and recovery.

So I would try a couple of things:

• update your POM to use narayana-jta (it's defined in the ModeShape BOM)
• if after the above change your test still fails, try using the latest build from the 3.x branch (which will de facto be 3.8.1)

[nl]

Thanks for having a look, Horia!

I notice the issue in our production enviroment where the code in the Runnable is modeled as an EJB with CMT. So the issue also occurs using a "real" transaction manager. Nevertheless I switched to narayana-jta but the issue still occurs (4946/5000).

[hchiorean]

Did you try running the latest 3.x code ? Also, did you try without registering ISPN as a synchronization (i.e without useSynchronization="true") ?

[nl]

Same without synchronization. But I needed to increase the doc count to 10000 to reproduce the issue.

I've not tried the 3.x branch yet. I'll check that later.

[hchiorean]

If the issue is still there with the latest 3.x, I suspect it may be another similar ISPN bug as MODE-2280. If that is the case, you could try writing a much simple test case using only the ISPN cache (and gets/puts like in modeshape/SchematicDbConcurrentTest.java at master · ModeShape/modeshape · GitHub) and then report the issue on the Infinispan JIRA. If you're unable to reproduce the issue using only Infinispan, then this may be a ModeShape bug.

BTW, what OS is this happening on ? (MODE-2280 seems to only be happening on Windows)

[nl]

• Same issue with 3.x branch.
• Issue is not depending on OS (Windows and Linux)

With the test case applied, can you try to increase the file node count to 10.000? You should face the issue aswell.

[hchiorean]

Running with 10.000 failed at the end, but only after 1h:30 minutes of running on my machine, which is totally impractical for any sort of investigation. Did you try running with other cache stores ?

Again, for any chance of actually investigating this, a test case which fails a lot sooner is required (also, as mentioned above I would try writing a test case directly against Infinispan, since I doubt this is ModeShape related).

[hchiorean]

If it helps, you can look at the ISPN test case I created for [ISPN-4810] Local Transactional Cache looses data when eviction is enabled and there are multiple readers and one writer… (should be really easy to adapt to your case & ISPN 5)