7 Replies Latest reply on Jan 27, 2015 9:57 AM by hchiorean

    Optimization in performance-test environment

    brmeyer

      Hey guys, lately I've been focused on scaling up and profiling Artificer.  The initial test is simple: push in a large amount of metadata-only artifacts (no documents, no additional children per artifact).  Initial profiles show a couple small CPU hotspots within Artificer itself, but nothing major.  No obvious memory issues came up, other than, of course, Infinispan sitting on a decent (but stable) chunk.


      The primary issue is either ModeShape or Infinispan become unresponsive after around 9k artifacts (currently just see the transactions start timing out -- haven't dug in any further).  I have a few optimization ideas, but would anyone be willing to look at our configs and see if anything obvious sticks out?

       

      Wildfly config: https://gist.github.com/brmeyer/6766cd6c7a930ded4440

      (Simple ModeShape 4.0 repo, Infinispan cache w/ string-keyed-jdbc-store, and H2 datasource)

       

      One thing I know we need to add is "document-optimization", right?  Although, am I correct that its threads assume more of a production environment with an interval on the order of hours?  Would any document-optimization setup be viable for a performance test where thousands of nodes are crammed in within minutes?

       

      Anything else obvious?  Apologies if I'm missing something simple...

        • 1. Re: Optimization in performance-test environment
          brmeyer

          After making a few optimizations within Aritificer, CPU performance has improved.  Performance does start to somewhat degrade after around 20k nodes, then the transactions stall just before 40k.  Is the lack of document-optimization the only issue?  Any other thoughts?

          One thing I know we need to add is "document-optimization", right?  Although, am I correct that its threads assume more of a production environment with an interval on the order of hours?  Would any document-optimization setup be viable for a performance test where thousands of nodes are crammed in within minutes?

          • 2. Re: Optimization in performance-test environment
            hchiorean

            There is one "mandatory" configuration option which is missing from your configuration: eviction. This is a must in any real system (unless you have unlimited memory) since without it you'll run OOM sooner or later - without it Infinispan/ModeShape will hold all the entries in memory and those will accumulate over time. You enable it via the  <eviction strategy="LRU" max-entries="...."/> element. The max-entries is something you'll have to "play with" to come up with an optimal setting for your app.

            The other (non performance related) setting you need when running in WF is to enable READ_COMMITTED isolation: see Configuration - ModeShape 4 - Project Documentation Editor.

             

            The document optimization feature has been designed to "alleviate" the particular case of storing large amounts of children under the same parent. I say alleviate because in general ModeShape will not perform well with this type of structure and there is no way around it. Ideally you should design/prefer deep hierarchies, not wide ones. Of course the actual number of siblings from which performance will start degrading significantly is context dependent and something you should profile (we have simple tests where 100k nodes under the same parent work really well, but 500k don't). You can read more here: Large numbers of child nodes - ModeShape 4 - Project Documentation Editor

             

            Also, because of this bug: [MODE-2391] Optimization feature cannot triggered by enabling the document-optimization-child-count-target in Wildfly co… the document optimization feature cannot be enabled atm. in WF & 4.1. This should be fixed in the upcoming 4.2 release.

            1 of 1 people found this helpful
            • 3. Re: Optimization in performance-test environment
              brmeyer

              Ouch, I can't believe I missed the ISPN config.  Thanks much for pointing that out...

               

              And thanks for the doc-opt info.  We're already following the path recommendations in https://modeshape.wordpress.com/2014/08/14/improving-performance-with-large-numbers-of-child-nodes/, so it sounds like it wouldn't really help us anyway.

               

              Thanks Horia!

              • 4. Re: Optimization in performance-test environment
                brmeyer

                There's one other issue I'm seeing, using the same configs (but with ISPN corrected).  This test uses a much larger node with a fair amount of fields, including a Binary containing an XSD file.  If I rapidly add the nodes, something like the following happens:

                 

                ...

                rootNode.addNode(...)    <-- inserting an [sramp:baseArtifactType] node that includes an [sramp:uuid] property

                session.save()

                QueryManager jcrQueryManager = session.getWorkspace().getQueryManager();

                String jcrSql2Query = String.format("SELECT * FROM [sramp:baseArtifactType] WHERE [sramp:uuid] = '%1$s'", artifactUuid);    <-- artifactUuid is the [sramp:uuid] inserted above

                Query jcrQuery = jcrQueryManager.createQuery(jcrSql2Query, JCRConstants.JCR_SQL2);

                QueryResult jcrQueryResult = jcrQuery.execute();

                NodeIterator jcrNodes = jcrQueryResult.getNodes();

                 

                After around 30 or so iterations, the query doesn't return the newly created node.  Continuing the loop, the query will periodically return the node, but returns nothing for the majority.

                 

                Are we hitting some sort of MS or ISPN race condition?  Transactional issue?  Wildfly 8.2, MS 4.0, and ISPN 6.0.2

                 

                If you think it merits a JIRA, let me know -- I'll try to come up with a test case

                • 5. Re: Optimization in performance-test environment
                  hchiorean

                  Whatever it is you're seeing, it's not straightforward:

                  - are you using index definitions ? If yes, are they configured to be synchronous ?

                  - are the nodes which are not being returned by the query present in the repository at all ? (if you try to retrieve them by path for example)

                  - are you using a single writer thread or multiple writer threads ? (by writer thread I mean any thread that calls session.save) If you're using multiple writers (sequencers for example always run off separate threads) have you configured Infinispan to use READ_COMMITTED isolation ?

                  - if you're using multiple writer threads, you should be aware of this issue: [MODE-2280] Child node not found under high concurrency when eviction is enabled and SingleFile store is used - JBoss Is… which is caused by a bug in Infinispan not limited only to SingleFile but other cache stores as well

                  etc...

                   

                  If you can reproduce your issue in a runnable piece of code, feel free to attach it & open a JIRA. Thanks.

                  1 of 1 people found this helpful
                  • 6. Re: Re: Optimization in performance-test environment
                    brmeyer

                    - are you using index definitions ? If yes, are they configured to be synchronous ?

                    Not currently using index definitions

                    - are the nodes which are not being returned by the query present in the repository at all ? (if you try to retrieve them by path for example)

                    Yes, I believe so -- appears to be limited to querying

                    - are you using a single writer thread or multiple writer threads ? (by writer thread I mean any thread that calls session.save) If you're using multiple writers (sequencers for example always run off separate threads) have you configured Infinispan to use READ_COMMITTED isolation ?

                    Multiple writer threads (the performance test has to go through EJB).  However, the write and read are on the same thread each time, if that makes a difference.

                    - if you're using multiple writer threads, you should be aware of this issue: [MODE-2280] Child node not found under high concurrency when eviction is enabled and SingleFile store is used - JBoss Is… which is caused by a bug in Infinispan not limited only to SingleFile but other cache stores as well

                    Bingo, that's probably it.  I have a few workarounds that may allivate that in our specific case, but I'll keep an eye on it.

                     

                    Thanks again!

                    • 7. Re: Re: Optimization in performance-test environment
                      hchiorean

                      If you're certain that the nodes are persisted in the repository, can you try retrieving them via the parent node or path ? If that works, then it can't be the previously mentioned Infinispan issue, as that results in nodes not being added to parents at all. If this is the case, then it might be something else so a test-case might be worth it.

                      If however you're unable to retrieve the nodes via path/parent, then most likely it is the ISPN bug you're seeing.