6 Replies Latest reply on Feb 15, 2017 5:48 AM by Vincent Ragot

    Slow DataGrid server insert

    ALEXIS BOURDEAU Newbie

      Hello,

       

      I have developped a java client to insert into a datagrid server 300 000 Payments objects.

      I have one node (standalone.xml)

       

      It takes around 10 min to absorb. This is quite long.
      the code is the following :

       

      FileManager fm = new FileManager();

              List<Paiement> paiements = fm.readFile();

              if ((paiements == null) || (paiements.size() == 0))

                  System.out.println("nothing to print");

              else

              { // from here, it takes 10 minutes

                  for (Paiement paie : paiements) {

                      remoteCache.putAsync(paie.getId(), paie);

                  }

              }

       

      How can I decrease the time to insert this volume of object in the grid ?

       

      Thanks a lot !

        • 1. Re: Slow DataGrid server insert
          Tristan Tarrant Master

          You should batch things and use the bulk methods (putAll) as they are more efficient.

          1 of 1 people found this helpful
          • 2. Re: Slow DataGrid server insert
            Vincent Ragot Newbie

            Hello,

            Thanks for your response.

            It seem interesting, I will try this.

            1 of 1 people found this helpful
            • 3. Re: Slow DataGrid server insert
              Vincent Ragot Newbie

              I'm worked on the same project, and I tested the putAll operation with 1 element into the map.

               

              But it throws an exception:

               

              org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=7 returned server error (status=0x86): org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 20 seconds for key [B0x2800 and requestor CommandUUID{address=mind7-virtual-machine, id=70}. Lock is held by CommandUUID{address=mind7-virtual-machine, id=11}

                  at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)

                  at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:132)

                  at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:118)

                  at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)

                  at org.infinispan.client.hotrod.impl.operations.PutAllOperation.executeOperation(PutAllOperation.java:56)

                  at org.infinispan.client.hotrod.impl.operations.PutAllOperation.executeOperation(PutAllOperation.java:25)

                  at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:54)

                  at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putAll(RemoteCacheImpl.java:224)

                  at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putAll(RemoteCacheSupport.java:49)

                  at PaiementManager.addPaiementFromFile(PaiementManager.java:151)

                  at Main.main(Main.java:43)

               

              Have you an Idea ?

              • 4. Re: Slow DataGrid server insert
                ALEXIS BOURDEAU Newbie

                Hello,

                we found the bottleneck.

                We got in out cache this set :

                 

                <!-- Enable indexing using the RAM Lucene directory provider -->

                                    <indexing index="ALL">

                                        <property name="default.directory_provider">ram</property>

                                    </indexing>

                 

                removing this deccrease greatly the time necessary to insert data.

                 

                We have to investigate this.

                • 5. Re: Slow DataGrid server insert
                  Tristan Tarrant Master

                  Aha. Indexing does indeed slow things down.

                  A couple of things: for bulk loading you can add the SKIP_INDEXING flag:

                   

                  RemoteCache noIndexingCache = cache.withFlags(Flag.SKIP_INDEXING);

                   

                  and then you use the noIndexingCache to do all your put/putAlls. At the end you can fire off the Mass Indexer on the server, either through the 'mass-reindex' CLI operation, through JMX  or through the console. If you want to do it from the HotRod client, you could deploy a server task to do that.

                   

                  Also, you should not use the "ram" directory provider as it provides really poor performance. Either use the infinispan one or the lucene filesystem-based one.

                  1 of 1 people found this helpful
                  • 6. Re: Slow DataGrid server insert
                    Vincent Ragot Newbie

                    Hello,

                     

                    Can you explain me how to index my data with Mass Indexer ?

                    I found some tutoriels, but I don't know how to insert Mass Indexer into my code.

                     

                    Thanks