6 Replies Latest reply on Feb 15, 2017 5:48 AM by vincent_ragot

Slow DataGrid server insert

alexis.bourdeau Jan 31, 2017 8:09 AM

Hello,

I have developped a java client to insert into a datagrid server 300 000 Payments objects.

I have one node (standalone.xml)

It takes around 10 min to absorb. This is quite long.
the code is the following :

FileManager fm = new FileManager();

List<Paiement> paiements = fm.readFile();

if ((paiements == null) || (paiements.size() == 0))

System.out.println("nothing to print");

else

{ // from here, it takes 10 minutes

for (Paiement paie : paiements) {

remoteCache.putAsync(paie.getId(), paie);

}

How can I decrease the time to insert this volume of object in the grid ?

Thanks a lot !

1. Re: Slow DataGrid server insert

nadirx Jan 31, 2017 8:32 AM (in response to alexis.bourdeau)

You should batch things and use the bulk methods (putAll) as they are more efficient.
1 of 1 people found this helpful
Actions
2. Re: Slow DataGrid server insert

vincent_ragot Jan 31, 2017 8:36 AM (in response to nadirx)

Hello,
Thanks for your response.
It seem interesting, I will try this.
1 of 1 people found this helpful
Actions
3. Re: Slow DataGrid server insert

vincent_ragot Jan 31, 2017 9:50 AM (in response to nadirx)

I'm worked on the same project, and I tested the putAll operation with 1 element into the map.

But it throws an exception:

org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=7 returned server error (status=0x86): org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 20 seconds for key [B0x2800 and requestor CommandUUID{address=mind7-virtual-machine, id=70}. Lock is held by CommandUUID{address=mind7-virtual-machine, id=11}
    at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)
    at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:132)
    at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:118)
    at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
    at org.infinispan.client.hotrod.impl.operations.PutAllOperation.executeOperation(PutAllOperation.java:56)
    at org.infinispan.client.hotrod.impl.operations.PutAllOperation.executeOperation(PutAllOperation.java:25)
    at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:54)
    at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putAll(RemoteCacheImpl.java:224)
    at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putAll(RemoteCacheSupport.java:49)
    at PaiementManager.addPaiementFromFile(PaiementManager.java:151)
    at Main.main(Main.java:43)

Have you an Idea ?
Actions
4. Re: Slow DataGrid server insert

alexis.bourdeau Feb 1, 2017 10:41 AM (in response to alexis.bourdeau)

Hello,

we found the bottleneck.
We got in out cache this set :


                    <indexing index="ALL">
                        <property name="default.directory_provider">ram</property>
                    </indexing>

removing this deccrease greatly the time necessary to insert data.

We have to investigate this.
Actions
5. Re: Slow DataGrid server insert

nadirx Feb 1, 2017 11:08 AM (in response to alexis.bourdeau)

Aha. Indexing does indeed slow things down.
A couple of things: for bulk loading you can add the SKIP_INDEXING flag:

RemoteCache noIndexingCache = cache.withFlags(Flag.SKIP_INDEXING);

and then you use the noIndexingCache to do all your put/putAlls. At the end you can fire off the Mass Indexer on the server, either through the 'mass-reindex' CLI operation, through JMX or through the console. If you want to do it from the HotRod client, you could deploy a server task to do that.

Also, you should not use the "ram" directory provider as it provides really poor performance. Either use the infinispan one or the lucene filesystem-based one.
1 of 1 people found this helpful
Actions
6. Re: Slow DataGrid server insert

vincent_ragot Feb 15, 2017 5:48 AM (in response to nadirx)

Hello,

Can you explain me how to index my data with Mass Indexer ?
I found some tutoriels, but I don't know how to insert Mass Indexer into my code.

Thanks
Actions

Go to original post