Slow DataGrid server insert

[alexis.bourdeau]

Hello,

I have developped a java client to insert into a datagrid server 300 000 Payments objects.

I have one node (standalone.xml)

It takes around 10 min to absorb. This is quite long.
the code is the following :

FileManager fm = new FileManager();

        List<Paiement> paiements = fm.readFile();

        if ((paiements == null) || (paiements.size() == 0))

            System.out.println("nothing to print");

        else

        { // from here, it takes 10 minutes

            for (Paiement paie : paiements) {

                remoteCache.putAsync(paie.getId(), paie);

            }

        }

How can I decrease the time to insert this volume of object in the grid ?

Thanks a lot !

[nadirx]

You should batch things and use the bulk methods (putAll) as they are more efficient.

[vincent_ragot]

Hello,

Thanks for your response.

It seem interesting, I will try this.

[vincent_ragot]

I'm worked on the same project, and I tested the putAll operation with 1 element into the map.

But it throws an exception:

org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for messageId=7 returned server error (status=0x86): org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 20 seconds for key [B0x2800 and requestor CommandUUID{address=mind7-virtual-machine, id=70}. Lock is held by CommandUUID{address=mind7-virtual-machine, id=11}

    at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:343)

    at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:132)

    at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:118)

    at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)

    at org.infinispan.client.hotrod.impl.operations.PutAllOperation.executeOperation(PutAllOperation.java:56)

    at org.infinispan.client.hotrod.impl.operations.PutAllOperation.executeOperation(PutAllOperation.java:25)

    at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:54)

    at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putAll(RemoteCacheImpl.java:224)

    at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putAll(RemoteCacheSupport.java:49)

    at PaiementManager.addPaiementFromFile(PaiementManager.java:151)

    at Main.main(Main.java:43)

Have you an Idea ?

[alexis.bourdeau]

Hello,

we found the bottleneck.

We got in out cache this set :

<!-- Enable indexing using the RAM Lucene directory provider -->

                    <indexing index="ALL">

                        <property name="default.directory_provider">ram</property>

                    </indexing>

removing this deccrease greatly the time necessary to insert data.

We have to investigate this.

[nadirx]

Aha. Indexing does indeed slow things down.

A couple of things: for bulk loading you can add the SKIP_INDEXING flag:

RemoteCache noIndexingCache = cache.withFlags(Flag.SKIP_INDEXING);

and then you use the noIndexingCache to do all your put/putAlls. At the end you can fire off the Mass Indexer on the server, either through the 'mass-reindex' CLI operation, through JMX  or through the console. If you want to do it from the HotRod client, you could deploy a server task to do that.

Also, you should not use the "ram" directory provider as it provides really poor performance. Either use the infinispan one or the lucene filesystem-based one.

[vincent_ragot]

Hello,

Can you explain me how to index my data with Mass Indexer ?

I found some tutoriels, but I don't know how to insert Mass Indexer into my code.

Thanks