How to remove all the duplicate results in Hibernate Search with Infinispan?

[abey4315266]

I'm using Infinispan  6.0.2 with Hibernate Search 4.4.0.

In the begining, after I execute a query like

CacheQuery cq = SearchManager.getQuery(query,Hibernate.class).projection("id");

I use the cq.list() to get "id" (not the unique id).

But now the number of results reaches 300.000, because of the designing fo DB(cant change), the duplicate id is almost 29,000. (The value for each id is correct)

I wrote this to get "id":

for(int i=0;i<listObject.size();i++)
{
   Object[] rdf = (Object[])listObject.get(i);
   if(!result.contains((String) rdf[0])) 
   result.add((String) rdf[0]);
}

But even if I parallel this part, it also takes a long time.

I saw there is a method called setResultTransformer(Criteria.DISTINCT_ROOT_ENTITY) in Hibernate, but I dont know how to do it in Infinispan.

Or is there a method in Infinispan doing the same thing like 'DISTINCT' of SQL??

[rvansa]

I can't answer your question positively (it really seems it's not available), but are you sure you want to get all those object in one huge list? It's not scaling, and you're asking for OOME. Have you considered using iterator with lazy fetch options instead?

But anyway, if you think you'll always be able to hold all results in-memory, you can just use HashSet and ignore the duplicities, can't you?

[sannegrinovero]

Hi Stephan,

no there is no function like "DISTINCT" as it is not a relational database (The Hibernate Criteria API need a database, in this case neither Infinispan nor a Lucene index includes de-duplication functionality).

In your case I would have the query sorted on your "id" field, so you can iterate on the list to skip results in a more efficient way: you'd only need to verify if the next result is the same as the previous one. At least that's more efficient than using the contains method.

In a future version of Infinispan we hope to provide an integration of Map/Reduce on query results, so you wouldn't have to fetch the results first but will be able to run a "de-duplication job" on the results before fetching them.

[abey4315266]

Thanks for the reponses. They are really helpful.