- Querying involves 2 types of operations:
- Storing data and getting it indexed
- Hot Rod is a platform independent protocol and so it needs to be able to index types that are universal (i.e. String UTF-8 byte array, numerics). It's expected that Infinispan Query module will be used in the backend whose underlying technology is Lucene. Currently Lucene supports indexing String and numeric data.
- The tricky thing about storing data is:
- On one side, stored value comes in a byte[], so it needs to be transformed into something that Lucene understands.
- Secondly, when data is stored, there are multiple flags or options that can be associated with the operation. For example, when a String is stored, you can decide whether the String should be indexed, or not...etc.
- Deciding all the format of all these options is probably one of the most labourious things, but Hibernate Search uses Apache Avro already to format some of this stuff, so Hot Rod could maybe piggyback on this format. This needs further investigation.
- Querying data:
- Normally, querying involves a 2 phase process:
- First, the query is executed and the keys that fulfill the query are returned.
- Take the keys and paralellise the retrieval of values associated w/ these keys.
- This might be a bit inefficient in some cases, i.e. the resultset is small
- In this case, feeding back the results from the same node where the query was executed might be more optiomal.
- Hot Rod should be able to decide what's the best course of action
- Normally, querying involves a 2 phase process:
- Storing data and getting it indexed
- Querying will need streaming at some level
- Netty provides with ChunkedInput and ChunkedWriteHandler interfaces that allows streaming to be implemented over a particular protocol.
- Based on some early investigation, ChunkedInput would need to be implemented specifically for querying to provide next resultset.
- Other notes:
- Ability to support compression with the client indicating what are the supported compression methods.
- It has been suggested that on the server side, Hot Rod could pass pre-processed strings to Hibernate Search / Infinispan Query, bypassing annotations.
- Taking in account that REST querying is something we'd want to develop in the future, try to build this up in such way that REST can reuse it.
- Also, check what REST offers from a querying perspective.
Comments