We currently use Indri for our evaluation framework. One of our current goals is to implement a similar evaluation framework for ElasticSearch.
Requirements
With Indri and associated tools we can do the following:
- Using a single index, dynamically change retrieval models and parameters
- Indexing is only required if applying stopping or stemming (or field storage)
- IndriRunQuery allows easy changing of retrieval model and outputing to TREC format
- TREC format support simple evaluation via trec_eval.
ElasticSearch Woes
Unfortunately, the ElasticSearch similarity is set for an index at indexing time. This means that, to evaluate a particular parameter combination, we'd need to re-index the complete collection. There are changes proposed to later versions of ElasticSearch. But it seems that we might want to stick with Indri for now.
- Similarity should accept dynamic settings when possible https://github.com/elastic/elasticsearch/issues/20339