We currently use Indri for our evaluation frameworkprocess. One of our current goals The goal of NDS-867 is to implement a similar evaluation framework for based on ElasticSearch.
Requirements
With Indri (and associated related tools) we can do the following:
- Using a single index, dynamically change retrieval models and parameters through IndriRunQuery
- Indexing Re-indexing is only required if applying stopping or stemming (for major transformations such as stopping, stemming, or field storage).
- IndriRunQuery allows easy changing of retrieval model and outputing to TREC formatTREC format support support TREC formatted output for simple evaluation via trec_eval
- New retrieval models can be added by hacking Indri or the ir-utils framework.
- We have processes for multi-threaded and soon distributed parameter sweeps and cross-validation.
What we don't have under this framework:
- A pattern to support structured retrieval (ElasticSearch is JSON-native, so we'd need to have a JSON-based indexing strategy in Indri – which it does support)
- ElasticSearch/Luceneisms – specific model implementations
- Snowball stemmer support (Indri uses Porter and Krovetz). That said we can always pre-stem with Snowball for evaluation.
ElasticSearch Woes
Unfortunately, the ElasticSearch similarity is set for an index at indexing timecreation. This means that, to evaluate a particular parameter combination, we'd need to re-index the complete collection for each combination. This is likely prohibitive. There are changes proposed to later versions of ElasticSearch. But it seems that we might want to stick with Indri for now.
- Similarity should accept dynamic settings when possible https://github.com/elastic/elasticsearch/issues/20339
What options do we have:
- Evaluate with Indri, implement with ElasticSearch: One approach would be to continue to use Indri as the evaluation framework and translate results to implementation decisions in ElasticSearch. This isn't ideal, but seems like it might be the best
- Evaluate with Lucene, implement with ElasticSearch: Since ES is Lucene-based, we could develop an evaluation framework around Lucene. ir-tools already has some of the ingredients. This would allow us to work within some of the constraints of ElasticSearch (i.e., specific model implementations).