You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

We currently use Indri for our evaluation process. The goal of NDS-867 is to implement a similar evaluation framework based on ElasticSearch.

Requirements

With Indri (and related tools) we can do the following:

  • Using a single index, dynamically change retrieval models and parameters through IndriRunQuery
  • Re-indexing is only required for major transformations such as stopping, stemming, or field storage.
  • IndriRunQuery support TREC formatted output for simple evaluation via trec_eval
  • New retrieval models can be added by hacking Indri or the ir-utils framework.
  • We have processes for multi-threaded and soon distributed parameter sweeps and cross-validation.

What we don't have under this framework:

  • A pattern to support structured retrieval (ElasticSearch is JSON-native, so we'd need to have a JSON-based indexing strategy in Indri – which it does support)
  • ElasticSearch/Luceneisms – specific model implementations
  • Snowball stemmer support (Indri uses Porter and Krovetz). That said we can always pre-stem with Snowball for evaluation.

ElasticSearch Woes

Unfortunately, the ElasticSearch similarity is set for an index at creation.  This means that, to evaluate a particular parameter combination, we'd need to re-index the complete collection for each combination. This is likely prohibitive. There are changes proposed to later versions of ElasticSearch.  But it seems that we might want to stick with Indri for now.

What options do we have:

  • Evaluate with Indri, implement with ElasticSearch: One approach would be to continue to use Indri as the evaluation framework and translate results to implementation decisions in ElasticSearch.  This isn't ideal, but seems like it might be the best
  • Evaluate with Lucene, implement with ElasticSearch: Since ES is Lucene-based, we could develop an evaluation framework around Lucene. ir-tools already has some of the ingredients. This would allow us to work within some of the constraints of ElasticSearch (i.e., specific model implementations).


  • No labels