Query characterization and performance prediction

Design notes for NDS-919 - Getting issue details... STATUS  and  NDS-920 - Getting issue details... STATUS .

Query characterization

From the BioCADDIE Results, we can see that the PubMed and Wikipedia expansion models provide some improvement, but not at the higher ranks. As is often the case with expansion, inspection of individual queries shows that, while some queries benefit from expansion, others do not. 

The following plots illustrate the effect of varying the Dirichlet mu parameter (a) and RM3 fbOrigWeight parameter for RM3 (b), PubMed RM3 (c), and Wikipedia RM(3).  For the RM3 models, all other parameters are fixed at their cross-validated values. The X axis shows the BioCADDIE topics, the Y-axis is the difference in NDCG from the cross-validated QL/Dirichlet baseline. The boxplots represent one point per parameter value – blue for low and red for high values.  The green dots are the cross-validated fbOrigWeight values for the RM models.

Looking at a few examples:

• In plot (a) we see that NDCG decreases for topic T10 as mu increases, but for T13 varying mu has little effect.
• In plot (b) we see that decreasing the fbOrigWeight (and therefore increasing the effect of the expansion terms) has a positive effect for topics T11 and T13, but a negative effect for topic T7.
• In plot (c) we see that PubMed expansion has a negative effect for T3 as compared to RM3 expansion.
• Similarly, in plot (d) we see that Wikipedia expansion in general has a negative effect compared to RM3 or PubMed RM3 expansion.

In each of these cases, the fbOrigWeight that controls the mixing of the original and feedback query is relatively fixed – learned on average from the training queries. We can explore whether we can reliably predict when to apply one model or another or to predict the fbOrigWeight mixing parameter via query performance prediction methods.

Query performance prediction

A central goal of query performance prediction or query difficulty estimation is to identify features, both pre- and post-retrieval, that can be used to predict the performance of a query. This is generally done by predicting average precision. The predicted average precision can be used, for example, to select between two different models.  Unfortunately, there are no comprehensive reviews of predictor effectiveness.

For BioCADDIE, we are focused on expansion models and therefore are primarily concerned with adaptive feedback. Lv and Zhai's (2009) approach seems to be the most applicable – estimating the feedback mixing parameter per-query.  This will require the following:

• A framework for implementing baseline and custom predictors (ir-utils or otherwise)
• Ability to generate a set of pre- and post-retrieval predictor values for each query for multiple collections. This will output a matrix of queries to predictors.
• Calculate correlation (Pearson and Spearman) between the predictor and a given metric or parameter (eg., RM3 lambda)
• Ability to select features (manually or automatically) and to construct a predictive model (i.e, regression) using one or more predictors.
• Evaluate the predictive model via cross validation.

Adaptive feedback

One approach explored by Lv and Zhai (2009) is to learn a model to predict the expansion mixing weight. They found six features to be predictive of the feedback weight in a linear model:

• Divergence (QFBDiv_A): KL-divergence of query and feedback documents
• Query clarity (QEnt_R1): Relative entropy of the query compared to the collection
• Log query clarity (QEnt_R3): Log of the relative entropy of the query compared to the collection
• Feedback radius (FBRadius): average divergence between each document and the centroid of the feedback documents.
• Exponentiated feedback clarity (FBEnt_R2): Exponentiated relative entropy of feedback documents to the collection
• Topic model clarity (FBEnt_R3): Relative entropy of the feedback document topic model to the collection.

All of these predictors are post-retrieval predictors, some with significant overhead.

References

Lv, Y., & Zhai, C. (2009). Adaptive Relevance Feedback in Information Retrieval. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (pp. 255–264). New York, NY, USA: ACM. http://doi.org/10.1145/1645953.1645988