Query characterization and performance prediction

Design notes for NDS-919 - Getting issue details... STATUS  and  NDS-920 - Getting issue details... STATUS .

Query characterization

From the BioCADDIE Results, we can see that the PubMed and Wikipedia expansion models provide some improvement, but not at the higher ranks. As is often the case with expansion, inspection of individual queries shows that, while some queries benefit from expansion, others do not. 

The following plots illustrate the effect of varying the Dirichlet mu parameter (a) and RM3 fbOrigWeights for RM3 (b), PubMed RM3 (c), and Wikipedia RM(3).  For the RM3 models, all other parameters are fixed at the cross-validated value. The X axis shows the BioCADDIE topics, the Y-axis is the difference in NDCG from the cross-validated QL/Dirichlet baseline. The boxplots represent one point per parameter value – shared blue for low and red for high values.  The green dots are the cross-validated parameter value for the compared model.

Looking at a few examples:

• In plot (a) we see that NDCG decreases for topic T10 as mu increases, but for T13 varying mu has little effect.
• In plot (b) we see that decreasing the fbOrigWeight (and therefore increasing the effect of the expansion terms) has a positive effect for topics T11 and T13, but a negative effect for topic T7.
• In plot (c) we see that PubMed expansion has a negative effect for T3 as compared to RM3 expansion.
• Similarly, in plot (d) we see that Wikipedia expansion in general has a negative effect compared to RM3 or PubMed RM3 expansion.