Page History

...

The following plots illustrate the effect of varying the Dirichlet mu parameter (a) and RM3 fbOrigWeight parameter for RM3 (b), PubMed RM3 (c), and Wikipedia RM(3).  For the RM3 models, all other parameters are fixed at their cross-validated values. The X axis shows the BioCADDIE topics, the Y-axis is the difference in NDCG from the cross-validated QL/Dirichlet baseline. The boxplots represent one point per parameter value – blue for low and red for high values.  The green dots are the cross-validated fbOrigWeight values for the RM models.

NDCG@1000

Looking at a few examples:

• In plot (a) we see that NDCG decreases for topic T10 as mu increases, but for T13 varying mu has little effect.
• In plot (b) we see that decreasing the fbOrigWeight (and therefore increasing the effect of the expansion terms) has a positive effect for topics T11 and T13, but a negative effect for topic T7.
• In plot (c) we see that PubMed expansion has a negative effect for T3 as compared to RM3 expansion.
• Similarly, in plot (d) we see that Wikipedia expansion in general has a negative effect compared to RM3 or PubMed RM3 expansion.

NDCG@20

Looking instead a the higher ranks, we see a much more muted effect of expansion with the BioCADDIE collection. As above, expansion appears to be effective for a small number of test queries (< 0.5)

In each of these cases, the fbOrigWeight that controls the mixing of the original and feedback query is relatively fixed – learned on average from the training queries. We can explore whether we can reliably predict when to apply one model or another or to predict the fbOrigWeight mixing parameter via query performance prediction methods.

...