...
- Clarity (Cronen-Townshend, 2002)
- Drift ( Shtok, Kurland, Carmel, 2009)
- Deviation (Perez-Iglesia and Araujo, 2010)
- Absolute/relative divergence (Lv & Zhai)
Preliminary results
NDCG
The following predictors were found to be significant from (overfitted) analysis of test data. These are predictors for fbOrigWeight
...
Using the "varSCQ + deviation" model to estimate fbOrigWeight for the held-out query using leave-one-query-out crossvalidation, we see NDCG=0.59 compared to NDCG=0.56. This is a significant difference with 1-tailed t-test and p < 0.01
NDCG@20
NDCG@20 is a bit of a challenge. Only one standard predictor, devation, was found to be a significant predictor:
- deviation (R^20.31, p<0.01)
With two predictors, again deviation and varSCQ provide the best fit:
- varSCQ ~ deviation (R^2 0.43, p<0.01)
Using the two-predictor model to estimate fbOrigWeight, we see NDCG@20=0.53 versus 0.52. There is no significant difference between these two.
References
Carmel, D., & Yom-Tov, E. (2010). Estimating the Query Difficulty for Information Retrieval. Synthesis Lectures on Information Concepts, Retrieval, and Services, 2(1), 1–89. http://doi.org/10.2200/S00235ED1V01Y201004ICR015
...