...
Results when using the combined, orig topics on the biocaddie_all index using the baselines/ scripts to sweep parameter combinations and the mkeval.sh script to perform LOOCV for each desired metric.
| Model | MAP | p-value | NDCG | p-value | P@20 | p-value | NDCG@20 | p-value | P@100 | p-value | NDCG@100 | p-value | Notes | Date |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TFIDF | 0.1661 | - | 0.3749 | - | 0.395 | - | 0.3234 | - | 0.2505 | - | 0.2962 | - | Sweep b and k1 | 5/16/17 |
| Okapi | 0.1348- | 0.9938 | 0.3279- | 0.9935 | 0.3225- | 0.9626 | 0.2558- | 0.9766 | 0.212 | 0.9482 | 0.2547- | 0.9717 | Sweep b, k1, k3 | 5/16/17 |
| QL (JM) | 0.1283- | 0.98 | 0.3388 | 0.7939 | 0.2825 | 0.9267 | 0.2079- | 0.9628 | 0.23 | 0.7278 | 0.2508 | 0.891 | Sweep lambda | 5/16/17 |
| QL (Dir) | 0.1447 | 0.9051 | 0.3689 | 0.5642 | 0.285- | 0.9688 | 0.2324- | 0.9676 | 0.2415 | 0.6319 | 0.2666 | 0.8402 | Sweep mu | 5/16/17 |
| QL (TS) | 0.1606 | 0.632 | 0.3724 | 0.5259 | 0.36 | 0.7977 | 0.304 | 0.738 | 0.2475 | 0.5424 | 0.2991 | 0.4605 | Sweep mu and lambda | 5/ |
| 16/17 | ||||||||||||||
| RM3 | 0.1795 | 0.3093 | 0.373 | 0.5156 | 0.4025 | 0.4428 | 0.3234 | 0.8438 | 0.247 | 0.5367 | 0.2913 | 0.5462 | Sweep mu, fbDocs, fbTerms, and lambda | 5/22/17 |
| SDM | 0.1575 | 0.7099 | 0.3962 | 0.3021 | 0.42 | 0.3554 | 0.3446 | 0.2989 | 0.2695 | 0.2764 | 0.3045 | 0.3995 | Sweep mu, w1, w2, w3 | 5/22/17 |
***No improvement over TFIDF baselines when using original queries.
- indicates significant decrement in performance over TFIDF baselines (p>0.95)
Feedback/SDM models
Green rows are runs with Garrick's framework, all others are with Craig's framework.
...