Baselines
The following baselines were run using the combined, short topics on the biocaddie_all index using the baselines/ scripts to sweep parameter combinations and the mkeval.sh script to perform LOOCV for each desired metric.
| Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
|---|---|---|---|---|---|---|---|---|
| TFIDF | 0.2781 | 0.5324 | 0.5405 | 0.4593 | 0.4024 | 0.4369 | Sweep b and k1 | 5/17/17 |
| Okapi | 0.2894 | 0.5629+ | 0.5643 | 0.4927+ | 0.4029 | 0.4516 | Sweep b, k1, k3 | 5/17/17 |
| QL (JM) | 0.2661 | 0.5311 | 0.5381 | 0.4853 | 0.3852 | 0.437 | Sweep lambda | 5/17/17 |
| QL (Dir) | 0.2976+ | 0.5656+ | 0.5643 | 0.5004 | 0.4119 | 0.457 | Sweep mu | 5/17/17 |
| QL (TS) | 0.2931+ | 0.5653+ | 0.5714 | 0.4827 | 0.4024 | 0.4507 | Sweep mu and lambda | 5/17/17 |
+ indicates significant improvement over TFIDF baselines (p < 0.05)
Feedback/SDM models
Green rows are runs with Garrick's framework, all others are with Craig's framework.
| Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
|---|---|---|---|---|---|---|---|---|
| QL (Dir) | 0.2976+ | 0.5656+ | 0.5643 | 0.5004 | 0.4119 | 0.457 |
| 5/15/17 |
| QL (Dir) | 0.2853 | 0.5304 | 0.5667 | 0.5054 | 0.3986 | 0.4541 | Sweep mu={50, 250, 500, 1000, 2500, 5000, 10000} | |
| RM3 | 0.2846 | 0.5381 | 0.5738 | 0.5054 | 0.4157 | 0.4710 | Sweep mu, fbDocs, fbTerms, and lambda | |
| RM3 | 0.2982 | 0.5381 | 0.5452 | 0.5152 | 0.4038 | 0.4768 | Sweep mu={100, 250, 500, 750, 1000, 2000, 3000}, fbDocs={10, 25, 50, 75, 100}, fbTerms={10, 25, 50, 75, 100}, and lambda=[0.0, 1.0] | |
| RM3 (stopped) | 0.3023 | 0.5471 | 0.5476 | 0.5064 | 0.3886 | 0.463 | Sweep mu, fbDocs, fbTerms, and lambda | |
| RM3 (stopped) | 0.3001 | 0.5423 | 0.5833 | 0.5150 | 0.3938 | 0.4534 | Sweep mu={100, 250, 500, 750, 1000, 2000, 3000}, fbDocs={10, 25, 50, 75, 100}, fbTerms={10, 25, 50, 75, 100}, and lambda=[0.0, 1.0] | |
| Pubmed | 0.2945 | 0.5998+ | 0.5190 | 0.4759 | 0.4190 | 0.4607 | Use mu=2500 for pubmed query sweeping fbDocs, fbTerms, and lambda. Sweep mu for final retrieval. | 5/15/17 |
| Wikipedia | 0.3051 | 0.5894+ | 0.5619 | 0.4821 | 0.3948 | 0.4613 | Use mu=2500 for wikipedia query sweeping fbDocs, fbTerms, and lambda. Sweep mu for final retrieval. | 5/15/17 |
| SDM | 0.2874 | 0.5558 | 0.4833 | 0.4706 | 0.4019 | 0.4559 | Sweep mu, w1, w2, w3 | 5/4/17 |
| FDM | 0.293 | 0.5594+ | 0.5500 | 0.4960 | 0.3933 | 0.4560 | Sweep mu, w1, w2, w3 | 5/8/17 |
| OKAPI Exp | 0.3011 | 0.5106 | 0.5071 | 0.4552 | 0.3871 | 0.4375 | OKAPI expansion assuming k1=1 and k3=1, for now |
+ indicates significant improvement over QL (Dir) baselines (p < 0.05)
| Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
|---|---|---|---|---|---|---|---|---|
| Dir (Krovetz) | 0.2959+ | 0.5443+ | 0.5905+ | 0.5150 | 0.4124+ | 0.4655 | + means significant improvement of Dir (unstemmed) | 5/15/17 |
| RM3 (krovetz) | 0.2858 | 0.5592 | 0.5643 | 0.4999 | 0.3952 | 0.4605 | ||
| Pubmed (Krovetz) | 0.3285+ | 0.6008+ | 0.5667 | 0.4821 | 0.4333+ | 0.4627 | + means improvement of Dir (krovetz) |