Baselines
The following baselines were run using the test, short topics on the biocaddie_all index using the baselines/ scripts to sweep parameter combinations and the mkeval.sh script to perform LOOCV for each desired metric.
| Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
|---|---|---|---|---|---|---|---|---|
| TFIDF | 0.3349 | 0.5525 | 0.6433 | 0.5552 | 0.5007 | 0.4899 | Sweep b and k1 | 6/22/17 |
| Okapi | 0.342 | 0.5845 | 0.68 (p-value=0.0587) | 0.5597 (p-value=0.0553) | 0.5067 | 0.5143 | Sweep b, k1, k3 | 6/22/17 |
| QL (JM) | 0.2988 | 0.5242 | 0.6667 | 0.5829 | 0.4747 | 0.4974 | Sweep lambda | 6/22/17 |
| QL (Dir) | 0.3284 | 0.544 | 0.6833 | 0.5923 | 0.5013 | 0.5099 | Sweep mu | 6/22/17 |
| QL (TS) | 0.3261 | 0.545 | 0.6667 | 0.5549 | 0.4853 | 0.4906 | Sweep mu and lambda | 6/22/17 |
+ indicates significant improvement over TFIDF baselines (p < 0.05)
Okapi metric results are used from manual run instead of extracted data from combined eval files (due to mismatch data)
Feedback/SDM models
| Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
|---|---|---|---|---|---|---|---|---|
| TFIDF | 0.3349 | 0.5525 | 0.6433 | 0.5552 | 0.5007 | 0.4899 | Sweep b and k1 | 6/22/17 |
| RM3 | 0.3379 | 0.5717 | 0.6833 | 0.6031+ | 0.5113 | 0.5249+ | Sweep mu, fbDocs, fbTerms, and lambda | 6/22/17 |
| RM3 (stopped) | 0.3417 | 0.5827+ | 0.68 | 0.5807 | 0.506 | 0.5134 | Sweep mu, fbDocs, fbTerms, and lambda | 6/22/17 |
| Pubmed | 0.3534 | 0.6009+ | 0.6567 | 0.5732 | 0.5207 | 0.5074 | Use mu=2500 for pubmed query sweeping fbDocs, fbTerms, and lambda. Sweep mu for final retrieval. | 6/22/17 |
| Wikipedia | 0.3553 | 0.6029+ | 0.6767 | 0.5503 | 0.4893 | 0.4981 | Use mu=2500 for wikipedia query sweeping fbDocs, fbTerms, and lambda. Sweep mu for final retrieval. | 6/22/17 |
| SDM | 0.3372 | 0.5715 | 0.6067 | 0.5526 | 0.512 | 0.5067 | Sweep mu, w1, w2, w3 | 6/22/17 |
| FDM | 0.3418 | 0.5818 | 0.6767 | 0.5707 | 0.5047 | 0.5063 | Sweep mu, w1, w2, w3 | 6/22/17 |
| OKAPI Exp | 0.3725 | 0.6023 | 0.6633 | 0.6007 (p-value=0.0525) | 0.5413 | 0.5569+ | OKAPI expansion | 6/22/17 |
| Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
|---|---|---|---|---|---|---|---|---|
| Dir (Krovetz) | 0.3663+ | 0.6022+ | 0.7033+ | 0.6079 (p-value=0.0587) | 0.5173 | 0.5225+ | 6/22/17 | |
| RM3 (krovetz) | 0.3604 | 0.5809+ | 0.7167+ | 0.5788 | 0.4967 | 0.5368+ | 6/22/17 | |
| Pubmed (Krovetz) | 0.3841 | 0.6225+ | 0.7067+ | 0.5831 | 0.5353 | 0.5162 | 6/22/17 |
+ indicates significant improvement over TFIDF baselines (p < 0.05)
For verification of the results in above tables.