...
No Format |
---|
cd ~/biocaddie/baselines
./<model>.sh <topic> <collection> <year> | parallel -j 20 bash -c "{}" |
IndriRunQuery outputs for different baselines are stored at:
...
No Format |
---|
cd ~/biocaddie Rscript scripts/compare_treccds.R <collection> <from model> <to model> <topic> <year> |
Results (rm3 compared to be updatedtfidf baseline)
Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
---|---|---|---|---|---|---|---|---|
tfidf | 0.0957 | 0.3017 | 0.3 | 0.2353 | 0.1857 | 0.194 | Sweep b and k1 | 06/05/17 |
Okapi | 0.0552- | 0.1831- | 0.23- | 0.1824- | 0.128- | 0.1477- | Sweep b, k1, k3 | 06/05/17 |
QL (JM) | 0.0971 | 0.2901 | 0.3167 | 0.2708 | 0.1827 | 0.212 | Sweep lambda | 06/05/17 |
QL (Dir) | 0.1026 | 0.3002 | 0.3217 | 0.2705 | 0.1937 | 0.2197+ | Sweep mu | 06/05/17 |
QL (TS) | 0.0976 | 0.3001 | 0.295 | 0.2555 | 0.1857 | 0.2145 | Sweep mu and lambda | 06/05/17 |
RM3 | 0.156+ | 0.3614+ | 0.375+ | 0.2926+ | 0.236+ | 0.2595+ | Sweep mu, fbDocs, fbTerms, and lambda | 06/06/17 |
No Format |
---|
root@integration-1:~/biocaddie# Rscript scripts/compare_treccds.R combined tfidf dir orig 2015 [1] "map 0.0957 0.1026 p= 0.2837" [1] "ndcg 0.3017 0.3002 p= 0.5381" [1] "P_20 0.3 0.3217 p= 0.2352" [1] "ndcg_cut_20 0.2353 0.2705 p= 0.0834" [1] "P_100 0.1857 0.1937 p= 0.2639" [1] "ndcg_cut_100 0.194 0.2197 p= 0.0396" root@integration-1:~/biocaddie# Rscript scripts/compare_treccds.R combined tfidf jm orig 2015 [1] "map 0.0957 0.0971 p= 0.4594" [1] "ndcg 0.3017 0.2901 p= 0.7606" [1] "P_20 0.3 0.3167 p= 0.3204" [1] "ndcg_cut_20 0.2353 0.2708 p= 0.1038" [1] "P_100 0.1857 0.1827 p= 0.5918" [1] "ndcg_cut_100 0.194 0.212 p= 0.1283" root@integration-1:~/biocaddie# Rscript scripts/compare_treccds.R combined tfidf okapi orig 2015 [1] "map 0.0957 0.0552 p= 0.9997" [1] "ndcg 0.3017 0.1831 p= 1" [1] "P_20 0.3 0.23 p= 0.9979" [1] "ndcg_cut_20 0.2353 0.1824 p= 0.9991" [1] "P_100 0.1857 0.128 p= 0.9999" [1] "ndcg_cut_100 0.194 0.1477 p= 0.9993" root@integration-1:~/biocaddie# Rscript scripts/compare_treccds.R combined tfidf two orig 2015 [1] "map 0.0957 0.0976 p= 0.432" [1] "ndcg 0.3017 0.3001 p= 0.5467" [1] "P_20 0.3 0.295 p= 0.5667" [1] "ndcg_cut_20 0.2353 0.2555 p= 0.2152" [1] "P_100 0.1857 0.1857 p= 0.5" [1] "ndcg_cut_100 0.194 0.2145 p= 0.0652" root@integration-1:~/biocaddie# Rscript scripts/compare_treccds.R combined tfidf rm3 orig 2015 [1] "map 0.0957 0.156 p= 0.0042" [1] "ndcg 0.3017 0.3614 p= 0.005" [1] "P_20 0.3 0.375 p= 0.0185" [1] "ndcg_cut_20 0.2353 0.2926 p= 0.0348" [1] "P_100 0.1857 0.236 p= 0.0155" [1] "ndcg_cut_100 0.194 0.2595 p= 0.0052" |
2. TREC CDS 2016 collection (http://trec-cds.appspot.com/2016.html)
...