Versions Compared

Old Version 18

changes.mady.by.user Craig Willis

Saved on

compared with

New Version Current

changes.mady.by.user Garrick Sherman

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following baselines were run using the combined, short topics on the biocaddie_all index using the baselines/ scripts to sweep parameter combinations and the mkeval.sh script to perform LOOCV for each desired metric.

ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
TFIDF0.
2524
27810.
459
53240.
4643
54050.
4144
45930.
3638
40240.
3973
4369Sweep b and k15/17/17
Okapi0.
2548
28940.
466
5629+0.
5+
56430.
4414
4927+0.
3419
40290.
3981
4516Sweep b, k1, k35/17/17
QL (JM)0.
2573
26610.
5159
53110.
5524+
53810.
4886+
48530.
3752
38520.
4249
437Sweep lambda5/17/17
QL (Dir)0.
2837
2976+0.
5306
5656+0.
5667+
56430.
5054+
50040.
3981
41190.
4541+
457

Sweep mu

5/17/17
QL (TS)0.
2794
2931+0.
5391
5653+0.
531
57140.
4702
48270.
391
40240.
4455
4507Sweep mu and lambda5/17/17

+ indicates significant improvement over TFIDF baselines (p < 0.05)

Results when using the combined, orig topics on the biocaddie_all index using the baselines/ scripts to sweep parameter combinations and the mkeval.sh script to perform LOOCV for each desired metric.

ModelMAPp-valueNDCGp-valueP@20p-valueNDCG@20p-valueP@100p-valueNDCG@100p-valueNotesDate
TFIDF0.1661-0.3749-0.395-0.3234-0.2505-0.2962-Sweep b and k15/16/17
Okapi0.1348-0.99380.3279-0.99350.3225-0.96260.2558-0.97660.2120.94820.2547-0.9717Sweep b, k1, k35/16/17
QL (JM)0.1283-0.980.33880.79390.28250.92670.2079-0.96280.230.72780.25080.891Sweep lambda5/16/17
QL (Dir)0.14470.90510.36890.56420.285-0.96880.2324-0.96760.24150.63190.26660.8402

Sweep mu

5/16/17
QL (TS)0.16060.6320.37240.52590.360.79770.3040.7380.24750.54240.29910.4605Sweep mu and lambda5/16/17
RM30.17950.30930.3730.51560.40250.44280.32340.84380.2470.53670.29130.5462Sweep mu, fbDocs, fbTerms, and lambda5/22/17
SDM0.15750.70990.39620.3021 0.420.35540.34460.29890.26950.27640.30450.3995Sweep mu, w1, w2, w35/22/17

***No improvement over TFIDF baselines when using original queries.
- indicates significant decrement in performance over TFIDF baselines (p>0.95)  


Feedback/SDM models

Green rows are runs with Garrick's framework, all others are with Craig's framework.

ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
QL (Dir)0.2976+0.5656+0.56430.50040.41190.457

 

5/15/17
QL (Dir)0.
2837
28530.
5306
53040.56670.50540.
3981
39860.4541
 
Sweep mu5/15/17
RM30.28460.53810.57380.50540.41570.4710Sweep mu, fbDocs, fbTerms, and lambda5/15/17
RM30.28460.53950.56190.50720.41570.4710Sweep mu, fbDocs, fbTerms, and lambda5/16/17
RM3 (stopped)0.29480.58340.55710.50040.38050.4599Sweep mu, fbDocs, fbTerms, and lambda5/17/17
RM3 (stopped)0.30230.
5471
54830.
5476
57380.
5064
51180.
3886
39050.
463
4626Sweep mu, fbDocs, fbTerms, and lambda5/16/17
Pubmed0.
3076+
29450.
5853
5998+0.
5381
51900.
4855
47590.
4248
41900.
4501
4607Use mu=2500 for pubmed query sweeping fbDocs, fbTerms, and lambda. Sweep mu for final retrieval.5/15/17
Wikipedia0.
2947
30510.
5956
5894+0.
5738
56190.
4956
48210.
4062
39480.
4487
4613Use mu=2500 for wikipedia query sweeping fbDocs, fbTerms, and lambda. Sweep mu for final retrieval.5/15/17
SDM0.28740.55580.48330.47060.40190.4559Sweep mu, w1, w2, w35/16/17
FDM0.2930.5594+0.55000.49600.39330.4560Sweep mu, w1, w2, w35/8/17
OKAPI Exp0.31890.57450.58570.48940.42380.4783OKAPI expansion5/17/17

+ indicates significant improvement over QL (Dir) baselines (p < 0.05)

Document expansion models

ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
QL (Dir)0.28530.53040.56670.50540.39860.4541Sweep mu5/15/17
Pubmed QL0.24670.54160.41470.35000.35060.3608Sweep alpha5/22/17
Pubmed RM30.29490.5806+0.56430.47620.38190.4431Sweep mu, fbDocs, fbTerms, alpha, and lambda5/17/17
Wikipedia QL0.23910.52670.45000.37270.35350.3610Sweep alpha5/22/17
Wikipedia RM30.28000.5795+0.55240.47120.37240.4254Sweep mu, fbDocs, fbTerms, alpha, and lambda5/17/17
Pubmed+Wikipedia QL0.24670.54160.41470.35000.34590.3608Sweep alpha5/22/17
Pubmed+Wikipedia RM30.29230.5725+0.56670.46000.39330.4378Sweep mu, fbDocs, fbTerms, alpha, and lambda5/17/17


ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
Dir (Krovetz)0.
2944
2959+0.5443+0.5905+0.51500.
4033
4124+0.
4601
4655+ means significant improvement of Dir (unstemmed)5/15/17
RM3 (krovetz)0.30290.57340.56430.47140.40330.4753
5/17/17
Pubmed (Krovetz)0.3285+0.6008+0.56670.48210.4333+0.4627+ means improvement of Dir (krovetz)