Versions Compared

Old Version 37

changes.mady.by.user Thuong Phan

Saved on

compared with

New Version Current

changes.mady.by.user Garrick Sherman

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Results when using the combined, orig topics on the biocaddie_all index using the baselines/ scripts to sweep parameter combinations and the mkeval.sh script to perform LOOCV for each desired metric.

ModelMAPp-valueNDCGp-valueP@20p-valueNDCG@20p-valueP@100p-valueNDCG@100p-valueNotesDate
TFIDF0.1661-0.3749-0.395-0.3234-0.2505-0.2962-Sweep b and k15/
17
16/17
Okapi0.1348-0.99380.3279-0.99350.3225-0.96260.
2894
2558-0.
5629+
97660.
5643
2120.
4927+
94820.
4029
2547-0.
4516
9717Sweep b, k1, k35/
17
16/17
QL (JM)0.
2661
1283-0.980.33880.79390.28250.92670.2079-0.
5311
96280.
5381
230.
4853
72780.
3852
25080.
437
891Sweep lambda5/
17
16/17
QL (Dir)0.14470.90510.36890.
2976+
56420.285-0.96880.2324-0.
5656+
96760.
5643
24150.
5004
63190.
4119
26660.
457
8402

Sweep mu

5/
17
16/17
QL (TS)0.16060.6320.37240.52590.360.79770.3040.7380.24750.54240.29910.4605Sweep mu and lambda5/16/17
RM30.17950.30930.3730.51560.40250.44280.
2931+
32340.
5653+
84380.
5714
2470.
4827
53670.
4024
29130.
4507
5462Sweep mu, fbDocs, fbTerms, and lambda5/22/17
SDM0.15750.70990.39620.3021 0.420.35540.34460.29890.26950.27640.30450.3995Sweep mu, w1, w2, w35/22/17

***No improvement over TFIDF baselines when using original queries.
- indicates significant decrement in performance over TFIDF baselines (p>0.95)  


Feedback/SDM models

Green rows are runs with Garrick's framework, all others are with Craig's framework.

ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
QL (Dir)0.2976+0.5656+0.56430.50040.41190.457

 

5/15/17
QL (Dir)0.28530.53040.56670.50540.39860.4541Sweep mu5/15/17
RM30.28460.53810.57380.50540.41570.4710Sweep mu, fbDocs, fbTerms, and lambda5/15/17
RM30.28460.53950.56190.50720.41570.4710Sweep mu, fbDocs, fbTerms, and lambda5/16/17
RM3 (stopped)0.29480.58340.55710.50040.38050.4599Sweep mu, fbDocs, fbTerms, and lambda5/17/17
RM3 (stopped)0.30230.54830.57380.51180.39050.4626Sweep mu, fbDocs, fbTerms, and lambda5/16/17
Pubmed0.29450.5998+0.51900.47590.41900.4607Use mu=2500 for pubmed query sweeping fbDocs, fbTerms, and lambda. Sweep mu for final retrieval.5/15/17
Wikipedia0.30510.5894+0.56190.48210.39480.4613Use mu=2500 for wikipedia query sweeping fbDocs, fbTerms, and lambda. Sweep mu for final retrieval.5/15/17
SDM0.28740.55580.48330.47060.40190.4559Sweep mu, w1, w2, w35/16/17
FDM0.2930.5594+0.55000.49600.39330.4560Sweep mu, w1, w2, w35/8/17
OKAPI Exp0.325031890.584957450.58570.48940.42380.4783OKAPI expansion assuming k1=1 and k3=1, for now5/17/17

+ indicates significant improvement over QL (Dir) baselines (p < 0.05)

...

ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
QL (Dir)0.28530.53040.56670.50540.39860.4541Sweep mu5/15/17
Pubmed QL0.24670.54160.41470.35000.35060.3608Sweep alpha5/22/17
Pubmed RM30.29490.5806+0.56430.47620.38190.4431Sweep mu, fbDocs, fbTerms, alpha, and lambda5/17/17
Wikipedia QL0.23910.52670.45000.37270.35350.3610Sweep alpha5/22/17
Wikipedia RM30.28000.5795+0.55240.47120.37240.4254Sweep mu, fbDocs, fbTerms, alpha, and lambda5/17/17
Pubmed+Wikipedia QL0.24670.54160.41470.35000.34590.3608Sweep alpha5/22/17
Pubmed+Wikipedia RM30.29230.5725+0.56670.46000.39330.4378Sweep mu, fbDocs, fbTerms, alpha, and lambda5/17/17

...