Versions Compared

Old Version 6

changes.mady.by.user Thuong Phan

Saved on

compared with

New Version Current

changes.mady.by.user Thuong Phan

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

    a) Lucene Run (lucene-output)

Using biocaddie_all indexes

No Format
cd ~/biocaddie
baselines/new/<model>-lucene.sh <topics> <subset> <col>| parallel -j 20 bash -c "{}"
baselines/new/<model>-lucene.sh <topics> <subset> <col> <year>| parallel -j 20 bash -c "{}"

         Eg: baselines/new/dir-lucene.sh short test biocaddie| parallel -j 20 bash -c "{}"

    b) Evaluation and Cross-validation (lucene-eval, loocv) 

No Format
cd ~/biocaddie
scripts/new/mkeval-lucene.sh <model> <topics> <subset> <col>
scripts/new/mkeval-lucene.sh <model> <topics> <subset> <col> <year>

     

         Eg: scripts/new/mkeval-lucene.sh dir short test biocaddie

    c) Compare models

        We have to input running method for comparison:

        0 - both from and to models are from Indri run

        1 - both from and to models are from Lucene run

        2 - from model is from Indri run, to model is from Lucene run

        3 - from model is from Lucene run, to model is from Indri run

No Format
cd ~/biocaddie
Rscript scripts/new/compare.R <subset> <from> <to> <topics> <col>
Rscript scripts/new/compare.R <subset> <from> <to> <topics> <col> <year>

        Eg: Rscript scripts/new/compare.R test tfidf dir short biocaddie

4. Results

...

0.3677

(p-value=0.0526)

...

Sweep mu

...

      baselines/new/tfidf-lucene.sh short test biocaddie| parallel -j 20 bash -c "{}"
     baselines/new/jm-lucene.sh short test biocaddie| parallel -j 20 bash -c "{}"
     baselines/new/bm25-lucene.sh short test biocaddie| parallel -j 20 bash -c "{}"
    baselines/new/rocchio-lucene.sh short test biocaddie| parallel -j 20 bash -c "{}"

Using biocaddie_all.snowball indexes

No Format
cd ~/biocaddie
baselines/new/<model>-lucene-snowball.sh <topics> <subset> <col>| parallel -j 20 bash -c "{}"
baselines/new/<model>-lucene-snowball.sh <topics> <subset> <col> <year>| parallel -j 20 bash -c "{}"

Eg: baselines/new/dir-lucene-snowball.sh short test biocaddie| parallel -j 20 bash -c "{}"
      baselines/new/tfidf-lucene-snowball.sh short test biocaddie| parallel -j 20 bash -c "{}"
     baselines/new/jm-lucene-snowball.sh short test biocaddie| parallel -j 20 bash -c "{}"
     baselines/new/bm25-lucene-snowball.sh short test biocaddie| parallel -j 20 bash -c "{}"
    baselines/new/rocchio-lucene-snowball.sh short test biocaddie| parallel -j 20 bash -c "{}"

    b) Evaluation and Cross-validation (lucene-eval, loocv) 

No Format
cd ~/biocaddie
scripts/new/mkeval-lucene.sh <model> <topics> <subset> <col>
scripts/new/mkeval-lucene.sh <model> <topics> <subset> <col> <year>

Eg: scripts/new/mkeval-lucene.sh dir short test biocaddie
       scripts/new/mkeval-lucene.sh tfidf short test biocaddie
       scripts/new/mkeval-lucene.sh jm short test biocaddie
       scripts/new/mkeval-lucene.sh bm25 short test biocaddie
       scripts/new/mkeval-lucene.sh rocchio short test biocaddie

       scripts/new/mkeval-lucene.sh dir-snowball short test biocaddie
       scripts/new/mkeval-lucene.sh tfidf-snowball short test biocaddie
       scripts/new/mkeval-lucene.sh jm-snowball short test biocaddie
       scripts/new/mkeval-lucene.sh bm25-snowball short test biocaddie
       scripts/new/mkeval-lucene.sh rocchio-snowball short test biocaddie

    c) Compare models

        We have to input running method for comparison:

        0 - both from and to models are from Indri run

        1 - both from and to models are from Lucene run

        2 - from model is from Indri run, to model is from Lucene run

        3 - from model is from Lucene run, to model is from Indri run

No Format
cd ~/biocaddie
Rscript scripts/new/compare.R <subset> <from> <to> <topics> <col>
Rscript scripts/new/compare.R <subset> <from> <to> <topics> <col> <year>


        Eg: Rscript scripts/new/compare.R test tfidf dir short biocaddie

              Rscript scripts/new/compare.R test tfidf-snowball dir-snowball short biocaddie

4. Results

Using biocaddie_all indexes.

ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
classic tfidf0.32820.58240.68670.54780.50130.5018No parameters07/05/17
BM250.35430.6105+0.7467+0.5917+0.5060.5186Sweep b, k107/05/17
QL (JM)0.33820.60220.72330.5710.50.4996Sweep lambda07/05/17
QL (Dir)

0.3675

(p-value=0.0548)

0.6163+

(p-value=0.0502)

0.65670.56640.52130.522

Sweep mu

07/05/17
Rocchio0.4044+

0.6417

(p-value=0.0533)

0.69670.54030.4920.4912Sweep b, k1, fbTerms, fbDocs, fbOrigWeight07/05/17

Using biocaddie_all.snowball indexes

ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
classic tfidf (tfidf-snowball)0.33750.59440.66670.52560.49870.5002No parameters07/06/17
BM25 (bm25-snowball)0.3764+0.6239+0.73+0.6006+0.5413+0.539+Sweep b, k107/06/17
QL (JM) (jm-snowball)0.34480.60580.670.5813+0.49870.5289+Sweep lambda07/06/17
QL (Dir) (dir-snowball)

0.3776+

0.6315+

0.70330.6006+0.53070.5365+

Sweep mu

07/06/17
Rocchio (rocchio-snowball)0.3959

0.6052

0.72670.598+0.54530.525Sweep b, k1, fbTerms, fbDocs, fbOrigWeight07/06/17

Difference between unstemmed and stemmed indexes

ModelMAPNDCGP@20NDCG@20P@100NDCG@100NotesDate
classic tfidf0.32820.58240.68670.54780.50130.5018No parameters07/10/17
classic tfidf (tfidf-snowball)0.33750.59440.66670.52560.49870.5002No parameters07/10/17
BM250.35430.61050.74670.59170.5060.5186Sweep b, k107/10/17
BM25 (bm25-snowball)0.3764+0.62390.730.60060.5413+0.539+Sweep b, k107/10/17
QL (JM)0.33820.60220.72330.5710.50.4996Sweep lambda07/10/17
QL (JM) (jm-snowball)0.34480.60580.670.58130.49870.5289+Sweep lambda07/10/17
QL (Dir)

0.3675

0.6163

0.65670.56640.52130.522

Sweep mu

07/10/17
QL (Dir) (dir-snowball)

0.3776

0.6315

(p-value=0.0534)

0.7033+0.6006+0.53070.5365

Sweep mu

07/10/17
Rocchio0.4044

0.6417

0.69670.54030.4920.4912Sweep b, k1, fbTerms, fbDocs, fbOrigWeight07/11/17
Rocchio (rocchio-snowball)0.3959

0.6052-

0.72670.598+0.5453+0.525Sweep b, k1, fbTerms, fbDocs, fbOrigWeight07/11/17


Verification

Using biocaddie_all indexes:

No Format
thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf dir short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3282 0.3675 p= 0.0548"
[1] "ndcg 0.5824 0.6163 p= 0.0502"
[1] "P_20 0.6867 0.6567 p= 0.9461"
[1] "ndcg_cut_20 0.5478 0.5664 p= 0.186"
[1] "P_100 0.5013 0.5213 p= 0.2168"
[1] "ndcg_cut_100 0.5018 0.522 p= 0.1401"


thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf jm short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3282 0.3382 p= 0.1719"
[1] "ndcg 0.5824 0.6022 p= 0.0932"
[1] "P_20 0.6867 0.7233 p= 0.0831"
[1] "ndcg_cut_20 0.5478 0.571 p= 0.145"
[1] "P_100 0.5013 0.5 p= 0.5301"
[1] "ndcg_cut_100 0.5018 0.4996 p= 0.5552"


thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf bm25 short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3282 0.3543 p= 0.0846"
[1] "ndcg 0.5824 0.6105 p= 0.0148"
[1] "P_20 0.6867 0.7467 p= 0.0491"
[1] "ndcg_cut_20 0.5478 0.5917 p= 0.0496"
[1] "P_100 0.5013 0.506 p= 0.428"
[1] "ndcg_cut_100 0.5018 0.5186 p= 0.2195"

thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf rocchio short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3282 0.4044 p= 0.0188"
[1] "ndcg 0.5824 0.6417 p= 0.0533"
[1] "P_20 0.6867 0.6967 p= 0.3785"
[1] "ndcg_cut_20 0.5478 0.5403 p= 0.6276"
[1] "P_100 0.5013 0.492 p= 0.6184"
[1] "ndcg_cut_100 0.5018 0.4912 p= 0.6071"

Using biocaddie_all.snowball indexes

No Format
thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf-snowball dir-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3375 0.3776 p= 0.0387"
[1] "ndcg 0.5944 0.6315 p= 0.0072"
[1] "P_20 0.6667 0.7033 p= 0.1042"
[1] "ndcg_cut_20 0.5256 0.6006 p= 0.0046"
[1] "P_100 0.4987 0.5307 p= 0.0652"
[1] "ndcg_cut_100 0.5002 0.5365 p= 0.0207"

thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf-snowball jm-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3375 0.3448 p= 0.2782"
[1] "ndcg 0.5944 0.6058 p= 0.2069"
[1] "P_20 0.6667 0.67 p= 0.475"
[1] "ndcg_cut_20 0.5256 0.5813 p= 0.0161"
[1] "P_100 0.4987 0.4987 p= 0.5"
[1] "ndcg_cut_100 0.5002 0.5289 p= 0.0117"

thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf-snowball bm25-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3375 0.3764 p= 0.0284"
[1] "ndcg 0.5944 0.6239 p= 0.011"
[1] "P_20 0.6667 0.73 p= 0.0331"
[1] "ndcg_cut_20 0.5256 0.6006 p= 0.0045"
[1] "P_100 0.4987 0.5413 p= 0.0326"
[1] "ndcg_cut_100 0.5002 0.539 p= 0.0149"


thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf rocchio-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3282 0.3959 p= 0.0427"
[1] "ndcg 0.5824 0.6052 p= 0.2869"
[1] "P_20 0.6867 0.7267 p= 0.1189"
[1] "ndcg_cut_20 0.5478 0.598 p= 0.0424"
[1] "P_100 0.5013 0.5453 p= 0.1152"
[1] "ndcg_cut_100 0.5018 0.525 p= 0.2733"

Compare results between unstemmed and stemmed indexes:

No Format
thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidf tfidf-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3282 0.3375 p= 0.1463"
[1] "ndcg 0.5824 0.5944 p= 0.1454"
[1] "P_20 0.6867 0.6667 p= 0.808"
[1] "ndcg_cut_20 0.5478 0.5256 p= 0.8715"
[1] "P_100 0.5013 0.4987 p= 0.5819"
[1] "ndcg_cut_100 0.5018 0.5002 p= 0.5652"

thphan@biocaddie-dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test dir dir-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.3675 0.3776 p= 0.0842"
[1] "ndcg 0.6163 0.6315 p= 0.0534"
[1] "P_20 0.6567 0.7033 p= 0.0011"
[1] "ndcg_cut_20 0.5664 0.6006 p= 0.0222"
[1] "P_100 0.5213 0.5307 p= 0.1942"
[1] "ndcg_cut_100 0.522 0.5365 p= 0.0645"

thphan@biocaddie-dev:/data/thphan/biocaddie$
No Format
root@integration-1:~/biocaddie# Rscript scripts/new/compare.R test tfidfjm dirjm-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.32833382 0.36773448 p= 0.05262603"
[1] "ndcg 0.58166022 0.61696058 p= 0.04083358"
[1] "P_20 0.69337233 0.666767 p= 0.92518885"
[1] "ndcg_cut_20 0.5462571 0.56765813 p= 0.15332551"
[1] "P_100 0.50075 0.52134987 p= 0.205355"
[1] "ndcg_cut_100 0.4996 0.52215289 p= 0.11620026"

root@integrationthphan@biocaddie-1:~/biocaddie#dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidfbm25 bm25-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.32833543 0.35443764 p= 0.08250317"
[1] "ndcg 0.58166105 0.60616239 p= 0.02390945"
[1] "P_20 0.69337467 0.7573 p= 0.06128548"
[1] "ndcg_cut_20 0.54625917 0.59586006 p= 0.03022775"
[1] "P_100 0.5007506 0.50675413 p= 0.40720209"
[1] "ndcg_cut_100 0.49965186 0.5182539 p= 0.19750441"

root@integrationthphan@biocaddie-1:~/biocaddie#dev:/data/thphan/biocaddie$ Rscript scripts/new/compare.R test tfidfrocchio jmrocchio-snowball short biocaddie
Please enter run methods for comparison:
        0: both are Indri
        1: both are Lucene
        2: from is Indri, to is Lucene
        3: from is Lucene, to is Indri
1
[1] "map 0.32834044 0.33673959 p= 0.22316841"
[1] "ndcg 0.58166417 0.60166052 p= 0.07539667"
[1] "P_20 0.69336967 0.72337267 p= 0.13562037"
[1] "ndcg_cut_20 0.54625403 0.5713598 p= 0.13120465"
[1] "P_100 0.5007492 0.50075453 p= 0.50035"
[1] "ndcg_cut_100 0.49964912 0.5017525 p= 0.44410625"