...
One surprising result was that Query likelihood baselines with smoothing (such as JM, Dir and TS) did not improve the retrieval results over TFIDF for any metrics in TREC CDS and Ohsumed collections as bioCADDIE or previous studies did (http://trec.nist.gov/pubs/trec23/papers/pro-UCLA_MII_clinical.pdf) or. However, type of queries could be an important factor that might cause the differences in retrieval results. This was also mentioned in the study of Zhai C (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.58.8978) that queries with only keywords tend to perform better than more verbose queries.queries (*** Note: Zhai also mentioned longer queries performed better than short queries)
We tried to examine the difference in using verbose queries and keyword queries on Ohsumed collection.
Using original queries (verbose queries) for OHSUMED
Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
---|---|---|---|---|---|---|---|---|
tfidf | 0.3188 | 0.6084 | 0.45 | 0.4255 | 0.2657 | 0.4625 | Sweep b and k1 | 06/07/17 |
QL (JM) | 0.2545- | 0.5527- | 0.3908- | 0.3882- | 0.2135- | 0.3883- | Sweep lambda | 06/07/17 |
QL (Dir) | 0.2924- | 0.5866- | 0.3975 | 0.4018- | 0.2492- | 0.432- | Sweep mu | 06/07/17 |
QL (TS) | 0.2934- | 0.5828- | 0.4092- | 0.4122 | 0.2508- | 0.4385- | Sweep mu and lambda | 06/07/17 |
Using manually refined queries (mostly keywords) for OHSUMED
Model | MAP | NDCG | P@20 | NDCG@20 | P@100 | NDCG@100 | Notes | Date |
---|---|---|---|---|---|---|---|---|
tfidf | 0.315 | 0.5949 | 0.4198 | 0.3802 | 0.2614 | 0.4454 | Sweep b and k1 | 06/09/17 |
QL (JM) | 0.2587- | 0.5466- | 0.3817- | 0.3608 | 0.2257- | 0.3806- | Sweep lambda | 06/09/17 |
QL (Dir) | 0.3027- | 0.5883 | 0.4087 | 0.379 | 0.261 | 0.4333- | Sweep mu | 06/09/17 |
QL (TS) | 0.3052- | 0.5871 | 0.4159 | 0.3896 | 0.2627 | 0.4354- | Sweep mu and lambda | 06/09/17 |
We can see that when using keyword queries, the difference in retrieval results between tfidf and QL is smaller.
...
Also number of queries used for running baselines in each collection could be considered for the difference.
Below is the cross validation results for map metric of tfidf, dir, jm and two baselines for all queries in Ohsumed collection (to be continued)there are only 97 queries listed instead of 106 because 7 other queries contain invalid punctuations for trec_eval to process)
Compared to tfidf, dir has 26/97 queries with higher map values; jm has 19/97 queries with higher map values; and two stage has 27/97 queries with higher map values.
Hence, if we use a smaller number of queries (such as less than 25 queries), there are possibilities that QL might yield better results than tfidf.
tfidf | dir | jm | two | |||||
---|---|---|---|---|---|---|---|---|
OHSU-TEST1 | k1=0.7:b=0.5.eval | 0 | 500.eval | 0 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 0 |
OHSU-TEST11 | k1=0.7:b=0.5.eval | 0.0013 | 500.eval | 0.005 | lambda=0.7.eval | 0.0059 | mu=250:lambda=0.6.eval | 0.004 |
OHSU-TEST12 | k1=0.7:b=0.5.eval | 0.0627 | 500.eval | 0.0138 | lambda=0.7.eval | 0.0084 | mu=250:lambda=0.6.eval | 0.0171 |
OHSU-TEST14 | k1=0.7:b=0.5.eval | 0.0425 | 500.eval | 0.0315 | lambda=0.7.eval | 0.047 | mu=250:lambda=0.6.eval | 0.0566 |
OHSU-TEST15 | k1=0.7:b=0.5.eval | 0 | 500.eval | 0 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 0 |
OHSU-TEST16 | k1=0.7:b=0.5.eval | 0 | 500.eval | 0 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 0 |
OHSU-TEST17 | k1=0.7:b=0.4.eval | 0.0996 | 500.eval | 0.0727 | lambda=0.7.eval | 0.0991 | mu=250:lambda=0.6.eval | 0.0857 |
OHSU-TEST19 | k1=0.7:b=0.5.eval | 0.0273 | 500.eval | 0.0226 | lambda=0.7.eval | 0.0174 | mu=250:lambda=0.6.eval | 0.0229 |
OHSU-TEST2 | k1=0.7:b=0.5.eval | 0 | 500.eval | 0 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 0 |
OHSU-TEST20 | k1=0.7:b=0.5.eval | 0.0445 | 500.eval | 0.0478 | lambda=0.7.eval | 0.0529 | mu=250:lambda=0.6.eval | 0.0493 |
OHSU-TEST21 | k1=0.7:b=0.5.eval | 0.0176 | 500.eval | 0.0128 | lambda=0.7.eval | 0.0057 | mu=250:lambda=0.6.eval | 0.0107 |
OHSU-TEST22 | k1=0.7:b=0.5.eval | 0.0244 | 500.eval | 0.0213 | lambda=0.7.eval | 0.0278 | mu=250:lambda=0.6.eval | 0.0244 |
OHSU-TEST23 | k1=0.7:b=0.5.eval | 0.0611 | 500.eval | 0.0383 | lambda=0.7.eval | 0.0329 | mu=250:lambda=0.6.eval | 0.0382 |
OHSU-TEST25 | k1=0.7:b=0.5.eval | 0.1011 | 500.eval | 0.1199 | lambda=0.7.eval | 0.0832 | mu=250:lambda=0.6.eval | 0.1104 |
OHSU-TEST26 | k1=0.7:b=0.5.eval | 0.018 | 500.eval | 0.0165 | lambda=0.7.eval | 0.0161 | mu=250:lambda=0.6.eval | 0.0155 |
OHSU-TEST29 | k1=0.7:b=0.5.eval | 0.25 | 500.eval | 0.25 | lambda=0.7.eval | 0.2511 | mu=250:lambda=0.6.eval | 0.25 |
OHSU-TEST30 | k1=0.7:b=0.5.eval | 0.1667 | 1000.eval | 0.3333 | lambda=0.7.eval | 1 | mu=250:lambda=0.7.eval | 0.5 |
OHSU-TEST31 | k1=0.7:b=0.5.eval | 0.0131 | 500.eval | 0.0109 | lambda=0.7.eval | 0.011 | mu=250:lambda=0.6.eval | 0.0111 |
OHSU-TEST32 | k1=0.7:b=0.5.eval | 0.021 | 500.eval | 0.014 | lambda=0.7.eval | 0.0098 | mu=250:lambda=0.6.eval | 0.0132 |
OHSU-TEST33 | k1=0.7:b=0.5.eval | 0.0908 | 500.eval | 0.0425 | lambda=0.7.eval | 0.0195 | mu=250:lambda=0.6.eval | 0.0782 |
OHSU-TEST34 | k1=0.7:b=0.5.eval | 0.1071 | 500.eval | 0.0929 | lambda=0.7.eval | 0.1626 | mu=250:lambda=0.6.eval | 0.1174 |
OHSU-TEST35 | k1=0.7:b=0.5.eval | 0.0068 | 500.eval | 0.0024 | lambda=0.7.eval | 0.0019 | mu=250:lambda=0.6.eval | 0.0029 |
OHSU-TEST37 | k1=0.7:b=0.5.eval | 0.0773 | 500.eval | 0.0597 | lambda=0.7.eval | 0.0559 | mu=250:lambda=0.6.eval | 0.0694 |
OHSU-TEST38 | k1=0.7:b=0.5.eval | 0.0439 | 500.eval | 0.031 | lambda=0.7.eval | 0.0337 | mu=250:lambda=0.6.eval | 0.0407 |
OHSU-TEST39 | k1=0.7:b=0.5.eval | 0 | 500.eval | 0 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 0 |
OHSU-TEST4 | k1=0.7:b=0.5.eval | 0.2 | 500.eval | 0.1667 | lambda=0.7.eval | 0.1667 | mu=250:lambda=0.6.eval | 0.1667 |
OHSU-TEST40 | k1=0.7:b=0.5.eval | 0.0013 | 500.eval | 0.002 | lambda=0.7.eval | 0.0016 | mu=250:lambda=0.6.eval | 0.0021 |
OHSU-TEST41 | k1=0.7:b=0.5.eval | 0.0023 | 500.eval | 0.0036 | lambda=0.7.eval | 0.0048 | mu=250:lambda=0.6.eval | 0.0034 |
OHSU-TEST42 | k1=0.7:b=0.5.eval | 0.0074 | 500.eval | 0.0067 | lambda=0.7.eval | 0.0064 | mu=250:lambda=0.6.eval | 0.0067 |
OHSU-TEST43 | k1=0.7:b=0.5.eval | 7.00E-04 | 500.eval | 6.00E-04 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 7.00E-04 |
OHSU-TEST5 | k1=0.7:b=0.5.eval | 0.0982 | 500.eval | 0.0913 | lambda=0.7.eval | 0.0643 | mu=250:lambda=0.6.eval | 0.0868 |
OHSU-TEST7 | k1=0.7:b=0.5.eval | 0 | 500.eval | 0 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 0 |
OHSU-TEST8 | k1=0.7:b=0.5.eval | 0.0175 | 500.eval | 0.0203 | lambda=0.7.eval | 0.0089 | mu=250:lambda=0.6.eval | 0.021 |
OHSU-TEST9 | k1=0.7:b=0.5.eval | 0 | 500.eval | 0 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 0 |
OHSU1 | k1=0.7:b=0.5.eval | 0.1978 | 500.eval | 0.2136 | lambda=0.7.eval | 0.1919 | mu=250:lambda=0.6.eval | 0.2157 |
OHSU10 | k1=0.7:b=0.5.eval | 0.1683 | 500.eval | 0.138 | lambda=0.7.eval | 0.1184 | mu=250:lambda=0.6.eval | 0.1371 |
OHSU11 | k1=0.7:b=0.5.eval | 0.2714 | 500.eval | 0.2335 | lambda=0.7.eval | 0.2166 | mu=250:lambda=0.6.eval | 0.2162 |
OHSU12 | k1=0.7:b=0.5.eval | 0.5681 | 500.eval | 0.5692 | lambda=0.7.eval | 0.6 | mu=250:lambda=0.6.eval | 0.5797 |
OHSU13 | k1=0.7:b=0.5.eval | 0.3131 | 500.eval | 0.2962 | lambda=0.7.eval | 0.2169 | mu=250:lambda=0.6.eval | 0.2643 |
OHSU14 | k1=0.7:b=0.5.eval | 0.3825 | 500.eval | 0.3546 | lambda=0.7.eval | 0.2714 | mu=250:lambda=0.6.eval | 0.3519 |
OHSU15 | k1=0.7:b=0.5.eval | 0.1553 | 500.eval | 0.2467 | lambda=0.7.eval | 0.2296 | mu=250:lambda=0.6.eval | 0.2442 |
OHSU16 | k1=0.7:b=0.5.eval | 0.1987 | 500.eval | 0.1674 | lambda=0.7.eval | 0.1554 | mu=250:lambda=0.6.eval | 0.1757 |
OHSU17 | k1=0.7:b=0.5.eval | 0.436 | 500.eval | 0.4677 | lambda=0.7.eval | 0.2697 | mu=250:lambda=0.6.eval | 0.4261 |
OHSU18 | k1=0.7:b=0.5.eval | 0.3092 | 500.eval | 0.3346 | lambda=0.7.eval | 0.2672 | mu=250:lambda=0.6.eval | 0.3239 |
OHSU19 | k1=0.7:b=0.5.eval | 0.7802 | 500.eval | 0.7808 | lambda=0.7.eval | 0.7756 | mu=250:lambda=0.6.eval | 0.7796 |
OHSU2 | k1=0.7:b=0.5.eval | 0.38 | 500.eval | 0.4186 | lambda=0.7.eval | 0.2564 | mu=250:lambda=0.6.eval | 0.4064 |
OHSU20 | k1=0.7:b=0.5.eval | 0.607 | 500.eval | 0.6242 | lambda=0.7.eval | 0.6596 | mu=250:lambda=0.6.eval | 0.6292 |
OHSU21 | k1=0.7:b=0.5.eval | 0.4549 | 500.eval | 0.4568 | lambda=0.7.eval | 0.3549 | mu=250:lambda=0.6.eval | 0.4636 |
OHSU22 | k1=0.7:b=0.5.eval | 0.1981 | 500.eval | 0.1747 | lambda=0.7.eval | 0.16 | mu=250:lambda=0.6.eval | 0.1852 |
OHSU23 | k1=0.7:b=0.5.eval | 0.5543 | 500.eval | 0.4215 | lambda=0.7.eval | 0.4487 | mu=250:lambda=0.6.eval | 0.4242 |
OHSU24 | k1=0.7:b=0.5.eval | 0.606 | 500.eval | 0.4991 | lambda=0.7.eval | 0.3824 | mu=250:lambda=0.6.eval | 0.5521 |
OHSU25 | k1=0.7:b=0.5.eval | 0.0466 | 500.eval | 0.0703 | lambda=0.7.eval | 0.0569 | mu=250:lambda=0.6.eval | 0.0711 |
OHSU26 | k1=0.7:b=0.5.eval | 0.8438 | 500.eval | 0.829 | lambda=0.7.eval | 0.8408 | mu=250:lambda=0.6.eval | 0.8305 |
OHSU27 | k1=0.7:b=0.5.eval | 0.6065 | 500.eval | 0.6296 | lambda=0.7.eval | 0.5555 | mu=250:lambda=0.6.eval | 0.6381 |
OHSU28 | k1=0.7:b=0.5.eval | 0.0867 | 500.eval | 0.0803 | lambda=0.7.eval | 0.0681 | mu=250:lambda=0.6.eval | 0.0844 |
OHSU29 | k1=0.7:b=0.5.eval | 0.4462 | 500.eval | 0.4381 | lambda=0.7.eval | 0.4798 | mu=250:lambda=0.6.eval | 0.4447 |
OHSU3 | k1=0.7:b=0.5.eval | 0.5177 | 500.eval | 0.4651 | lambda=0.7.eval | 0.4347 | mu=250:lambda=0.6.eval | 0.4797 |
OHSU30 | k1=0.7:b=0.5.eval | 0.5813 | 500.eval | 0.5806 | lambda=0.7.eval | 0.4513 | mu=250:lambda=0.6.eval | 0.5674 |
OHSU31 | k1=0.7:b=0.5.eval | 0.1089 | 500.eval | 0.0968 | lambda=0.7.eval | 0.1081 | mu=250:lambda=0.6.eval | 0.1002 |
OHSU32 | k1=0.7:b=0.5.eval | 0.1039 | 500.eval | 0.076 | lambda=0.7.eval | 0.0598 | mu=250:lambda=0.6.eval | 0.083 |
OHSU33 | k1=0.7:b=0.5.eval | 0.2543 | 500.eval | 0.151 | lambda=0.7.eval | 0.1089 | mu=250:lambda=0.6.eval | 0.1654 |
OHSU34 | k1=0.7:b=0.5.eval | 0.0517 | 500.eval | 0.0436 | lambda=0.7.eval | 0.0431 | mu=250:lambda=0.6.eval | 0.0439 |
OHSU35 | k1=0.7:b=0.5.eval | 0.8056 | 500.eval | 0.5475 | lambda=0.7.eval | 0.4332 | mu=250:lambda=0.6.eval | 0.7356 |
OHSU36 | k1=0.7:b=0.5.eval | 0.2005 | 500.eval | 0.164 | lambda=0.7.eval | 0.1623 | mu=250:lambda=0.6.eval | 0.1589 |
OHSU37 | k1=0.7:b=0.5.eval | 0.2567 | 500.eval | 0.0755 | lambda=0.7.eval | 0.0423 | mu=250:lambda=0.6.eval | 0.1038 |
OHSU38 | k1=0.7:b=0.5.eval | 0.6375 | 500.eval | 0.5826 | lambda=0.7.eval | 0.5679 | mu=250:lambda=0.6.eval | 0.603 |
OHSU39 | k1=0.7:b=0.5.eval | 0.125 | 500.eval | 0.0691 | lambda=0.7.eval | 0.0457 | mu=250:lambda=0.6.eval | 0.0655 |
OHSU4 | k1=0.7:b=0.5.eval | 0.6214 | 500.eval | 0.5542 | lambda=0.7.eval | 0.5198 | mu=250:lambda=0.6.eval | 0.5606 |
OHSU40 | k1=0.7:b=0.5.eval | 0.4596 | 500.eval | 0.3519 | lambda=0.7.eval | 0.189 | mu=250:lambda=0.6.eval | 0.41 |
OHSU41 | k1=0.7:b=0.5.eval | 0.3057 | 500.eval | 0.2482 | lambda=0.7.eval | 0.2234 | mu=250:lambda=0.6.eval | 0.2549 |
OHSU42 | k1=0.7:b=0.5.eval | 0.1898 | 500.eval | 0.1836 | lambda=0.7.eval | 0.1861 | mu=250:lambda=0.6.eval | 0.1892 |
OHSU43 | k1=0.7:b=0.5.eval | 0.7551 | 500.eval | 0.7262 | lambda=0.7.eval | 0.6398 | mu=250:lambda=0.6.eval | 0.737 |
OHSU44 | k1=0.7:b=0.5.eval | 0.4242 | 500.eval | 0.359 | lambda=0.7.eval | 0.3526 | mu=250:lambda=0.6.eval | 0.3724 |
OHSU45 | k1=0.7:b=0.5.eval | 0.0676 | 500.eval | 0.034 | lambda=0.7.eval | 0.0107 | mu=250:lambda=0.6.eval | 0.0596 |
OHSU46 | k1=0.7:b=0.5.eval | 0.2438 | 500.eval | 0.2487 | lambda=0.7.eval | 0.1712 | mu=250:lambda=0.6.eval | 0.2513 |
OHSU47 | k1=0.7:b=0.5.eval | 0.3565 | 500.eval | 0.3483 | lambda=0.7.eval | 0.3376 | mu=250:lambda=0.6.eval | 0.3644 |
OHSU48 | k1=0.7:b=0.5.eval | 0.2768 | 500.eval | 0.4356 | lambda=0.7.eval | 0.3877 | mu=250:lambda=0.6.eval | 0.387 |
OHSU49 | k1=0.7:b=0.5.eval | 0.2734 | 500.eval | 0.2411 | lambda=0.8.eval | 0.1215 | mu=250:lambda=0.6.eval | 0.2288 |
OHSU5 | k1=0.7:b=0.5.eval | 0.6262 | 500.eval | 0.5909 | lambda=0.7.eval | 0.5647 | mu=250:lambda=0.6.eval | 0.5992 |
OHSU50 | k1=0.7:b=0.5.eval | 0.3306 | 500.eval | 0.2609 | lambda=0.7.eval | 0.2804 | mu=250:lambda=0.6.eval | 0.2603 |
OHSU51 | k1=0.7:b=0.5.eval | 0.3498 | 500.eval | 0.3026 | lambda=0.8.eval | 0.1902 | mu=250:lambda=0.6.eval | 0.263 |
OHSU52 | k1=0.7:b=0.5.eval | 0.126 | 500.eval | 0.1618 | lambda=0.7.eval | 0.1068 | mu=250:lambda=0.6.eval | 0.1503 |
OHSU53 | k1=0.7:b=0.5.eval | 0.0958 | 500.eval | 0.0967 | lambda=0.7.eval | 0.0123 | mu=250:lambda=0.6.eval | 0.0768 |
OHSU54 | k1=0.7:b=0.5.eval | 0.3455 | 500.eval | 0.2924 | lambda=0.7.eval | 0.3081 | mu=250:lambda=0.6.eval | 0.3254 |
OHSU55 | k1=0.7:b=0.5.eval | 0.0947 | 500.eval | 0.1603 | lambda=0.7.eval | 0.1134 | mu=250:lambda=0.6.eval | 0.1614 |
OHSU56 | k1=0.7:b=0.5.eval | 0.1287 | 500.eval | 0.1358 | lambda=0.7.eval | 0.1144 | mu=250:lambda=0.6.eval | 0.1004 |
OHSU57 | k1=0.7:b=0.5.eval | 0.3136 | 500.eval | 0.3096 | lambda=0.7.eval | 0.3194 | mu=250:lambda=0.6.eval | 0.3329 |
OHSU58 | k1=0.7:b=0.5.eval | 0 | 500.eval | 0 | lambda=0.7.eval | 0 | mu=250:lambda=0.6.eval | 0 |
OHSU59 | k1=0.7:b=0.5.eval | 0.3361 | 500.eval | 0.3125 | lambda=0.7.eval | 0.3089 | mu=250:lambda=0.6.eval | 0.3126 |
OHSU6 | k1=0.7:b=0.5.eval | 0.1673 | 500.eval | 0.1533 | lambda=0.7.eval | 0.1334 | mu=250:lambda=0.6.eval | 0.1774 |
OHSU60 | k1=0.7:b=0.5.eval | 0.1021 | 500.eval | 0.013 | lambda=0.7.eval | 0.009 | mu=250:lambda=0.6.eval | 0.0199 |
OHSU61 | k1=0.7:b=0.5.eval | 0.0313 | 500.eval | 0.0612 | lambda=0.7.eval | 0.0675 | mu=250:lambda=0.6.eval | 0.0546 |
OHSU62 | k1=0.7:b=0.5.eval | 0.1278 | 500.eval | 0.0815 | lambda=0.7.eval | 0.1016 | mu=250:lambda=0.6.eval | 0.0958 |
OHSU63 | k1=0.7:b=0.5.eval | 7.00E-04 | 500.eval | 0.0018 | lambda=0.7.eval | 0.0011 | mu=250:lambda=0.6.eval | 0.0011 |
OHSU7 | k1=0.7:b=0.5.eval | 0.2013 | 500.eval | 0.1095 | lambda=0.7.eval | 0.1296 | mu=250:lambda=0.6.eval | 0.1375 |
OHSU8 | k1=0.7:b=0.5.eval | 0.0915 | 500.eval | 0.0683 | lambda=0.7.eval | 0.0745 | mu=250:lambda=0.6.eval | 0.0687 |
OHSU9 | k1=0.7:b=0.5.eval | 0.5243 | 500.eval | 0.4755 | lambda=0.7.eval | 0.4899 | mu=250:lambda=0.6.eval | 0.4789 |