Page History

...

One surprising result was that Query likelihood baselines with smoothing (such as JM, Dir and TS) did not improve the retrieval results over TFIDF for any metrics in TREC CDS and Ohsumed collections as bioCADDIE or previous studies did (http://trec.nist.gov/pubs/trec23/papers/pro-UCLA_MII_clinical.pdf) or. However, type of queries could be an important factor that might cause the differences in retrieval results. This was also mentioned in the study of Zhai C (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.58.8978) that queries with only keywords tend to perform better than more verbose queries .(*** Note: Zhai also mentioned longer queries performed better than short queries)

We tried to examine the difference in using verbose queries and keyword queries on Ohsumed collection.

...

Also number of queries used for running baselines in each collection could be considered for the difference.

Below is the cross validation results for map metric of tfidf, dir, jm and two baselines for all queries in Ohsumed collection (there are only 97 queries listed instead of 106 because 7 other queries contain invalid punctuations for trec_eval to process)

Compared to tfidf, dir has 26/97 queries with higher map values; jm has 19/97 queries with higher map values; and two stage has 27/97 queries with higher map values.

Hence, if we use a smaller number of queries (such as less than 25 queries), there are possibilities that QL might yield better results than tfidf.

	tfidf		dir		jm		two
OHSU-TEST1	k1=0.7:b=0.5.eval	0	500.eval	0	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	0
OHSU-TEST11	k1=0.7:b=0.5.eval	0.0013	500.eval	0.005	lambda=0.7.eval	0.0059	mu=250:lambda=0.6.eval	0.004
OHSU-TEST12	k1=0.7:b=0.5.eval	0.0627	500.eval	0.0138	lambda=0.7.eval	0.0084	mu=250:lambda=0.6.eval	0.0171
OHSU-TEST14	k1=0.7:b=0.5.eval	0.0425	500.eval	0.0315	lambda=0.7.eval	0.047	mu=250:lambda=0.6.eval	0.0566
OHSU-TEST15	k1=0.7:b=0.5.eval	0	500.eval	0	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	0
OHSU-TEST16	k1=0.7:b=0.5.eval	0	500.eval	0	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	0
OHSU-TEST17	k1=0.7:b=0.4.eval	0.0996	500.eval	0.0727	lambda=0.7.eval	0.0991	mu=250:lambda=0.6.eval	0.0857
OHSU-TEST19	k1=0.7:b=0.5.eval	0.0273	500.eval	0.0226	lambda=0.7.eval	0.0174	mu=250:lambda=0.6.eval	0.0229
OHSU-TEST2	k1=0.7:b=0.5.eval	0	500.eval	0	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	0
OHSU-TEST20	k1=0.7:b=0.5.eval	0.0445	500.eval	0.0478	lambda=0.7.eval	0.0529	mu=250:lambda=0.6.eval	0.0493
OHSU-TEST21	k1=0.7:b=0.5.eval	0.0176	500.eval	0.0128	lambda=0.7.eval	0.0057	mu=250:lambda=0.6.eval	0.0107
OHSU-TEST22	k1=0.7:b=0.5.eval	0.0244	500.eval	0.0213	lambda=0.7.eval	0.0278	mu=250:lambda=0.6.eval	0.0244
OHSU-TEST23	k1=0.7:b=0.5.eval	0.0611	500.eval	0.0383	lambda=0.7.eval	0.0329	mu=250:lambda=0.6.eval	0.0382
OHSU-TEST25	k1=0.7:b=0.5.eval	0.1011	500.eval	0.1199	lambda=0.7.eval	0.0832	mu=250:lambda=0.6.eval	0.1104
OHSU-TEST26	k1=0.7:b=0.5.eval	0.018	500.eval	0.0165	lambda=0.7.eval	0.0161	mu=250:lambda=0.6.eval	0.0155
OHSU-TEST29	k1=0.7:b=0.5.eval	0.25	500.eval	0.25	lambda=0.7.eval	0.2511	mu=250:lambda=0.6.eval	0.25
OHSU-TEST30	k1=0.7:b=0.5.eval	0.1667	1000.eval	0.3333	lambda=0.7.eval	1	mu=250:lambda=0.7.eval	0.5
OHSU-TEST31	k1=0.7:b=0.5.eval	0.0131	500.eval	0.0109	lambda=0.7.eval	0.011	mu=250:lambda=0.6.eval	0.0111
OHSU-TEST32	k1=0.7:b=0.5.eval	0.021	500.eval	0.014	lambda=0.7.eval	0.0098	mu=250:lambda=0.6.eval	0.0132
OHSU-TEST33	k1=0.7:b=0.5.eval	0.0908	500.eval	0.0425	lambda=0.7.eval	0.0195	mu=250:lambda=0.6.eval	0.0782
OHSU-TEST34	k1=0.7:b=0.5.eval	0.1071	500.eval	0.0929	lambda=0.7.eval	0.1626	mu=250:lambda=0.6.eval	0.1174
OHSU-TEST35	k1=0.7:b=0.5.eval	0.0068	500.eval	0.0024	lambda=0.7.eval	0.0019	mu=250:lambda=0.6.eval	0.0029
OHSU-TEST37	k1=0.7:b=0.5.eval	0.0773	500.eval	0.0597	lambda=0.7.eval	0.0559	mu=250:lambda=0.6.eval	0.0694
OHSU-TEST38	k1=0.7:b=0.5.eval	0.0439	500.eval	0.031	lambda=0.7.eval	0.0337	mu=250:lambda=0.6.eval	0.0407
OHSU-TEST39	k1=0.7:b=0.5.eval	0	500.eval	0	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	0
OHSU-TEST4	k1=0.7:b=0.5.eval	0.2	500.eval	0.1667	lambda=0.7.eval	0.1667	mu=250:lambda=0.6.eval	0.1667
OHSU-TEST40	k1=0.7:b=0.5.eval	0.0013	500.eval	0.002	lambda=0.7.eval	0.0016	mu=250:lambda=0.6.eval	0.0021
OHSU-TEST41	k1=0.7:b=0.5.eval	0.0023	500.eval	0.0036	lambda=0.7.eval	0.0048	mu=250:lambda=0.6.eval	0.0034
OHSU-TEST42	k1=0.7:b=0.5.eval	0.0074	500.eval	0.0067	lambda=0.7.eval	0.0064	mu=250:lambda=0.6.eval	0.0067
OHSU-TEST43	k1=0.7:b=0.5.eval	7.00E-04	500.eval	6.00E-04	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	7.00E-04
OHSU-TEST5	k1=0.7:b=0.5.eval	0.0982	500.eval	0.0913	lambda=0.7.eval	0.0643	mu=250:lambda=0.6.eval	0.0868
OHSU-TEST7	k1=0.7:b=0.5.eval	0	500.eval	0	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	0
OHSU-TEST8	k1=0.7:b=0.5.eval	0.0175	500.eval	0.0203	lambda=0.7.eval	0.0089	mu=250:lambda=0.6.eval	0.021
OHSU-TEST9	k1=0.7:b=0.5.eval	0	500.eval	0	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	0
OHSU1	k1=0.7:b=0.5.eval	0.1978	500.eval	0.2136	lambda=0.7.eval	0.1919	mu=250:lambda=0.6.eval	0.2157
OHSU10	k1=0.7:b=0.5.eval	0.1683	500.eval	0.138	lambda=0.7.eval	0.1184	mu=250:lambda=0.6.eval	0.1371
OHSU11	k1=0.7:b=0.5.eval	0.2714	500.eval	0.2335	lambda=0.7.eval	0.2166	mu=250:lambda=0.6.eval	0.2162
OHSU12	k1=0.7:b=0.5.eval	0.5681	500.eval	0.5692	lambda=0.7.eval	0.6	mu=250:lambda=0.6.eval	0.5797
OHSU13	k1=0.7:b=0.5.eval	0.3131	500.eval	0.2962	lambda=0.7.eval	0.2169	mu=250:lambda=0.6.eval	0.2643
OHSU14	k1=0.7:b=0.5.eval	0.3825	500.eval	0.3546	lambda=0.7.eval	0.2714	mu=250:lambda=0.6.eval	0.3519
OHSU15	k1=0.7:b=0.5.eval	0.1553	500.eval	0.2467	lambda=0.7.eval	0.2296	mu=250:lambda=0.6.eval	0.2442
OHSU16	k1=0.7:b=0.5.eval	0.1987	500.eval	0.1674	lambda=0.7.eval	0.1554	mu=250:lambda=0.6.eval	0.1757
OHSU17	k1=0.7:b=0.5.eval	0.436	500.eval	0.4677	lambda=0.7.eval	0.2697	mu=250:lambda=0.6.eval	0.4261
OHSU18	k1=0.7:b=0.5.eval	0.3092	500.eval	0.3346	lambda=0.7.eval	0.2672	mu=250:lambda=0.6.eval	0.3239
OHSU19	k1=0.7:b=0.5.eval	0.7802	500.eval	0.7808	lambda=0.7.eval	0.7756	mu=250:lambda=0.6.eval	0.7796
OHSU2	k1=0.7:b=0.5.eval	0.38	500.eval	0.4186	lambda=0.7.eval	0.2564	mu=250:lambda=0.6.eval	0.4064
OHSU20	k1=0.7:b=0.5.eval	0.607	500.eval	0.6242	lambda=0.7.eval	0.6596	mu=250:lambda=0.6.eval	0.6292
OHSU21	k1=0.7:b=0.5.eval	0.4549	500.eval	0.4568	lambda=0.7.eval	0.3549	mu=250:lambda=0.6.eval	0.4636
OHSU22	k1=0.7:b=0.5.eval	0.1981	500.eval	0.1747	lambda=0.7.eval	0.16	mu=250:lambda=0.6.eval	0.1852
OHSU23	k1=0.7:b=0.5.eval	0.5543	500.eval	0.4215	lambda=0.7.eval	0.4487	mu=250:lambda=0.6.eval	0.4242
OHSU24	k1=0.7:b=0.5.eval	0.606	500.eval	0.4991	lambda=0.7.eval	0.3824	mu=250:lambda=0.6.eval	0.5521
OHSU25	k1=0.7:b=0.5.eval	0.0466	500.eval	0.0703	lambda=0.7.eval	0.0569	mu=250:lambda=0.6.eval	0.0711
OHSU26	k1=0.7:b=0.5.eval	0.8438	500.eval	0.829	lambda=0.7.eval	0.8408	mu=250:lambda=0.6.eval	0.8305
OHSU27	k1=0.7:b=0.5.eval	0.6065	500.eval	0.6296	lambda=0.7.eval	0.5555	mu=250:lambda=0.6.eval	0.6381
OHSU28	k1=0.7:b=0.5.eval	0.0867	500.eval	0.0803	lambda=0.7.eval	0.0681	mu=250:lambda=0.6.eval	0.0844
OHSU29	k1=0.7:b=0.5.eval	0.4462	500.eval	0.4381	lambda=0.7.eval	0.4798	mu=250:lambda=0.6.eval	0.4447
OHSU3	k1=0.7:b=0.5.eval	0.5177	500.eval	0.4651	lambda=0.7.eval	0.4347	mu=250:lambda=0.6.eval	0.4797
OHSU30	k1=0.7:b=0.5.eval	0.5813	500.eval	0.5806	lambda=0.7.eval	0.4513	mu=250:lambda=0.6.eval	0.5674
OHSU31	k1=0.7:b=0.5.eval	0.1089	500.eval	0.0968	lambda=0.7.eval	0.1081	mu=250:lambda=0.6.eval	0.1002
OHSU32	k1=0.7:b=0.5.eval	0.1039	500.eval	0.076	lambda=0.7.eval	0.0598	mu=250:lambda=0.6.eval	0.083
OHSU33	k1=0.7:b=0.5.eval	0.2543	500.eval	0.151	lambda=0.7.eval	0.1089	mu=250:lambda=0.6.eval	0.1654
OHSU34	k1=0.7:b=0.5.eval	0.0517	500.eval	0.0436	lambda=0.7.eval	0.0431	mu=250:lambda=0.6.eval	0.0439
OHSU35	k1=0.7:b=0.5.eval	0.8056	500.eval	0.5475	lambda=0.7.eval	0.4332	mu=250:lambda=0.6.eval	0.7356
OHSU36	k1=0.7:b=0.5.eval	0.2005	500.eval	0.164	lambda=0.7.eval	0.1623	mu=250:lambda=0.6.eval	0.1589
OHSU37	k1=0.7:b=0.5.eval	0.2567	500.eval	0.0755	lambda=0.7.eval	0.0423	mu=250:lambda=0.6.eval	0.1038
OHSU38	k1=0.7:b=0.5.eval	0.6375	500.eval	0.5826	lambda=0.7.eval	0.5679	mu=250:lambda=0.6.eval	0.603
OHSU39	k1=0.7:b=0.5.eval	0.125	500.eval	0.0691	lambda=0.7.eval	0.0457	mu=250:lambda=0.6.eval	0.0655
OHSU4	k1=0.7:b=0.5.eval	0.6214	500.eval	0.5542	lambda=0.7.eval	0.5198	mu=250:lambda=0.6.eval	0.5606
OHSU40	k1=0.7:b=0.5.eval	0.4596	500.eval	0.3519	lambda=0.7.eval	0.189	mu=250:lambda=0.6.eval	0.41
OHSU41	k1=0.7:b=0.5.eval	0.3057	500.eval	0.2482	lambda=0.7.eval	0.2234	mu=250:lambda=0.6.eval	0.2549
OHSU42	k1=0.7:b=0.5.eval	0.1898	500.eval	0.1836	lambda=0.7.eval	0.1861	mu=250:lambda=0.6.eval	0.1892
OHSU43	k1=0.7:b=0.5.eval	0.7551	500.eval	0.7262	lambda=0.7.eval	0.6398	mu=250:lambda=0.6.eval	0.737
OHSU44	k1=0.7:b=0.5.eval	0.4242	500.eval	0.359	lambda=0.7.eval	0.3526	mu=250:lambda=0.6.eval	0.3724
OHSU45	k1=0.7:b=0.5.eval	0.0676	500.eval	0.034	lambda=0.7.eval	0.0107	mu=250:lambda=0.6.eval	0.0596
OHSU46	k1=0.7:b=0.5.eval	0.2438	500.eval	0.2487	lambda=0.7.eval	0.1712	mu=250:lambda=0.6.eval	0.2513
OHSU47	k1=0.7:b=0.5.eval	0.3565	500.eval	0.3483	lambda=0.7.eval	0.3376	mu=250:lambda=0.6.eval	0.3644
OHSU48	k1=0.7:b=0.5.eval	0.2768	500.eval	0.4356	lambda=0.7.eval	0.3877	mu=250:lambda=0.6.eval	0.387
OHSU49	k1=0.7:b=0.5.eval	0.2734	500.eval	0.2411	lambda=0.8.eval	0.1215	mu=250:lambda=0.6.eval	0.2288
OHSU5	k1=0.7:b=0.5.eval	0.6262	500.eval	0.5909	lambda=0.7.eval	0.5647	mu=250:lambda=0.6.eval	0.5992
OHSU50	k1=0.7:b=0.5.eval	0.3306	500.eval	0.2609	lambda=0.7.eval	0.2804	mu=250:lambda=0.6.eval	0.2603
OHSU51	k1=0.7:b=0.5.eval	0.3498	500.eval	0.3026	lambda=0.8.eval	0.1902	mu=250:lambda=0.6.eval	0.263
OHSU52	k1=0.7:b=0.5.eval	0.126	500.eval	0.1618	lambda=0.7.eval	0.1068	mu=250:lambda=0.6.eval	0.1503
OHSU53	k1=0.7:b=0.5.eval	0.0958	500.eval	0.0967	lambda=0.7.eval	0.0123	mu=250:lambda=0.6.eval	0.0768
OHSU54	k1=0.7:b=0.5.eval	0.3455	500.eval	0.2924	lambda=0.7.eval	0.3081	mu=250:lambda=0.6.eval	0.3254
OHSU55	k1=0.7:b=0.5.eval	0.0947	500.eval	0.1603	lambda=0.7.eval	0.1134	mu=250:lambda=0.6.eval	0.1614
OHSU56	k1=0.7:b=0.5.eval	0.1287	500.eval	0.1358	lambda=0.7.eval	0.1144	mu=250:lambda=0.6.eval	0.1004
OHSU57	k1=0.7:b=0.5.eval	0.3136	500.eval	0.3096	lambda=0.7.eval	0.3194	mu=250:lambda=0.6.eval	0.3329
OHSU58	k1=0.7:b=0.5.eval	0	500.eval	0	lambda=0.7.eval	0	mu=250:lambda=0.6.eval	0
OHSU59	k1=0.7:b=0.5.eval	0.3361	500.eval	0.3125	lambda=0.7.eval	0.3089	mu=250:lambda=0.6.eval	0.3126
OHSU6	k1=0.7:b=0.5.eval	0.1673	500.eval	0.1533	lambda=0.7.eval	0.1334	mu=250:lambda=0.6.eval	0.1774
OHSU60	k1=0.7:b=0.5.eval	0.1021	500.eval	0.013	lambda=0.7.eval	0.009	mu=250:lambda=0.6.eval	0.0199
OHSU61	k1=0.7:b=0.5.eval	0.0313	500.eval	0.0612	lambda=0.7.eval	0.0675	mu=250:lambda=0.6.eval	0.0546
OHSU62	k1=0.7:b=0.5.eval	0.1278	500.eval	0.0815	lambda=0.7.eval	0.1016	mu=250:lambda=0.6.eval	0.0958
OHSU63	k1=0.7:b=0.5.eval	7.00E-04	500.eval	0.0018	lambda=0.7.eval	0.0011	mu=250:lambda=0.6.eval	0.0011
OHSU7	k1=0.7:b=0.5.eval	0.2013	500.eval	0.1095	lambda=0.7.eval	0.1296	mu=250:lambda=0.6.eval	0.1375
OHSU8	k1=0.7:b=0.5.eval	0.0915	500.eval	0.0683	lambda=0.7.eval	0.0745	mu=250:lambda=0.6.eval	0.0687
OHSU9	k1=0.7:b=0.5.eval	0.5243	500.eval	0.4755	lambda=0.7.eval	0.4899	mu=250:lambda=0.6.eval	0.4789

...

Space shortcuts

Page tree

Versions Compared

Old Version 7

New Version 8

Key