PMID- 22509475
OWN - NLM
STAT- PubMed-not-MEDLINE
DCOM- 20120823
LR  - 20211021
IS  - 2093-369X (Electronic)
IS  - 2093-3681 (Print)
IS  - 2093-3681 (Linking)
VI  - 18
IP  - 1
DP  - 2012 Mar
TI  - An Evaluation of Multiple Query Representations for the Relevance Judgments used 
      to Build a Biomedical Test Collection.
PG  - 65-73
LID - 10.4258/hir.2012.18.1.65 [doi]
AB  - OBJECTIVES: The purpose of this study is to validate a method that uses multiple 
      queries to create a set of relevance judgments used to indicate which documents 
      are pertinent to each query when forming a biomedical test collection. METHODS: 
      The aspect query is the major concept of this research; it can represent every 
      aspect of the original query with the same informational need. Manually generated 
      aspect queries created by 15 recruited participants where run using the BM25 
      retrieval model in order to create aspect query based relevance sets (QRELS). In 
      order to demonstrate the feasibility of these QRELSs, The results from a 2004 
      genomics track run supported by the National Institute of Standards and 
      Technology (NIST) were used to compute the mean average precision (MAP) based on 
      Text Retrieval Conference (TREC) QRELSs and aspect-QRELSs. The rank correlation 
      was calculated using both Kendall's and Spearman's rank correlation methods. 
      RESULTS: We experimentally verified the utility of the aspect query method by 
      combining the top ranked documents retrieved by a number of multiple queries 
      which ranked the order of the information. The retrieval system correlated highly 
      with rankings based on human relevance judgments. CONCLUSIONS: Substantial 
      results were shown with high correlations of up to 0.863 (p < 0.01) between the 
      judgment-free gold standard based on the aspect queries and the human-judged gold 
      standard supported by NIST. The results also demonstrate that the aspect query 
      method can contribute in building test collections used for medical literature 
      retrieval.
FAU - Ryu, Borim
AU  - Ryu B
AD  - Department of Biomedical Engineering, Seoul National University College of 
      Medicine, Seoul, Korea.
FAU - Choi, Jinwook
AU  - Choi J
LA  - eng
PT  - Journal Article
DEP - 20120331
PL  - Korea (South)
TA  - Healthc Inform Res
JT  - Healthcare informatics research
JID - 101534553
PMC - PMC3324757
OTO - NOTNLM
OT  - Correlation Studies
OT  - Evaluation Studies
OT  - Gold Standard
OT  - Information Storage and Retrieval
OT  - MEDLINE
COIS- No potential conflict of interest relevant to this article was reported.
EDAT- 2012/04/18 06:00
MHDA- 2012/04/18 06:01
PMCR- 2012/03/01
CRDT- 2012/04/18 06:00
PHST- 2012/01/12 00:00 [received]
PHST- 2012/02/21 00:00 [revised]
PHST- 2012/03/10 00:00 [accepted]
PHST- 2012/04/18 06:00 [entrez]
PHST- 2012/04/18 06:00 [pubmed]
PHST- 2012/04/18 06:01 [medline]
PHST- 2012/03/01 00:00 [pmc-release]
AID - 10.4258/hir.2012.18.1.65 [doi]
PST - ppublish
SO  - Healthc Inform Res. 2012 Mar;18(1):65-73. doi: 10.4258/hir.2012.18.1.65. Epub 
      2012 Mar 31.