PMID- 37724961
OWN - NLM
STAT- MEDLINE
DCOM- 20230921
LR  - 20231031
IS  - 1527-1315 (Electronic)
IS  - 0033-8419 (Linking)
VI  - 308
IP  - 3
DP  - 2023 Sep
TI  - Prediction Variability to Identify Reduced AI Performance in Cancer Diagnosis at 
      MRI and CT.
PG  - e230275
LID - 10.1148/radiol.230275 [doi]
AB  - Background A priori identification of patients at risk of artificial intelligence 
      (AI) failure in diagnosing cancer would contribute to the safer clinical 
      integration of diagnostic algorithms. Purpose To evaluate AI prediction 
      variability as an uncertainty quantification (UQ) metric for identifying cases at 
      risk of AI failure in diagnosing cancer at MRI and CT across different cancer 
      types, data sets, and algorithms. Materials and Methods Multicenter data sets and 
      publicly available AI algorithms from three previous studies that evaluated 
      detection of pancreatic cancer on contrast-enhanced CT images, detection of 
      prostate cancer on MRI scans, and prediction of pulmonary nodule malignancy on 
      low-dose CT images were analyzed retrospectively. Each task's algorithm was 
      extended to generate an uncertainty score based on ensemble prediction 
      variability. AI accuracy percentage and partial area under the receiver operating 
      characteristic curve (pAUC) were compared between certain and uncertain patient 
      groups in a range of percentile thresholds (10%-90%) for the uncertainty score 
      using permutation tests for statistical significance. The pulmonary nodule 
      malignancy prediction algorithm was compared with 11 clinical readers for the 
      certain group (CG) and uncertain group (UG). Results In total, 18 022 images were 
      used for training and 838 images were used for testing. AI diagnostic accuracy 
      was higher for the cases in the CG across all tasks (P < .001). At an 80% 
      threshold of certain predictions, accuracy in the CG was 21%-29% higher than in 
      the UG and 4%-6% higher than in the overall test data sets. The lesion-level pAUC 
      in the CG was 0.25-0.39 higher than in the UG and 0.05-0.08 higher than in the 
      overall test data sets (P < .001). For pulmonary nodule malignancy prediction, 
      accuracy of AI was on par with clinicians for cases in the CG (AI results vs 
      clinician results, 80% [95% CI: 76, 85] vs 78% [95% CI: 70, 87]; P = .07) but 
      worse for cases in the UG (AI results vs clinician results, 50% [95% CI: 37, 64] 
      vs 68% [95% CI: 60, 76]; P < .001). Conclusion An AI-prediction UQ metric 
      consistently identified reduced performance of AI in cancer diagnosis. (c) RSNA, 
      2023 Supplemental material is available for this article. See also the editorial 
      by Babyn in this issue.
FAU - Alves, Natalia
AU  - Alves N
AUID- ORCID: 0000-0002-7034-0137
AD  - From the Department of Medical Imaging, Radboudumc, Route 767, Room 2.30, Geert 
      Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands (N.A., J.S.B., K.V.V., 
      C.J., M.d.R., J.H., H.H.); Department of Medicine, Section of Pulmonary Medicine, 
      Herlev-Gentofte Hospital, Herlev, Denmark (Z.S.); and Department of Clinical 
      Medicine, University of Copenhagen, Copenhagen, Denmark (Z.S.).
FAU - Bosma, Joeran S
AU  - Bosma JS
AUID- ORCID: 0000-0002-6315-124X
AD  - From the Department of Medical Imaging, Radboudumc, Route 767, Room 2.30, Geert 
      Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands (N.A., J.S.B., K.V.V., 
      C.J., M.d.R., J.H., H.H.); Department of Medicine, Section of Pulmonary Medicine, 
      Herlev-Gentofte Hospital, Herlev, Denmark (Z.S.); and Department of Clinical 
      Medicine, University of Copenhagen, Copenhagen, Denmark (Z.S.).
FAU - Venkadesh, Kiran V
AU  - Venkadesh KV
AUID- ORCID: 0000-0002-4846-9049
AD  - From the Department of Medical Imaging, Radboudumc, Route 767, Room 2.30, Geert 
      Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands (N.A., J.S.B., K.V.V., 
      C.J., M.d.R., J.H., H.H.); Department of Medicine, Section of Pulmonary Medicine, 
      Herlev-Gentofte Hospital, Herlev, Denmark (Z.S.); and Department of Clinical 
      Medicine, University of Copenhagen, Copenhagen, Denmark (Z.S.).
FAU - Jacobs, Colin
AU  - Jacobs C
AUID- ORCID: 0000-0003-1180-3805
AD  - From the Department of Medical Imaging, Radboudumc, Route 767, Room 2.30, Geert 
      Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands (N.A., J.S.B., K.V.V., 
      C.J., M.d.R., J.H., H.H.); Department of Medicine, Section of Pulmonary Medicine, 
      Herlev-Gentofte Hospital, Herlev, Denmark (Z.S.); and Department of Clinical 
      Medicine, University of Copenhagen, Copenhagen, Denmark (Z.S.).
FAU - Saghir, Zaigham
AU  - Saghir Z
AUID- ORCID: 0000-0002-2693-8617
AD  - From the Department of Medical Imaging, Radboudumc, Route 767, Room 2.30, Geert 
      Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands (N.A., J.S.B., K.V.V., 
      C.J., M.d.R., J.H., H.H.); Department of Medicine, Section of Pulmonary Medicine, 
      Herlev-Gentofte Hospital, Herlev, Denmark (Z.S.); and Department of Clinical 
      Medicine, University of Copenhagen, Copenhagen, Denmark (Z.S.).
FAU - de Rooij, Maarten
AU  - de Rooij M
AUID- ORCID: 0000-0001-7257-7907
AD  - From the Department of Medical Imaging, Radboudumc, Route 767, Room 2.30, Geert 
      Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands (N.A., J.S.B., K.V.V., 
      C.J., M.d.R., J.H., H.H.); Department of Medicine, Section of Pulmonary Medicine, 
      Herlev-Gentofte Hospital, Herlev, Denmark (Z.S.); and Department of Clinical 
      Medicine, University of Copenhagen, Copenhagen, Denmark (Z.S.).
FAU - Hermans, John
AU  - Hermans J
AUID- ORCID: 0000-0001-9207-0548
AD  - From the Department of Medical Imaging, Radboudumc, Route 767, Room 2.30, Geert 
      Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands (N.A., J.S.B., K.V.V., 
      C.J., M.d.R., J.H., H.H.); Department of Medicine, Section of Pulmonary Medicine, 
      Herlev-Gentofte Hospital, Herlev, Denmark (Z.S.); and Department of Clinical 
      Medicine, University of Copenhagen, Copenhagen, Denmark (Z.S.).
FAU - Huisman, Henkjan
AU  - Huisman H
AUID- ORCID: 0000-0001-6753-3221
AD  - From the Department of Medical Imaging, Radboudumc, Route 767, Room 2.30, Geert 
      Grooteplein Zuid 10, 6525 GA, Nijmegen, the Netherlands (N.A., J.S.B., K.V.V., 
      C.J., M.d.R., J.H., H.H.); Department of Medicine, Section of Pulmonary Medicine, 
      Herlev-Gentofte Hospital, Herlev, Denmark (Z.S.); and Department of Clinical 
      Medicine, University of Copenhagen, Copenhagen, Denmark (Z.S.).
LA  - eng
PT  - Journal Article
PT  - Multicenter Study
PT  - Research Support, Non-U.S. Gov't
PL  - United States
TA  - Radiology
JT  - Radiology
JID - 0401260
SB  - IM
EIN - Radiology. 2023 Oct;309(1):e239023. PMID: 37906017
MH  - Male
MH  - Humans
MH  - Artificial Intelligence
MH  - Retrospective Studies
MH  - *Mental Disorders
MH  - Magnetic Resonance Imaging
MH  - *Lung Neoplasms/diagnostic imaging
MH  - Tomography, X-Ray Computed
EDAT- 2023/09/19 17:42
MHDA- 2023/09/21 06:42
CRDT- 2023/09/19 10:02
PHST- 2023/09/21 06:42 [medline]
PHST- 2023/09/19 17:42 [pubmed]
PHST- 2023/09/19 10:02 [entrez]
AID - 10.1148/radiol.230275 [doi]
PST - ppublish
SO  - Radiology. 2023 Sep;308(3):e230275. doi: 10.1148/radiol.230275.