Catalog of Regulatory Science Tools to Help Assess New Medical Devices
This regulatory science tool presents a method (web-based decision tree) that may help developers select appropriate metric and endpoint for Artificial Intelligence (AI) / Machine learning (ML) classification algorithms in medical imaging.
Technical Description
The MIC-MET Tree tool is a web-based decision tree where users input the task context, which includes information on the task type, reference standard nature, and algorithm/device output type. Based on these inputs, the tool provides a list of suitable performance evaluation metrics/approaches. Each metric in the list links to a dedicated web page with metric definitions, and resources such as references, video tutorials, and relevant software links.
Intended Purposes
This tool is intended to aid developers in selecting performance metrics for evaluating AI/ML devices that include medical imaging-based classification outputs.
The tool is designed for Artificial Intelligence / Machine Learning (AI/ML) based medical devices that perform binary or multi-class classification in the field of medical imaging. Examples include devices classifying chest Computed tomography (CT) images as lung nodule present/absent (binary), categorizing anatomical sites in real-time endoscopy video frames (multi-class nominal), or ranking medical conditions as benign, moderate risk, or high risk (multi-class ordinal).
Some Related FDA Product Codes: (This is not an exhaustive list)
- PIB: Diabetic Retinopathy Detection Device
- POK: Computer-Assisted Diagnostic Software for Lesions Suspicious for Cancer
- QAS: Radiological Computer-Assisted Triage and Notification Software
- QAS: Radiological Computer Assisted Detection/Diagnosis Software for Fracture
- QDQ: Radiological Computer Assisted Detection/Diagnosis Software for Lesions Suspicious for Cancer
- QFM: Radiological Computer-Assisted Prioritization Software for Lesions
- QPN: Software Algorithm Device to Assist Users in Digital Pathology
Testing
The validation process for this tool involved information checking and reviews conducted by two key groups 1) experts in the medical imaging field and 2) regulatory experts, as described below.
- An earlier version of this tool, Medical Imaging and Data Resource Center (MIDRC) version, has been validated:
- through cross-checking of the information within the MIDRC Technology and Development Project 3c/3d research group (including FDA participants and experts from the University of Chicago, the University of California Los Angeles, and Puente Solutions),
- through feedback collection from other groups in MIDRC that are independent from the TDP3c/d group (including general AI/ML algorithm users/developers). For details, please see https://www.midrc.org/performance-metrics-decision-tree)
- through live demos at highly influential annual meetings in the medical imaging field, including The American Association of Physicists in Medicine (AAPM) and the international society for optics and photonics (SPIE)-medical imaging conferences to audiences comprising medical imaging experts and AI/ML algorithm users.
- Validation from the target user population: Further validation for the FDA version was done by gathering insights from the internal regulatory experts. The validation efforts included obtaining feedback to ensure its suitability for regulatory application.
Presentations and publications that are most relevant to prior work in MIDRC collaboration regarding this tool can be found in Supporting Documentation.
Limitations
The limitation of this tool includes the following:
- This tool focuses exclusively on medical imaging-based AI/ML devices with a classification function. Other tasks such as segmentation, estimation, and image reconstruction are not covered by this tool.
- AI/ML algorithm development, or how to set a performance goal for the algorithm evaluation is out of the scope of this tool.
Supporting Documentation
Tool web page, User manual, instructions for use can be found on ZingTree Supplementary materials.
- K. Drukker, B. Sahiner, T. Hu, G. H. Kim, H. M. Whitney, N. Baughan, K. J. Myers, M. L. Giger, M. McNitt-Gray, MIDRC-MetricTree: a decision tree-based tool for recommending performance metrics in artificial intelligence-assisted medical image analysis, J. Med. Imag. 11(2), 024504 (2024), doi: 10.1117/1.JMI.11.2.024504.
- K. Drukker, B. Sahiner, T. Hu, G. Hyun Kim, E. Townley, H. M. Whitney, N. Baughan, K. J. Myers, M. L. Giger, M. McNitt-Gray, “The Decision Tree Based Tools Developed by the Medical Imaging and Data Resource Center (MIDRC) Technology Development Project (TDP) 3c Effort”, Proc. SPIE 12467, Medical Imaging 2023: Image Perception, Observer Performance, and Technology Assessment, 124670E (3 April 2023); https://doi.org/10.1117/12.2654250
- K. Drukker, B. Sahiner, T. Hu, G. Hyun Kim, E. Townley, H. M. Whitney, N. Baughan, K. J. Myers, M. L. Giger, M. McNitt-Gray, “The Medical Imaging and Data Resource Center (MIDRC) Technology Development Project (TDP) 3c: Developing Tools to Assist in Task-specific Performance Evaluation for Machine Learning Algorithms Employing MIDRC Data”, AAPM 2022, July. https://w4.aapm.org/meetings/2022AM/programInfo/programAbs.php?sid=10678&aid=65039