Catalog of Regulatory Science Tools to Help Assess New Medical Devices
Technical Description
The Whole Slide Image Processing and Performance Assessment Tool contains three main modules that (1) accept WSIs to generate image patches for AI/ML models, (2) accept image patches to generate an Aperio ImageScope annotation file for pathologist review of AI/ML model results on ImageScope, and (3) accept outputs of AI/ML binary-classification models to generate performance results in terms of ROC (receiver operating characteristic) curve, area under the curve and other binary metrics (sensitivity, specificity, precision, recall, F-1 score) and their confidence intervals.
The tool provides the following components:
- WSI handler for whole slide image processing and analysis
- Extraction of pathologist annotations and extraction of image patches from the annotated areas of the whole slide images
- Mapping the Region of Interests (ROIs) into the corresponding WSI to be viewed by Aperio ImageScope visualizer for the pathologist validation
- Performance assessment of the ML results and statistical analysis
Intended Purpose
The Whole Slide Image Processing and Performance Assessment Tool is designed to offer functionalities to AI/ML developers to accelerate their processing pipelines and to provide a set of tools for validation and performance evaluation. The tool converts the whole slide image and corresponding annotation file into image patches ready for input into AI/ML models. It also maps ML-detected ROI data back into the WSI to be viewable by Aperio ImageScope application for pathologist review. Finally, it does performance assessment for binary classification machine learning models. The intended users are researchers in Digital Pathology, as well as device developers who wish to work with histopathology whole slide images.
Testing
The tool has been tested to verify each module works as intended. The outputs of each function have been validated manually, and the visualization and mapping of patches to WSI functions have been verified by cross-checking with the commercial software ImageScope. For a given whole-slide image and its corresponding annotation, ValidPath generates image patches ready for input into AI/ML models [1]. For several AI/ML tasks, ValidPath provides model performance evaluation results in terms of ROC (Receiver Operating Characteristic) curve, area under the curve, and other binary metrics (sensitivity, specificity, precision, recall, F-1 score) along with their confidence intervals [1-3].
Limitations
- The current ROI file generation method only produces results compatible with the Aperio ImageScope Application
- Users require basic Python knowledge to use the tool and basic AI/ML and image processing knowledge to interpret the results.
Supporting Documentation
The documentation is embedded in the tool as a user guide for Python scripts usage. Please refer to the attached PDF file and access online documentation at the following links:
- Online Documentation: https://didsr.github.io/ValidPath/index.html
- Github Page: https://github.com/didsr/ValidPath
[1] Kahaki, S., Hagemann, I. S., Cha, K. H., Trindade, C., Petrick, N., Kostelecky, N., Borden, L. E., Atwi, D., Fung, K. M., & Chen, W. (2024). End-to-end deep learning method for predicting hormonal treatment response in women with atypical endometrial hyperplasia or endometrial cancer. Journal of medical imaging (Bellingham, Wash.), 11(1), 017502. https://doi.org/10.1117/1.JMI.11.1.017502
[2] Kahaki, Seyed, et al. "Supervised deep learning model for ROI detection of atypical endometrial hyperplasia and endometrial cancer on histopathology whole slide images for predicting hormonal treatment response." Medical Imaging 2024: Digital and Computational Pathology, accepted with Proc. of SPIE, 2024.
[3] Kahaki, S., Hagemann, I. S., Cha, K., Trindade, C. J., Petrick, N., Kostelecky, N., & Chen, W. (2023). Weakly Supervised Deep Learning for Predicting the Response to Hormonal Treatment of Women with Atypical Endometrial Hyperplasia: A Feasibility Study. Proceedings of SPIE--the International Society for Optical Engineering, 12471, 124710T. https://doi.org/10.1117/12.2652912
Contact
Tool Reference
- In addition to citing relevant publications please reference the use of this tool using RST24DP02.01