Catalog of Regulatory Science Tools to Help Assess New Medical Devices
This regulatory science tool presents a method QuCAD to quantify the wait-time-saving benefits of radiological computer-aided triage and notification (CADt) devices in various simulated clinical environments.
Technical Description
QuCAD simulates the flow of patient images in a radiology department. The tool considers disease prevalence, image arrival rate to the reading queue, radiologists’ reading rates for different subgroup of patients, number of radiologists reviewing images, and the presence of interrupting (cut-in-line) cases that require a radiologist’s immediate attention. Simulated patients are simultaneously placed in two arms of study: one without the CADt device (i.e. standard of care) and one with a CADt device operating at a given sensitivity and specificity threshold. The wait-time-saving performance metric of the CADt is the time difference in waiting time for positively diagnosed studies between the two study arms.
It should be noted that although this work is motivated by the need for evaluating CADt devices, simulation and calculations presented in this tool can be applied to any algorithm that prioritizes customers based on its binary outputs.
The core of this software is a python-based, standalone application that can be used on a command line.
- The input of the tool is a configuration file that defines the clinical settings (see example config.dat in the GitHub QuCAD repository [4]).
- The output of this tool includes a python pickled dictionary file and a set of plots that show the distributions and theoretical predictions on the patients’ wait-time.
Additional information on the inputs and outputs, including example code snippets, can be found in User Manual in the GitHub QuCAD repository [4].
Intended Purpose
QuCAD is intended to evaluate and assess the potential wait-time-saving effectiveness of a CADt device. CADt devices use artificial intelligence (AI) to analyze patient images and prioritize cases with suspected findings in a radiologist’s reading queue. The key benefit of CADt devices is to save patients’ waiting time for time-sensitive conditions such as stroke.
In addition to the CADt’s sensitivity and specificity, the special controls for CADt require a demonstration of effective triage. An understanding of the time saved in realistic situations with the use of the device is therefore important to provide evidence of effective triage.
QuCAD provides a method to estimate the device’s wait-time-saving benefit for a range of simulated clinical settings. By understanding the wait-time-savings along the ROC curve of a CADt device, device developers may optimize the operating point of their AI software based on the potential time-savings. This software can also be used by clinical experts to investigate the impacts on wait-time-saving performance in their clinics based on their flow of patient images and radiologists’ performance.
Related Product Codes
- QAS
- QFM
Testing
QuCAD consists of two independent parts: simulation and theory. Results using a wide range of input parameters are cross checked between simulation and theory to ensure that the time difference between scenarios with CADt and scenarios without CADt are validated for different subgroups of patient images.
The agreement between simulated and theoretical results was tested for a wide range of user input parameters, such as disease prevalence, CADt sensitivity and specificity operating point, image arrival rates and radiologists’ reading rates. For a given set of user inputs, QuCAD simulates the patient image flow with and without the CADt and records the averaged wait-time differences for the different subgroups of patient images, including diseased patient images. Independent of simulation, QuCAD is also capable of computing the mean wait-time differences for different patient image subgroups using Markov Chain-based queueing theory [1]. For a wide range of user inputs, simulated and theoretical results agree with each other [2, 3], which validates the outputs of QuCAD software tool.
Besides cross-checking between simulation and theory, the software has been evaluated using clinical data. The input parameters that characterize the queueing system of a radiology department in a tertiary medical center were extracted and compared with the output from the QuCAD software tool.
Limitations
QuCAD is currently limited to a single disease condition, i.e. a reading list only consists of images with and without the single time-sensitive disease condition. All images in the queue are analyzed by the CADt device, although a clinical radiological worklist may include images with other urgent disease conditions, images that are not analyzed by the CADt, and/or images that may be analyzed by multiple CADt devices.
QuCAD is currently designed for programmers without a graphical user interface (GUI). Inputs and outputs of the tool are documented in detail in the User Manual in the GitHub QuCAD repository [4].
Supporting Documentation
Tool website:
References
Harchol-Balter, M., Osogami, T., Scheller-Wolf, A., & Wierman, A. (2005). Multi-server queueing systems with multiple priority classes. Queueing Systems, 51(3–4), 331–360. doi:10.1007/s11134-005-2898-7 Thompson, Y. L. E., Levine, G. M., Chen, W., Sahiner, B., Li, Q., Petrick, N., … Samuelson, F. W. (2023). Evaluation of wait time saving effectiveness of triage algorithms. doi:10.48550/ARXIV.2303.07050 Thompson, Y. L. E., Levine, G., Chen, W., Sahiner, B., Li, Q., Petrick, N., & Samuelson, F. W. (2022, April 4). Wait-time-saving analysis and clinical effectiveness of Computer-Aided Triage and Notification (CADt) devices based on queueing theory. In C. R. Mello-Thoms & S. Taylor-Phillips (Eds.), Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment. doi:10.1117/12.2603184 https://github.com/DIDSR/QuCAD