U.S. flag An official website of the United States government

U.S. Food and Drug Administration

Multi-QuCAD: Software to Evaluate Wait-Time-Savings and Delays Due to CADt Devices for Multiple Diseases in Clinical Workflow

Catalog of Regulatory Science Tools to Help Assess New Medical Devices 

Tools that quantify wait-time-saving benefits and delays for patients with positive diagnoses across various disease conditions due to the implementation of multiple Computer-Aided Triage and Notification devices in a radiology workflow.

Technical Description

Multi-QuCAD is a software tool designed to evaluate the benefits of reduced wait times and the potential risks of delays for patients diagnosed with various disease conditions when multiple Computer-Aided Triage and Notification (CADt) devices are integrated into a radiology workflow.

Multi-QuCAD simulates the flow of patient images in a radiology department, accounting for factors such as patients’ disease conditions, the prevalence of each condition, the rate at which images arrive in the reading queue, radiologists’ reading rates for different patient subgroups, the number of radiologists reviewing images, and the presence of interrupting (cut-in-line) cases that require immediate attention. Simulated patients are placed simultaneously into two study arms: one without any CADt device (i.e. standard of care) and the other incorporating one or more CADt devices, each configured to a user-defined sensitivity and specificity threshold. The primary performance metric—wait-time difference—is calculated as the difference in waiting times for signal-present images between the two study arms. 

As an extension of the existing QuCAD Regulatory Science Tool (RST) [1], multi-QuCAD introduces three key differences. First, while QuCAD is limited to workflows with a single CADt device analyzing patient images for a single target disease condition, multi-QuCAD accommodates multiple CADt devices applied to patient images across various conditions. Second, QuCAD always assumes a preemptive-resume scheduling discipline, where radiologists are interrupted to address newly arrived higher-priority cases. In contrast, multi-QuCAD provides the flexibility to choose a non-preemptive discipline, allowing radiologists to complete reading lower-priority images before attending to high-priority cases. Finally, multi-QuCAD considers two distinct protocols for each scheduling discipline: a priority protocol, where all images flagged as suspicious by any CADt are moved to the top of the queue regardless of the condition, and a hierarchical protocol, where suspicious images are further prioritized based on the time sensitivity of their respective disease conditions.

Intended Purpose 

Multi-QuCAD is intended to evaluate and assess the potential benefits of reduced wait times and risks of delays for patients diagnosed with various disease conditions when multiple CADt devices are integrated into a radiology workflow. By simulating patient flow, this RST supports CADt developers in estimating the device’s impact on wait-time reductions and delay across different clinical settings. Clinical experts can also use multi-QuCAD to explore how factors such as patient image flow, AI operating thresholds, and radiologists’ performance may influence wait-time savings and delay in their own clinical practice. 

Testing

The framework for simulating patient image flow and the associated theoretical methods were developed based on QuCAD, which has been verified and validated for single-disease and 1-single CADt scenario [2, 3, 4]. The verification of multi-QuCAD involves four clinical scenarios, each assessing the agreement between simulation results and theoretical wait-time-savings across the full range of user input parameters [5].

Scenario 1: Two disease conditions with one CADt device

Two disease conditions (A and B) are considered in which A is considered as more time-sensitive than B. A CADt is deployed to prioritize patient images suspicious of condition A. For each scheduling discipline (preemptive-resume and non-preemptive), the simulated results of wait-time-savings for patient images with condition A and delay for images with condition B agree with the analytical solutions across a range of traffic intensity, disease prevalence of the two conditions, and the diagnostic performance of the CADt.

Scenario 2: Two disease conditions with two CADt devices

The same two disease conditions (A and B) in Study 1 are considered. However, two CADt devices are deployed: CADt-A to prioritize patient images suspicious of condition A, and CADt-B to prioritize images suspicious of B. For each scheduling discipline and clinical protocols (priority and hierarchical), the simulated results of wait-time-savings for patient images with condition A and that for images with condition B agree with the analytical solutions across a range of traffic intensity, disease prevalence of the two conditions, and the diagnostic performance of the two CADt devices.

Scenario 3: Three disease conditions with two CADt devices

Three disease conditions (A, B, and C) are considered in which A is more time-sensitive than B, and B is more time-sensitive than C. Two CADt devices are deployed: CADt-A to prioritize patient images suspicious of condition A, and CADt-C to prioritize images suspicious of C. For each scheduling discipline and clinical protocols, the simulated results of wait-time-savings for patient images with condition A and C and delay for images with condition B agree with the analytical solutions across a range of traffic intensity, disease prevalence of the three conditions, and the diagnostic performance of the two CADt devices.

Scenario 4: Nine disease conditions with four CADt devices

This scenario simulates a complex workflow in the real world in which patient images in the reading queue may have numerous disease conditions. For each scheduling discipline and clinical protocols, the simulated wait-time-savings and delay for patient images across the nine disease conditions agree with the analytical solutions.

Limitations

For preemptive-resume scheduling discipline with a hierarchical protocol, multi-QuCAD only provides simulation results without analytical solutions for two or more radiologists or non-equal radiologist reading times among different diseased and/or non-diseased patient subgroups. 

In the non-preemptive case (either priority or hierarchical protocol), only simulation results are available without analytical solutions when there is more than one radiologist, a non-zero fraction of interrupting patients, or non-equal radiologist reading times among different diseased and/or non-diseased patient subgroups. 

More information on these limitations is discussed in the Multi-QuCAD User Manual [6].

Supporting Documentation

[1] Thompson, Y. L. E., Zheng, J., Samuelson, F. W. (2023) QuCAD [Source Code] https://github.com/DIDSR/QuCAD/. 

[2] Thompson, Y. L. E., Levine, G., Chen, W., Sahiner, B., Li, Q., Petrick, N., & Samuelson, F. W. (2022, April 4). Wait-time-saving analysis and clinical effectiveness of Computer-Aided Triage and Notification (CADt) devices based on queueing theory. In C. R. Mello-Thoms & S. Taylor-Phillips (Eds.), Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment. doi:10.1117/12.2603184

[3] Thompson, Y. L. E., Levine, G. M., Chen, W., Sahiner, B., Li, Q., Petrick, N., … Samuelson, F. W. (2024). Applying queueing theory to evaluate wait-time-savings of triage algorithms. Queueing Systems, 108(3–4), 579–610. doi:10.1007/s11134-024-09927-w

[4] Thompson, Y.L.E., Fergus, J., Chung, J., Delfino, J.G., Chen, W., Petrick, N., … Samuelson, F.W. (2025). Workflow-Dependent Impact of AI-Triage on Patient Turnaround Time: Real-World Time-Savings and Insights from Model Predictions. Journal of the American College of Radiology, 23(1), 63-73. doi:10.1016/j.jacr.2025.07.033

[5] Mastrianni, M., Deshpande, R., Samuelson, F. W., Thompson, Y. L. E. (2025). A Quantitative Framework to Predict Wait-Time Impacts Due to AI-Triage Devices in a Multi-AI, Multi-Disease Workflow. Pre-Print. doi:10.48550/arXiv.2510.27104

[6] Thompson, Y. L. E., Mastrianni, M., Deshpande, R., Zheng, J., Samuelson, F. W. (2024) multi-QuCAD [Source Code] https://github.com/DIDSR/multi-QuCAD. 

Contact

Tool Reference