U.S. flag An official website of the United States government

U.S. Food and Drug Administration

iMRMC: Software to do Multi-reader Multi-case Statistical Analysis of Reader Studies

This regulatory science tool presents a computer model to assist investigators with analyzing and sizing multi-reader multi-case (MRMC) reader studies that compare the difference in the area under Receiver Operating Characteristic curves (AUCs) from two modalities.

Technical Description

The iMRMC application is a statistical software package that includes simulation tools to characterize bias and variance of the MRMC variance estimates.

The core elements of this application include the ability to perform MRMC variance analysis and the ability to size an MRMC trial.

  • There is an iMRMC R package (v2.1.0) entirely written in R to perform the main MRMC study analysis and has additional U-statistics analysis procedures. Examples for using the programs can be found in the R help files.
  • There is a legacy iMRMC java application (v4.0.3) that is a stand-alone, precompiled, license-free Java application and the source code. It can be used in GUI mode or on the command line.
  • Additional functionality of the GitHub package includes examples to guide users on how to perform noninferiority studies and analyses of binary data.
  • There are also functions to simulate MRMC agreement data and do MRMC analyses of limits of agreement. Examples for using these functions are shown in R markdown files.   

The software treats arbitrary study designs that are not "fully-crossed."

Intended Purpose 

The iMRMC package analyzes data from Multiple Readers and Multiple Cases (MRMC) studies, which are often imaging studies where clinicians (readers) evaluate patient images (cases). The MRMC methods may apply to any scenario in which clinicians interpret data to make clinical decisions. The iMRMC package calculates the reader-averaged area under the receiver operating characteristic curve: the AUC of the ROC curve. AUC is a diagnostic performance measure. Additional functions analyze other endpoints (binary performance, score differences, and limits of agreement). This package also estimates variances, confidence intervals and p-values. These uncertainty characteristics are needed for hypothesis tests to size and assess the efficacy of diagnostic imaging devices and computer aids (artificial intelligence).

The analysis is important because imaging studies are designed so that every reader reads every case in all modalities, a fully-crossed study. In this case, the data is cross-correlated, and the readers and cases are considered to be cross-correlated random effects. An MRMC analysis accounts for the variability and correlations from the readers and cases when estimating variances, confidence intervals, and p-values. The functions in this package can treat arbitrary study designs and studies with missing data, not just fully-crossed study designs.

The methods in the iMRMC package are not standard. The package permits industry statisticians to use a validated statistical analysis method without having to develop and validate it themselves.

Related FDA Product Codes

This tool may be used in the product development phase for the assessment of device performance of devices that fall into the following FDA product codes, but may be also appropriate in other cases:

  • KPS: System, Tomography, Computed, Emission
  • LLZ: System, Image Processing, Radiological
  • PAA: Automated Breast Ultrasound
  • POK: Computer-Assisted Diagnostic Software For Lesions Suspicious For Cancer
  • QDQ: Radiological Computer Assisted Detection/Diagnosis Software For Lesions Suspicious For Cancer
  • QBS: Radiological Computer Assisted Detection/Diagnosis Software For Fracture
  • QPN: Software Algorithm Device To Assist Users In Digital Pathology
  • QNP: Gastrointestinal lesion software detection system

Testing

The tool has been characterized through simulations (bias and variance of the estimates) and has been compared with other methods as appropriate for the task.

The following peer-reviewed research includes the detailed verification methods and results

  • Gallas, B. D., Chen, W., Cole, E., Ochs, R., Petrick, N., Pisano, E. D., Sahiner, B., Samuelson, F. W., & Myers, K. J. (2019). Impact of prevalence and case distribution in lab-based diagnostic imaging studies. Journal of Medical Imaging, 6(1), 015501. https://doi.org/10.1117/1.JMI.6.1.015501
    • Desc: Study that uses the software and related research methods and study designs in a large study. Supplementary materials include data and scripts to reproduce study results.
  • Gallas, B. D. (2006). One-shot estimate of MRMC variance: AUC. Acad Radiol, 13(3), 353–362. https://doi.org/10.1016/j.acra.2005.11.030
    • Desc: Original description of method and validation with simulations. Results comparable to jackknife resampling technique.
  • Gallas, B. D., Pennello, G. A., & Myers, K. J. (2007). Multireader multicase variance analysis for binary data. Journal of the Optical Society of America. A, Optics, Image Science, and Vision, 24(12), B70-80. https://doi.org/10.1364/josaa.24.000b70
    • Generalize method to binary performance measures.
  • Gallas, B. D., Bandos, A., Samuelson, F., & Wagner, R. F. (2009). A framework for random-effects ROC analysis: Biases with the bootstrap and other variance estimators. Commun Stat A-Theory, 38(15), 2586–2603. https://doi.org/10.1080/03610920802610084
    • Provide framework for understanding method and comparing to other methods analytically and with simulations.
  • Gallas, B. D., & Brown, D. G. (2008). Reader studies for validation of CAD systems. Neural Networks Special Conference Issue, 21(2), 387–397. https://doi.org/10.1016/j.neunet.2007.12.013
  • Chen, W., & Samuelson, F. W. (2014). The average receiver operating characteristic curve in multireader multicase imaging studies. The British journal of radiology, 87(1040), 20140016. https://doi.org/10.1259/bjr.20140016
  • Wen, S., & Gallas, B. D. (2022). Three-way mixed effect ANOVA to estimate MRMC limits of agreement. Statistics in Biopharmaceutical Research, 14(4), 532-541. https://doi.org/10.1080/19466315.2022.2063169
    • Desc: Description of MRMC limits of agreement method and MRMC agreement data simulation model. The simulation study is “fully-crossed”.
  • Wen, S., & Gallas, B. D. (2023). Expanding to Arbitrary Study Designs: ANOVA to Estimate Limits of Agreement for MRMC Studies. arXiv preprint arXiv:2312.16097. https://doi.org/10.48550/arXiv.2312.16097
    • Desc: Expand the MRMC limits of agreement analysis to arbitrary study designs.

Limitations

Currently, the tool can produce negative variance estimates if the relevant dataset is small. There is no confidence interval estimate for the MRMC limits of agreements.

Supporting Documentation

Tool websites:

User manual for R package

FAQs

Supplementary materials

Related Work

  • Chen, W., Gong, Q., Gallas, B.D. (2018). Paired split-plot designs of multireader multicase studies. Journal of Medical Imaging 5, 031410. https://doi.org/10.1117/1.JMI.5.3.031410
  • Obuchowski, N.A., Gallas, B.D., Hillis, S.L. (2012). Multi-Reader ROC studies with Split-Plot Designs: A Comparison of Statistical Methods. Acad Radiol 19, 1508– 1517. https://doi.org/10.1016/j.acra.2012.09.012
  • Gallas, B.D., Chan, H.-P., D’Orsi, C.J., Dodd, L.E., Giger, M.L., Gur, D., Krupinski,
  • E.A., Metz, C.E., Myers, K.J., Obuchowski, N.A., Sahiner, B., Toledano, A.Y., Zuley, M.L. (2012). Evaluating imaging and computer-aided detection and diagnosis devices at the FDA. Acad Radiol 19, 463–477. https://doi.org/10.1016/j.acra.2011.12.016
  • Obuchowski, N. A., Gallas, B. D., & Hillis, S. L. (2012). Multi-Reader ROC studies with Split-Plot Designs: A Comparison of Statistical Methods. Academic Radiology, 19(12), 1508–1517. https://doi.org/10.1016/j.acra.2012.09.012
  • Gallas, B. D., & Hillis, S. L. (2014). Generalized Roe and Metz ROC model: Analytic link between simulated decision scores and empirical AUC variances and covariances. J Med Img, 1(3), 031006. https://doi.org/doi:10.1117/1.JMI.1.3.031006

Contact

Tool Reference