U.S. flag An official website of the United States government

U.S. Food and Drug Administration

A "threshold-based" Approach to Determining an Acceptance Criterion for Computational Model Validation

Catalog of Regulatory Science Tools to Help Assess New Medical Devices 

 

This regulatory science tool presents a computer model that offers a “threshold-based” validation method, which servers a means to determine an acceptance criterion for computational models.

 

Technical Description

A “credible” computational model has the potential to provide a meaningful evaluation of safety in medical-device submissions [1,2]. The ASME V&V 40 standard [3] lists and discusses various credibility assessment factors that are essential to support model credibility. This standard does not, however, provide a mechanism to determine if and when the difference between the computational model and experimental results can be deemed acceptable. This mechanism is a key step in the credibility assessment process for computational models. 

This RST provides a way to determine a well-defined acceptance criterion for comparison error (the difference between the simulation results and validation experiments.) The threshold approach for establishing validation criteria is applicable for situations where threshold values for safety/performance are available for the quantity of interest. 

A detailed description of this approach is provided here.   

Intended Purpose 

The RST is intended to be used in conjunction with other, established methods of verification and validation [2,3,4,5]. This approach is intended for scenarios where a well-accepted safety/performance criterion for the specific Context of Use (COU) is available.  The RST is applicable to all computational models intended to provide evidence of safety in medical devices. The inputs to the RST are the mean values and uncertainties in the validation experiments, the model predictions, and the safety thresholds. The output of the model is a measure of the level of confidence that the model is sufficiently validated from a safety perspective.

Testing

The usage of this RST has been demonstrated in a study published in a peer-reviewed article by Hariharan et al., 2017. The study involved the comparison of computational fluid dynamics (CFD) results with experiments, with the goal of assessing the potential for blood damage. Additional details about the use of this RST can be obtained in the article by Hariharan et al., 2017.

Limitations

  1. While applying the threshold approach in isolation without other form of verification or validation there is a risk of classifying an inaccurate model as “valid”, in the sense that it produces values that are inaccurate but harmless.  However, this risk should be minimal if other forms of verification (e.g. comparison with established model results) and validation are included.
  2. Any significant change in the question of interest or the COU requires a new set of validation metrics or even new validation thresholds.
  3. The credibility of the threshold-based validation method is dependent on the accuracy of the threshold value. 

Additional details about this RST limitations can be obtained in the discussions section of the article by Hariharan et al., 2017.

Supporting Documentation

  1. FDA Guidance Document “Reporting of Computational Modeling Studies in Medical Device Submissions”.
  2. FDA Guidance Document “Assessing the Credibility of Computational Modeling and Simulation in Medical Device Submissions”.
  3. American Society of Mechanical Engineers. V&V 40–2018 Assessing Credibility of Computational Modeling through Verification and Validation: Application to Medical Devices. New York, NY: ASME; 2018.
  4. American Society of Mechanical Engineers. ASME V&V 20–2009—Standard for Verification and Validation in Computational Modeling of Fluid Dynamics and Heat Transfer. New York: ASME; 2009.
  5. American Society of Mechanical Engineers. V&V 10–2006 Guide for Verification and Validation in Computational Solid Mechanics. New York, NY: ASME; 2006.

Contact

Tool Reference

  • In addition to citing relevant publications please reference the use of this tool using RST24CM03.01