Infrared Thermographs- Methods for Evaluating Laboratory Performance | Center for Devices and Radiological Health

Catalog of Regulatory Science Tools to Help Assess New Medical Devices

This regulatory science tool presents a lab method for objective, quantitative assessment of infrared thermographs (IRTs) performance based on IEC 80601-2-59: 2017 [1]. Please see appendix.

Technical Description

Methods for evaluating stability and drift, image uniformity, minimum resolvable temperature difference, and radiometric temperature laboratory accuracy of IRTs are described in detail in peer-reviewed publication [2].

IRTs have been used for initial temperature assessment and triage of individuals for elevated temperatures in medical and non-medical environments. To achieve good accuracy, qualified IRT systems should be used, and appropriate procedures should be consistently applied. This tool provides methods for testing IRT laboratory accuracy which can enhance the methods in the international standard [1].

The tool may improve the utility of IRTs and aid in comparing IRT performance, thus improving the potential for producing high-quality countermeasures during infectious disease outbreaks.

Intended Purpose

The tool is intended for evaluation of IRT laboratory accuracy. It enables, streamlines, or otherwise improves verification, validation and quality control during design, development and manufacturing of IRTs. The testing described in this tool should be performed by persons with engineering background in thermography.

Testing

The method has been tested on two infrared cameras (A325sc, FLIR Systems Inc.; 8640 P-series, Infrared Cameras Inc.). Details are described in our publication [2].

Limitations

This tool is for IRTs that capture whole face thermal images. Therefore, there are requirements for camera configuration (for example, pixel number).
This tool can only be used to evaluate IRT laboratory accuracy. A complete assessment of IRT performance should also include clinical accuracy evaluation as described in a publication [3]
This tool is not intended for evaluating laboratory accuracy of some temperature scanner kiosks that identify the region of interest (ROI) with a visible light camera and to measure the ROI temperature with an infrared sensor with limited number of pixels. However, methods for evaluating those devices might be derived from this tool.

Supporting Documentation

This tool enhances the test methods found in the IEC 80601-2-59: 2017 standard [1] for evaluating the stability and drift, image uniformity, minimum resolvable temperature difference, and radiometric temperature laboratory accuracy of IRTs. The standard provides an overview of test methods as well as background information such as scope, definitions, testing equipment, procedure, and hazard.
Scientific details about the tool can be found in our publication [2].
A comparison of the enhanced methods provided by this tool with the methods specified in the standard can be found in the appendix.

Contact

RST_CDRH@fda.hhs.gov

For more information:

Catalog of Regulatory Science Tools to Help Assess New Medical Devices

Note: This tool was previously listed in the catalog as “Laboratory accuracy evaluation of infrared thermographs.”

Appendix: Method for evaluating laboratory accuracy of infrared thermographs

Acronym List

Acronym	Full name
ASTM	American Society for Testing and Materials
BB	blackbody
CS	calibration source (usually an extended area blackbody)
ETRS	external temperature reference source (usually a blackbody with known temperature and emissivity), often part of a ST
FOV	field of view
IEC	International Electrotechnical Commission (standards organization)
IR	Infrared
IRT	IR thermograph (also called IR camera), often works with an ETRS
ISO	International Organization for Standardization (standards organization)
MRTD	minimum resolvable temperature difference
NCIT	non-contact IR thermometers
ROI	region of interest
SD	standard deviation
ST	screening thermograph, often composed of an IRT and an ETRS
WTP	workable target plane (can be equal or less than the IRT’s FOV)

Symbol List

Symbol	Definition
u_D	standard uncertainty of the measurement drift
u_S	standard uncertainty of the measurement stability
u_U	standard uncertainty of the measurement uniformity of the WTP
u_ER	standard uncertainty of the ETRS temperature
u_MRTD	standard uncertainty caused by the MRTD
u_ST	uncertainty of the ST u_ST^2=u_D^2+ u_S^2+ u_U^2+ u_ER^2+u_MRTD^2
u_CS	standard uncertainty of the CS used in calibration of the ST
u	combined standard uncertainty of the laboratory accuracy u^2=u_CS^2+ u_ST^2
T_ST	temperature of the CS measured by the ST
T_CS	temperature of the CS
M_8h	mean value for 8 hours measurement
M_8h,min , M_8h,max	minimum and maximum of the mean value for 8 hours measurement
SD_8h	standard deviation for 8 hours of measurement

Setup, device, and environmental requirements:

Minimum WTP resolution in thermograms: 320 × 240 pixels
Minimum resolution for the subject's face in thermograms: 240 × 180 pixels
Minimum ETRS resolution in thermograms: 20 × 20 pixels
Minimum spatial resolution of ~1 mm /pixel (i.e., one pixel can image an area of 1×1mm2 on the face)
Ambient temperature: 18°C – 24°C
Relative humidity: 10% – 75%
Airflow from ventilation ducts should be deflected to minimize forced cooling or heating of the target.
No source of IR radiation (for example, incandescent and halogen lighting) surrounds the experimental setup.

Comparison:

Our study External Link Disclaimer [2] evaluated and enhanced some requirements and test method in the IEC 80601-2-59:2017 standard [1]. The following table compares the differences between the standard methods and our proposed methods.

		IEC 80601-2-59: 2017	Enhancements of RST
Stability and drift	Test procedure	See clause 201.101.4. The testing period should be the device's calibration interval or two weeks, whichever is longer.	Same as the standard. Detailed instruction is provided in Section 2.3.1. The testing period should be the device's calibration interval or one month, whichever is longer.
	Requirements	Stability: 3·〖SD〗_8h < 0.1°C Drift: M_(8h,max)-M_(8h,min)<0.1°C The combined stability and drift: Stability + Drift<0.2°C.	Stability: Standard uncertainty u_s=〖SD〗_8h <0.03°C, or Expanded uncertainty U_s=k〖·u〗_s<0.1°C, with coverage factor k=3 and confidence level >99%. Drift: Standard uncertainty 〖u_d=M〗_(8h,max)-M_(8h,min)<0.1°C
	Notes	Following the standard, we calculated the mean (M_frame) of the ROI on each captured frame, followed by the mean (M_8h) and SD (〖SD〗_8h) of all the M_frame values within one day (8 hours). M_(8h,max) and M_(8h,min) are the maximum and minimum M_8h values during the drift testing period. Since the drift trend can be monotonical, drift is the maximum difference between M_8h values during the testing period (i.e., the device's calibration interval or two weeks, whichever is longer). Calculation of M_frame will reduce random noise by a factor of √N, where N is the number of pixels in the ROI for mean calculation. The random noise is considered in the uniformity test. Based on our recommendations, the requirement for combined stability and drift is redundant and can be removed.
Uniformity of WTP	Test procedure	See Clause 201.101.6. The CS is repositioned at no less than 29 locations (four corners, center, and at least 24 random locations) throughout the WTP, and the CS temperature was measured with the IRT at each location.	See Section 2.3.2 for details. The CS is repositioned in front of the IRT so that the CS aperture can cover the entire FOV of the IRT. The camera should be focused on the working distance for uniformity evaluation to avoid the effect of focal length. The SD of temperature values from all the pixels within the WTP is defined as the standard uncertainty of uniformity (u_U)
	Requirements	The maximum difference between the 29 CS temperature values is less than 0.2°C.	u_{U <0.05°C}
	Notes	The standard does not explain whether the CS temperature should be read from one pixel or the average of several pixels. Since only 29 values are used, the test result has poor repeatability and large variation.
MRTD	Test procedure	See Clause 201.101.5. Complies with the ASTM E1213-14 standard External Link Disclaimer [4] A standard four-bar target is mounted in front of the CS (a differential blackbody). The target is imaged with the IRT and displayed on a video monitor where the image is viewed by an observer. The temperature difference (ΔT) between the bars and their conjugates, initially zero, is increased incrementally only until the observer can distinguish the four bars. This critical temperature difference is the MRTD for this observer. Repeat the procedure for multiple observers and report the median MRTD values for all the observer as the final MRTD result. The ASTM E1213-14 standard has no requirement for the spatial frequency and background temperature of the bar target, but requires these values be reported. Monitor quality and setting (e.g., brightness and contrast) and viewing environment might affect results.	See Section 2.3.3 for details. Same setup with the ASTM E1213-14 method. Instead of using observers, the contrast of the target bar images is digitally analyzed directly. The minimum visible contrast is defined as 10% and the ΔT value corresponding to the 10% contrast is defined as MRTD. MRTD can be affected by the target/ambient temperature. Since human face temperature usually ranges from 34 °C to 36 °C, we recommend the bar target temperature for MRTD measuring is set within this range. The MRTD values in the horizontal and vertical directions might be different and should be measured in both directions. MRTD will increase with the bar target spatial frequency and should be evaluated with different bar targets of different spatial frequencies. A graph of MRTD versus spatial frequency is a convenient form of reporting the data. The highest spatial frequency should at least be 0.2 cycles/mrad. The MRTD uncertainty (u_MRTD) is defined as the difference of MRTD values between the highest and lowest target spatial frequencies. The final u_MRTD value is the average of u_MRTD values in both the vertical and horizontal directions.
	Requirements	MRTD is not larger than 0.1°C	MRTD at the highest spatial frequency (≥0.2 cycles/mrad) is not greater than 0.1°C.
	Notes	The ASTM E1213-14 method is subjective, and thus susceptible to reader variability, which is not ideal for standardization. By omitting observers, our proposed method significantly simplifies the MRTD measurement. The ASTM E1213-14 method does not specify MRTD value at which spatial frequency should satisfy the requirement. The method does not discuss the effect of target temperature and direction either.
Radiometric temperature laboratory accuracy	Requirements	The measured radiometric temperature laboratory accuracy shall satisfy \|T_ST-T_CS \|+\|u\|≤0.5 over the range of at least 34°C to 39°C at no less than five CS temperature points.
CS emissivity	Requirements	≥0.998 (Annex BB)	0.98±0.01
CS emissivity	Note	It is difficult to find a commercial blackbody with emissivity ≥0.998 in the temperature range of 30°C to 40°C (clause 201.101.2.1). While some cavity blackbodies have emissivity around 0.99, they usually operate at temperature higher than 50°C. We were unable to identify a justification for the specified emissivity in the standard (references in the standard do not mention an emissivity of 0.998). We believe CS emissivity of 0.98±0.01 should be sufficient if proper emissivity compensation is applied in the IRT algorithm.
ETRS size	Requirements	Less than 10% of the face (Annex AA)	A larger size (15±20%) is also acceptable if it can be experimentally proven that the ETRS does not adversely affect the measurement.

Note: Contents in this appendix are based on references [1, 2].

References:

[1] IEC 80601-2-59: Medical electrical equipment - Part 2-59: Particular requirements for the basic safety and essential performance of screening thermographs for human febrile temperature screening External Link Disclaimer, IEC, Geneva, Switzerland, 2017.

https://www.youtube.com/watch?v=SbVc5YyMRAY

[2] P. Ghassemi, T. J. Pfefer, J. P. Casamento, R. Simpson, and Q. Wang, "Best practices for standardized performance testing of infrared thermographs intended for fever screening External Link Disclaimer," PLoS ONE, vol. 13, no. 9, p. e0203302, 2018.

[3] Q. Wang, Y. Zhou, P. Ghassemi, D. McBride, J. P. Casamento, and T. J. Pfefer, "Infrared Thermography for Measuring Elevated Body Temperature: Clinical Accuracy, Calibration, and Evaluation External Link Disclaimer," Sensors, vol. 22, p. 215, 2022.

[4] ASTM E1213-14: Standard Practice for Minimum Resolvable Temperature Difference for Thermal Imaging Systems External Link Disclaimer, ASTM, West Conshohocken, PA, USA, 2014.