Catalog of Regulatory Science Tools to Help Assess New Medical Devices
This regulatory science tool presents a method that can be utilized in the development of relevant medical devices to assist in the prediction of traumatic brain injury (TBI) and stroke according to resting electroencephalography (EEG).
Technical Description
This RST contains a set of machine or deep learning algorithms which can be utilized in the development of relevant medical devices to assist in the prediction of traumatic brain injury (TBI) and stroke according to resting electroencephalography (EEG). All algorithms were trained on the Temple University Hospital EEG Corpus (TUEG): a rich archive of 26,846 clinical EEG recordings collected at Temple University Hospital from 2002 – 2017. These algorithms are available for use on any resting EEG data in compliance with the requirements described below and on the GitHub readme file.
Intended Purpose
The RST is intended to assist with clinical assessment of medical devices where classification of resting EEG signals is needed (“Normal”, “TBI”, “Stroke”). This RST is designed for subjects aged between 18 and 65 who do not have any previous history of epilepsy.
This RST may assist in device development through the use of the included EEG preprocessing and feature extraction code and machine learning models that have been trained on a large dataset.
- The feature extraction code can be used in conjunction with the provided machine learning models or separately serving as potential biomarkers of TBI or stroke.
- The included trained classification models can be further trained, modified, or fine-tuned for one’s specific use case.
The provided MATLAB script allows for preprocessing of an EEG signal greater than 3-minutes in length and its prediction as either “Normal”, “TBI”, or “Stroke” based on a training set of resting EEGs from the TUEG.
Additionally, processing and training code is provided to allow the user to retrain the models on device or population specific datasets. The processing tools allow the user to extract the relevant information from the TUEG and preprocess raw EEG in the standard .edf file format.
Testing
Algorithms included in this RST only contain those demonstrating good performance (with AUC larger than 0.76). These algorithms have undergone performance testing through multiple methods, including random shuffling and testing on independent datasets.
Detailed methods of algorithm development and scientific verifications can be found in peer-reviewed publications:
- Caiola M, Babu A, Ye M (2023) EEG classification of traumatic brain injury and stroke from a nonspecific population using neural networks. PLOS Digital Health 2(7): e0000282. https://doi.org/10.1371/journal.pdig.0000282
- Vivaldi, N., Caiola, M., Solarana, K., & Ye, M. (2021). Evaluating Performance of EEG Data-Driven Machine Learning for Traumatic Brain Injury Classification. IEEE transactions on bio-medical engineering, 68(11), 3205–3216. https://doi.org/10.1109/TBME.2021.3062502
Limitations
- This is not a clinical diagnostic tool. All AUC evaluations were conducted using an independent dataset collected from the TUEG and further validation from other sites may be appropriate before use.
- The RST is currently developed based on publicly available patient data in the TUEG. This database has limitations, including the lack of information about the phase and severity of TBI and stroke. Therefore, whenever available, the tool needs to be further validated with data from more homogeneous populations of patients.
- The RST was only trained on subjects that were identified as “normal”, “TBI”, or “stroke” (see publications in Supporting Documentation for detailed methodology) and as such, there was no evaluation on how other diseases or conditions may affect the performance of the classification algorithms.
- Training data was discarded for those younger than 18 and those older than 65, making the model unsuitable for those outside of the 18-65 age range.
- Training data included multiple segments of 3-minute EEG from long continuous recordings. Approaches were taken to only include EEG segments that were most similar to those in the identified cohort (see publications in Supporting Documentation for detailed methodology). To account for sleep and other outliers, only the 50% most similar segments were used for training and the remaining were discarded.
- The classification algorithms use default thresholding to determine which class is selected and should be further optimized for specific sensitivity, specificity, or accuracy requirements.
- The classification prediction is based on the diagnosis noted on the medical record, without comparing with other modalities, e.g., imaging, blood-based markers.
Supporting Documentation
- EMATS - The repository includes several MATLAB functions, scripts, classes, and mat-files. To use, one can download or clone the repository to their local machine and run the main RUN.mlx script. See the Github README file for detailed instructions for use.
- Caiola M, Babu A, Ye M (2023) EEG classification of traumatic brain injury and stroke from a nonspecific population using neural networks. PLOS Digital Health 2(7): e0000282. https://doi.org/10.1371/journal.pdig.0000282
- Vivaldi, N., Caiola, M., Solarana, K., & Ye, M. (2021). Evaluating Performance of EEG Data-Driven Machine Learning for Traumatic Brain Injury Classification. IEEE transactions on bio-medical engineering, 68(11), 3205–3216. https://doi.org/10.1109/TBME.2021.3062502