Research Square · Sep 22, 2026
A Reproducible Framework for Auditing the Trustworthiness of Clinical Machine Learning Models
By Deep T
The deployment of machine learning (ML) models for clinical risk prediction is frequently challenged by domain shift when algorithms are transferred across heterogeneous healthcare settings. Conventional evaluation primarily emphasizes discriminative performance, such as the area under the receiver operating characteristic curve (ROC-AUC), which may overlook failures in probability calibration, explainability stability, robustness, and clinical utility. To address these challenges, we present AETHEL, an open-source and reproducible framework for systematically auditing the trustworthiness of clinical ML models. The framework evaluates multiple classification and survival models using harmonized datasets, including the Framingham Heart Study and the National Health and Nutrition Examination Survey (NHANES), through integrated assessment of discrimination, calibration, explainability, robustness, clinical utility, and cross-cohort generalization. Experimental evaluation demonstrates measurable degradation in predictive performance and calibration following cross-cohort transfer. While post-hoc calibration using Platt Scaling and Isotonic Regression improves probability calibration, explainability analyses reveal shifts in feature attribution under covariate drift. To quantify explanation stability, we introduce the Feature Agreement Score (FAS) and Top-k Jaccard Overlap (TkO) metrics, alongside visual clinical consequence analyses based on Decision Curve Analysis. AETHEL provides a unified, reproducible framework for auditing the trustworthiness of clinical machine learning models, supporting transparent evaluation and more informed deployment decisions in clinical settings.