How Reliable and Explainable Are Machine Learning Models for Dementia Detection? A Leakage-Aware Evaluation

Reported dementia-classification performance depends on how the label is defined, which clinical information the model sees, and whether participants are separated between training and test data. We examine these choices on OASIS-2 (373 visits, 150 participants) using demographics, education, the Mini-Mental State Examination (MMSE), and three global magnetic resonance imaging (MRI) measures. The outcome is a Clinical Dementia Rating (CDR) above zero at the visit, not diagnosis or conversion. Nine modeling procedures are tuned within five outer and three inner participant-grouped folds, and the procedure is selected inside each training partition. Conditional on this partitioning, the selected procedure reaches a pooled area under the receiver operating characteristic curve (ROC-AUC) of 0.867 (95% participant-bootstrap interval 0.817 to 0.915), accuracy 0.786, sensitivity 0.689, and specificity 0.864. MMSE alone gives mean fold ROC-AUC of 0.85 to 0.86. Against a matched clinical-only comparator, MRI raises logistic-regression ROC-AUC by 0.035 (exploratory interval 0.007 to 0.061), while the four tree-based contrasts have intervals spanning zero. Supplying CDR itself yields perfect accuracy in all five positive controls, and row-level cross-validation raises CatBoost from 0.865 to 0.913; the protocols also differ in stratification and tuning. Held-out XGBoost attributions are dominated by MMSE in every fold and describe model behavior, not validated clinical explanations. The MRI contribution is learner-dependent, and independent-cohort and prospective validation remain necessary.