Post-hoc reliability assessment and visual explanation for deep learning-based oral mucosal lesion detection

Background Modern deep learning detectors for oral mucosal lesions produce class predictions and softmax confidence scores, but neither reveals which learned visual patterns supported the prediction nor whether it can be trusted. We developed a post-hoc method that, without modifying an already-trained YOLO detector, makes the basis of each prediction explicit through embedding-space analysis and provides a training-distribution-aware estimate of its reliability. Methods A YOLOv11 detector was trained on 3104 intraoral photographs covering oral lichen planus (OLP), proliferative verrucous leukoplakia (PVL)/leukoplakia and oral squamous cell carcinoma (OSCC), supplemented with 464 healthy mucosa images. For every detection, a feature embedding was extracted from the deepest backbone layer, projected to a 2-D manifold using supervised neighborhood component analysis (NCA) and unsupervised uniform manifold approximation and projection (UMAP), and scored against per-class kernel density estimates fitted on correctly predicted training samples. Density scores were compared with softmax confidence using receiver operating characteristic (ROC) analysis, and nearest-neighbor retrieval in the embedding space provided visual explanations of individual predictions. Results Per-class detection F1-scores were 0.88 (OLP), 0.76 (PVL/Leukoplakia), 0.73 (OSCC) and 0.93 (healthy mucosa). Density-based ROC area-under-the-curve (AUC) values ranged from 0.92 to 0.95 across the four classes, with NCA slightly outperforming UMAP. The density score consistently identified low-confidence predictions that softmax did not flag, and the nearest-neighbor mosaic produced visually interpretable explanations of individual predictions. Conclusions Embedding-space density on a frozen YOLO detector provides an interpretable, post-hoc reliability measure for oral mucosal lesion detection. Combined with nearest-neighbor visual retrieval, it complements softmax confidence and supports post-hoc clinical audit without retraining.