An integrated deep learning and large language model for automated knee osteoarthritis grading and reporting from the Vietnam Osteoporosis Study

OBJECTIVES: Knee osteoarthritis (KOA) is a prevalent joint condition affecting multiple structures of the knee. Clinicians need reliable tools for early detection and accurate grading to support timely management, but radiographic grading remains subjective and time-consuming. We aimed to develop a deep learning and large language model (LLM) hybrid system for KOA severity classification and automated clinical reporting. METHODS: We analysed bilateral knee radiographs from 2435 adults (≥40 years) in the Vietnam Osteoporosis Study (VOS), using the Kellgren-Lawrence (KL) scale. Data were split into training (N = 1944) and independent testing (N = 491) sets. A hybrid deep learning pipeline combined knee-region object detection with KL grading, screening and feature localization. Outputs were integrated into a Gemini LLM module to generate structured clinical reports. RESULTS: The hybrid model achieved 79.1% accuracy and AUC 88.8% for five-class KL grading, and 89.5% accuracy (sensitivity: 87.5%, specificity: 90.2%) for KOA screening. The localization module consistently highlighted KOA-relevant features, including osteophytes, joint space narrowing, and subchondral sclerosis. The Gemini module converted model outputs into coherent, structured reports and provided context-aware answers to clinician queries grounded in the imaging results. CONCLUSIONS: We developed a deployable deep learning-Gemini system for KOA assessment. This integrated framework offers an interpretable, automated approach to KOA assessment that may improve screening and decision support in resource-limited settings. DEMO: https://www.youtube.com/watch?v=apXipdeXD5k.