Acute kidney injury (AKI) remains a significant challenge in patients who are hospitalized, contributing to increased morbidity, mortality, and health care costs (1). Machine learning-based models offer the potential for early AKI detection and timely intervention. However, rigorous external validation is necessary before clinical integration (2).
A recent study published in The New England Journal of Medicine Artificial Intelligence (3), “External Validation of a Commercial Acute Kidney Injury Predictive Model,” evaluated the Epic Risk of HA [Hospital-Acquired]-AKI model in a large health care system. This gradient-boosted ensemble model incorporates demographic, comorbidity, and clinical data to predict HA-AKI.
The study analyzed 39,891 patient encounters over 5 months, demonstrating moderate discrimination with an area under the receiver operating characteristic curve (AUROC) of 0.77 (95% confidence interval [CI], 0.76–0.78) at the encounter level and 0.76 (95% CI, 0.76–0.76) for a 48-hour prediction (Figure, A). The median lead time before HA-AKI onset was 21.6 hours, suggesting a window for early intervention. However, the model exhibited overprediction in high-risk subgroups and poor calibration, particularly for the higher AKI stages, that could impact clinical reliability.
Performance metrics of the Epic Risk of HA-AKI predictive model
Citation: Kidney News 17, 6; 10.62716/kn.000542025
Performance and limitations
The model achieved an area under the precision recall curve (AUPRC) of 0.49 at the encounter level and 0.19 at 48 hours, indicating moderate discrimination but suboptimal precision. Performance varied across subgroups, particularly in patients with higher baseline serum creatinine (Figure, B) or comorbidities such as congestive heart failure, diabetes, and hypertension. The model performed best at lower creatinine levels (AUROC, 0.79 for <0.50 mg/dL) but showed poor discrimination in the 3.50- to 3.99-mg/dL range (AUROC, 0.50), highlighting limitations in patients with high risk.
The model demonstrated better negative predictive value than positive predictive value, suggesting utility in ruling out HA-AKI. However, a high false-positive rate at lower thresholds may lead to unnecessary interventions, raising concerns about alert fatigue. Compared with other HA-AKI predictive models, the Epic Risk model outperformed logistic regression models but underperformed relative to advanced machine-learning models such as neural networks. External validation performance (AUROC, 0.77) was lower than internal validation performance (AUROC, 0.85), highlighting concerns regarding generalizability.
Clinical implications and future directions
Despite moderate predictive ability, previous studies on HA-AKI clinical decision support alerts have shown mixed results (4–7). Some reports indicate that HA-AKI alerts do not reduce dialysis initiation, mortality, or AKI progression. However, nephrotoxin-avoidance alerts and pharmacist involvement have shown potential benefits (8–10). The median lead time of 21.6 hours suggests that structured intervention pathways, rather than standalone alerts, may improve the clinical impact.
Further multicenter validation is needed to determine the model's performance across diverse health care settings, to assess the model's performance on prospective datasets, and to evaluate its performance in clinical settings to investigate its impact on the processes of care and clinical outcomes (2). Future studies should evaluate whether clinician response to model predictions improves outcomes, such as optimizing fluid management and avoiding nephrotoxins (2). More interpretable AI models are required to enhance transparency and trust in predictive tools.
The Epic Risk of HA-AKI model offers moderate predictive ability but is not yet ready for widespread clinical use without further validation. It may be most effective when integrated with structured interventions rather than used as an isolated risk score. This study underscores both the promise and the limitations of commercial AI-based HA-AKI prediction tools.
Footnotes
References
- 1.↑
Alfieri F, et al. Continuous and early prediction of future moderate and severe acute kidney injury in critically ill patients: Development and multi-centric, multi-national external validation of a machine-learning model. PLoS One 2023; 18:e0287398. doi: 10.1371/journal.pone.0287398
- 2.↑
Cheungpasitporn W, et al. Advances in critical care nephrology through artificial intelligence. Curr Opin Crit Care 2024; 30:533–541. doi: 10.1097/MCC.0000000000001202
- 3.↑
Dutta S, et al. External validation of a commercial acute kidney injury predictive model. NEJM AI 2024; 1:AIoa2300099. https://ai.nejm.org/doi/full/10.1056/AIoa2300099
- 4.↑
Hodgson LE, et al. The ICE-AKI study: Impact analysis of a clinical prediction rule and electronic AKI alert in general medical patients. PLoS One 2018; 13:e0200584. doi: 10.1371/journal.pone.0200584
- 5.
Wilson FP, et al. Electronic health record alerts for acute kidney injury: Multicenter, randomized clinical trial. BMJ 2021; 372:m4786. doi: 10.1136/bmj.m4786
- 6.
Park S, et al. Impact of electronic acute kidney injury (AKI) alerts with automated nephrologist consultation on detection and severity of AKI: A quality improvement study. Am J Kidney Dis 2018; 71:9–19. doi: 10.1053/j.ajkd.2017.06.008
- 7.↑
Zhao Y, et al. Effect of clinical decision support systems on clinical outcome for acute kidney injury: A systematic review and meta-analysis. BMC Nephrol 2021; 22:271. doi: 10.1186/s12882-021-02459-y
- 8.↑
Scott J, et al. Acute kidney injury electronic alerts: Mixed methods normalisation process theory evaluation of their implementation into secondary care in England. BMJ Open 2019; 9:e032925. doi: 10.1136/bmjopen-2019-032925
- 9.
Bailey S, et al. Implementation of clinical decision support to manage acute kidney injury in secondary care: An ethnographic study. BMJ Qual Saf 2020; 29:382–389. doi: 10.1136/bmjqs-2019-009932
- 10.↑
Kim K, et al. Real-time clinical decision support based on recurrent neural networks for in-hospital acute kidney injury: External validation and model interpretation. J Med Internet Res 2021; 23:e24120. doi: 10.2196/24120
