Can AI Assess Clinical Competency?

Author: Rasit Dinc

Introduction

The landscape of medical education is in a constant state of evolution, driven by the dual needs of ensuring the highest standards of patient care and equipping future clinicians with the skills to navigate an increasingly complex healthcare environment. A cornerstone of modern medical training is competency-based medical education (CBME), an approach that prioritizes the demonstration of specific, observable skills and professional behaviors over traditional, time-based training schedules [2]. As this paradigm has matured, a critical question has emerged: How can we assess competency in a way that is objective, scalable, and reliable? Increasingly, the answer appears to lie in the sophisticated capabilities of artificial intelligence (AI).

The Challenge of Traditional Assessment

Traditionally, assessing clinical competency has been a labor-intensive process, heavily reliant on direct observation by faculty, standardized patient encounters (like Objective Structured Clinical Examinations, or OSCEs), and written exams. While valuable, these methods are not without their limitations. Human judgment, for instance, can be prone to variability and unconscious bias, leading to inconsistencies in evaluation [5]. Furthermore, the sheer scale of assessment required in large medical programs presents a significant logistical and financial burden, often taking months to grade and provide feedback [5]. The COVID-19 pandemic further exposed these vulnerabilities by disrupting in-person examinations, creating what some have called a “big vacuum in assessing medical student clinical skills” and accelerating the search for innovative solutions [5].

AI's Emerging Role in Competency Evaluation

Artificial intelligence offers a powerful toolkit to augment and refine the assessment of clinical competency. Its application spans from analyzing procedural skills to evaluating complex cognitive and communicative abilities.

One of the most promising areas is the use of AI to analyze the vast amounts of data generated during training. For example, programs utilizing Entrustable Professional Activities (EPAs)—units of professional practice that can be entrusted to a trainee once sufficient competence is demonstrated—gather extensive narrative feedback and performance scores [1]. AI algorithms, particularly those using natural language processing (NLP), can analyze this data in real-time to identify trainees who may be at risk of falling behind, allowing for timely and targeted intervention [1]. At the University of Minnesota, researchers are using NLP to score thousands of SOAP notes generated by students after standardized patient encounters, reducing a process that once took over a month to a matter of moments and improving the reliability of the assessment [5].

Beyond textual analysis, AI is proving adept at evaluating psychomotor skills. Machine learning models can analyze video recordings of procedures, such as endotracheal intubation or surgical tasks performed in virtual reality simulations, to provide objective, granular feedback on performance [1]. These systems can measure hand motion, efficiency, and adherence to protocols with a level of precision that can surpass human observation, all while reducing faculty workload and standardizing the evaluation process [1].

The Future: A Symbiotic Approach

The integration of AI into clinical competency assessment is not about replacing human educators but empowering them. The goal is to create a “sociotechnical system for learning” where AI acts as a co-pilot, handling the data-intensive, repetitive aspects of evaluation, while faculty focus on mentorship, coaching, and the holistic development of the trainee [4]. AI can analyze nonverbal cues like empathy and eye contact from video-recorded patient interactions, providing objective data points that were previously subject to interpretation [5].

By observing trainees in their actual work environment and combining this with expert feedback and patient outcomes, AI-driven EPA assessments can provide robust, evidence-based decisions about a student's readiness for independent practice [1]. This data-rich approach not only ensures a higher standard of competency but also holds the potential to predict a trainee's future professional behavior and even help match graduates to the most suitable roles for their skill sets [1].

Conclusion

The question is no longer if AI can assess clinical competency, but how we can best integrate it into our educational frameworks. From analyzing narrative feedback to scoring procedural skills in virtual reality, AI provides tools to make assessment more objective, efficient, and insightful. While challenges remain in implementation and validation, the trajectory is clear. By embracing AI as a powerful ally, medical education can enhance the reliability of competency-based assessments, reduce the burden on educators, and ultimately, ensure that the next generation of health professionals is more prepared than ever to deliver safe and effective patient care.

References

[1] Dabbagh, A., Madadi, F., & Larijani, B. (2024). Role of AI in Competency-Based Medical Education: Using EPA as the Magicbox. Archives of Iranian Medicine, 27(11), 633–635. https://pmc.ncbi.nlm.nih.gov/articles/PMC11558609/

[2] Hamza, D. M., Hauer, K. E., Oswald, A., van Melle, E., Ladak, Z., Zuna, I., ... & Pusic, M. (2023). Making sense of competency-based medical education (CBME) literary conversations: a BEME scoping review: BEME Guide No 78. Medical Teacher, 45(8), 802-815.

[3] Jacobs, S. M., Lundy, N. N., Issenberg, S. B., & Chandran, L. (2023). Reimagining core entrustable professional activities for undergraduate medical education in the era of artificial intelligence. JMIR Medical Education, 9(1), e50903.

[4] Lentz, A., Siy, J. O., & Carraccio, C. (2021). AI-ssessment: towards assessment as a sociotechnical system for learning. Academic Medicine, 96(7S), S87–S88.

[5] Vitands, B. (2022, October 25). Artificial Intelligence Behind Improved Clinical Skills Assessment and Teaching. University of Minnesota Medical School. https://med.umn.edu/news/artificial-intelligence-behind-improved-clinical-skills-assessment-and-teaching