Opinion

Role of AI in Competency-Based Medical Education: Using EPA as the Magicbox

Cite this article as: Dabbagh A, Madadi F, Larijani B. Role of AI in competency-based medical education: using EPA as the magicbox. Arch Iran Med. 2024;27(11):633-635. doi: 10.34172/aim.31795

Competency-based medical education (CBME) is a modern and quality-based approach to medical education that aims to attain the trainees’ essential professional, behavioral, and scientific capabilities. Implementing entrustable professional activities (EPA) is the cornerstone and fundamental aspect of CBME, which guarantees the formative development of competence in the trainees and the attainment of quality in the required areas of education.

On the other hand, several researches and studies demonstrate with enough data that artificial intelligence can be used to predict different aspects of neurocognition including attitude and professional behaviors; in other words, artificial intelligence can predict the behaviors, movements, decisions, and reasoning in humans to a great extent; its predictive power, of course, depends on the amount of data and the type of processes.

Here, we would like to discuss the potential role of artificial intelligence in assessment of medical students in the CBME era.

Entrustable Professional Activities

Medical education systems are moving toward CBME, based on the concept that educators should attend to their healthcare profession whenever they become entrustable.^1,2 Traditional time-based assessment methods are proven to be insufficient for predicting the trustworthiness of residents; hence, medical teachers sought for more reliable assessment tools, one of which is EPA.³ Soon after being introduced by Ten Cate in 2005, EPAs gained popularity as an essential component of CBME.⁴ Many institutions have verified EPAs as an assessment tool for medical education including postgraduate medical trainees, undergraduate medical students, many subspecialties, and even junior faculty members.

To date many, EPAs have been described, and some pilot studies have validated these EPAs. So, the expected levels of entrustment have been defined and many have been even revised⁵; however, the role of using EPA in predicting the professional future of the trainees, especially in the field of medical education decision making, is not yet clear and well-defined.

A qualified anesthesiologist should be able not only to manage his own stress during critical periods, but also lead a team in this situation. Apparently traditional assessment methods are not adequate to assess such qualifications and EPAs would be promising.¹ Burkhart and colleagues conducted a survey among anesthesiology programs in USA, Canada, Switzerland, Germany, Austria and the Netherlands. Less than half of respondents already utilized EPAs and even those who developed EPAs had no specific decision-making system based on EPAs.⁶

Woodworth et al recently published their programmatic system for competency assessment of anesthesiology residency programs in USA. They validated their assessment system which consisted of a set of EPAs, procedural skill assessments, non-technical skill assessments and OSCEs via a mobile application over a period of two years. However, they also confronted some missing assessments.⁵

A drawback of EPA-based decision making is that the number of EPAs that a trainee in any medical discipline is required to complete is relatively large and based on the content of the curriculum in each discipline; however, this large amount of data per trainee when aggregated within an academic department or beyond, at the level of a college or a university, creates the potential opportunity to draw very diverse and determining conclusions from these data, especially with the help of artificial intelligence.⁵

Artificial Intelligence and CBME

Currently a life changing transformation is occurring due to fast development of AI. Expectedly, AI has influenced medical education in many ways as well, one of which is its utilization in student assessment; so, AI can enhance assessment in CBME models in various pathways.^7,8

First, some researchers analyzed the narrative feedback to predict those students at risk. Those programs which develop EPAs as an assessment tool, gather a huge amount of data over time. Analyzing the scores and feedback and reconciling them to other forms of assessment may lead to algorithms which may help us find students at risk of failure and intervene.^9,10

On the other hand, many articles reported AI’s role in assessment of psychomotor skills, mostly focusing on surgical skills and intubation technique.^11-13 These technologies might reduce the workload of faculty members, the need for training faculty and also omit human errors. AI can also score the trainees’ function in virtual reality.¹

Thinking big, AI can be part of work-based assessment and problem-based learning. By observing trainees in their working environment and combining it with expert feedback and patient outcomes, EPAs might be reliable for making evidence-based decisions about students.^7,14

Future Implementation of AI in EPA Assessment

Since EPA examines different areas based on 6 core competencies, the result of evaluation with the help of EPA is to examine the clinical performance of learners; the final result of which is the improvement of skill, attitude, and knowledge.^15,16 Therefore, EPA-driven data of each trainee can be used to predict or extrapolate their behavior patterns and, when needed, take steps to improve the behavioral patterns throughout the time when the trainee is under education.

Therefore, it is possible to use the EPA-obtained data to develop an AI algorithm, which, while predicting clinical performance, simultaneously creates the possibility of improving trainees’ performance. The AI-driven results could also be potentially used to match the graduated people in more appropriate individual positions; either for their professional tasks or any further training position during the future stages.

Another potential utilization of AI would be supervision in the work environment. In order to utilize EPAs in decision making, programs should collect large data during a long period of time which needs well-educated supervisors and data collecting systems.¹⁴ AI can be utilized as a supervisor in a digital working environment and validate supervisor’s scoring as well as reducing their workload when adequately trained.

Conclusion

Medical education confronts two impressive revolutions:

1. Introducing CBME and EPA as a tool for competency assessment;

2. AI utilization in medical education.

The current disadvantage of EPA is lack of an efficient decision-making system. Recent advances in EPA development and AI can be promising in solving this problem by analyzing the large amount of collected data from both human supervisors and machine learning raters. Using AI to analyze the data retrieved from EPA assessment results might help predict the future professional behavior of the current trainees.

References

1. Hamza DM, Hauer KE, Oswald A, van Melle E, Ladak Z, Zuna I. Making sense of competency-based medical education (CBME) literary conversations: a BEME scoping review: BEME Guide No 78. Med Teach 2023; 45(8):802-15. doi: 10.1080/0142159x.2023.2168525 [Crossref] [ Google Scholar]
2. Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR. Competency-based medical education: theory to practice. Med Teach 2010; 32(8):638-45. doi: 10.3109/0142159x.2010.501190 [Crossref] [ Google Scholar]
3. Brasel KJ, Lindeman B, Jones A, Sarosi GA, Minter R, Klingensmith ME. Implementation of entrustable professional activities in general surgery: results of a national pilot study. Ann Surg 2023; 278(4):578-86. doi: 10.1097/sla.0000000000005991 [Crossref] [ Google Scholar]
4. Mitchell M, Srinivasan M, West DC, Franks P, Keenan C, Henderson M. Factors affecting resident performance: development of a theoretical model and a focused literature review. Acad Med 2005; 80(4):376-89. doi: 10.1097/00001888-200504000-00016 [Crossref] [ Google Scholar]
5. Woodworth GE, Goldstein ZT, Ambardekar AP, Arthur ME, Bailey CF, Booth GJ. Development and pilot testing of a programmatic system for competency assessment in US anesthesiology residency training. Anesth Analg 2024; 138(5):1081-93. doi: 10.1213/ane.0000000000006667 [Crossref] [ Google Scholar]
6. Burkhart CS, Dell-Kuster S, Touchie C. Who can do this procedure? Using entrustable professional activities to determine curriculum and entrustment in anesthesiology - an international survey. Med Teach 2022; 44(6):672-8. doi: 10.1080/0142159x.2021.2020231 [Crossref] [ Google Scholar]
7. Jacobs SM, Lundy NN, Issenberg SB, Chandran L. Reimagining core entrustable professional activities for undergraduate medical education in the era of artificial intelligence. JMIR Med Educ 2023; 9:e50903. doi: 10.2196/50903 [Crossref] [ Google Scholar]
8. Narayanan S, Ramakrishnan R, Durairaj E, Das A. Artificial intelligence revolutionizing the field of medical education. Cureus 2023; 15(11):e49604. doi: 10.7759/cureus.49604 [Crossref] [ Google Scholar]
9. Gin BC, Ten Cate O, O’Sullivan PS, Hauer KE, Boscardin C. Exploring how feedback reflects entrustment decisions using artificial intelligence. Med Educ 2022; 56(3):303-11. doi: 10.1111/medu.14696 [Crossref] [ Google Scholar]
10. Maimone C, Dolan BM, Green MM, Sanguino SM, Garcia PM, O’Brien CL. Utilizing natural language processing of narrative feedback to develop a predictive model of pre-clerkship performance: lessons learned. Perspect Med Educ 2023; 12(1):141-8. doi: 10.5334/pme.40 [Crossref] [ Google Scholar]
11. Lim C, Ko HS, Cho S, Ohu I, Wang HE, Griffin R. Development of a hand motion-based assessment system for endotracheal intubation training. J Med Syst 2021; 45(8):81. doi: 10.1007/s10916-021-01755-2 [Crossref] [ Google Scholar]
12. Zhao S, Xiao X, Zhang X, Yan Meng WL, Soghier L, Hahn JK. Automated assessment system for neonatal endotracheal intubation using dilated convolutional neural network. Annu Int Conf IEEE Eng Med Biol Soc 2020; 2020:5455-8. doi: 10.1109/embc44109.2020.9176329 [Crossref] [ Google Scholar]
13. Lam K, Chen J, Wang Z, Iqbal FM, Darzi A, Lo B. Machine learning for technical skill assessment in surgery: a systematic review. NPJ Digit Med 2022; 5(1):24. doi: 10.1038/s41746-022-00566-0 [Crossref] [ Google Scholar]
14. Lentz A, Siy JO, Carraccio C. AI-ssessment: towards assessment as a sociotechnical system for learning. Acad Med 2021; 96(7S):S87-8. doi: 10.1097/acm.0000000000004104 [Crossref] [ Google Scholar]
15. Dabbagh A, Fadaeizadeh L, Gharaei B, Ghasemi M, Kamranmanesh M, Khorasanizadeh S. The role of entrustable professional activities in competency-based medical education for anesthesiology residents: a pilot phase. Anesth Pain Med 2022; 12(5):e130176. doi: 10.5812/aapm-130176 [Crossref] [ Google Scholar]
16. Matava CT, Alam F, Kealey A, Bahrey LA, McCreath GA, Walsh CM. The influence of resident and faculty gender on assessments in anesthesia competency-based medical education. Can J Anaesth 2023; 70(6):978-87. doi: 10.1007/s12630-023-02454-x [Crossref] [ Google Scholar]