Clinical artificial intelligence (AI) applications are rapidly developing but existing medical school curricula provide limited teaching covering this area. Here we describe an AI training curriculum we developed and delivered to Canadian medical undergraduates and provide recommendations for future training.
Artificial intelligence (AI) in medicine can potentially create workplace efficiencies and aid in clinical decision making. To guide AI applications safely, clinicians need some understanding of AI. Numerous commentaries advocate for AI concepts to be taught1, such as interpreting AI models and validation processes2. However, few structured programs have been implemented, especially on national scales. Pinto Dos Santos et al3. surveyed 263 medical students and 71% agreed they needed AI training. Teaching AI to medical audiences requires nuanced design to balance technical and non-technical concepts for learners who typically have a broad range of prior knowledge. We describe our experiences delivering an AI workshop series to three cohorts of medical students and make recommendations for future AI medical education based on this.
Objectives, timeline, and methodology
Our five week “Introduction to Medical AI” workshop for medical students was delivered three times between February 2019 and April 2021. A timeline of each workshop summarizing curricular changes is shown in Fig. 1. We had three major learning objectives motivating our curriculum: For learners to understand how data is processed in an AI application, analyze clinical implications of AI literature, and apply opportunities to collaborate with engineers in developing AI.
The first workshop ran from February to April 2019 at the University of British Columbia and all 8 participants provided positive feedback4. Due to COVID-19, the second workshop was offered virtually from October to November 2020, with 222 medical students and 3 resident physicians from 8 Canadian medical schools registered. Presentation slides and code were uploaded to an open-access website (http://ubcaimed.github.io). Major feedback from the first iteration included lectures being dense and material being overly theoretical. There was the additional challenge to serve 6 different time zones in Canada. Hence, the second workshop reduced sessions to 1 h each, condensed didactic material, added more case studies, and created template programs to allow participants to complete segments of code with minimal debugging (Box 1). Major feedback from the second iteration included positive reception of programming exercises and requests to demonstrate planning a machine learning project. Hence, in the third workshop which ran from March to April 2021 virtually to 126 medical students we included more interactive programming exercises and a project feedback session to demonstrate critical evaluation of projects using concepts from the workshop.
The most recent curriculum, from April 2021, is summarized in Table 1 and includes the targeted learning objectives for each topic. The workshop was designed for a novice level of technical proficiency, with no mathematics beyond a first-year undergraduate medical course. The curriculum was designed by 6 medical students and 3 instructors with engineering graduate degrees. Engineers proposed AI theory for teaching and medical students filtered for clinically relevant material.
The workshop consisted of lectures, case studies, and guided programming. In the first lecture, we reviewed select data analytics concepts from biostatistics including data visualization, logistic regression, and comparing descriptive versus inferential statistics. Although data analytics is fundamental to AI, we excluded topics such as data mining, significance tests, or interactive visualizations. This was due to time constraints and because several senior students had previous biostatistics training and were keen to cover more unique machine learning topics. The subsequent lectures presented current state-of-the-art methods and discussed AI problem formulation, strengths and limitations of AI models and model validation. Lectures were reinforced with case studies from the literature and from existing AI devices. We emphasized the skills needed to assess model performance and feasibility for a clinical problem, including understanding limitations of current AI devices. For example, we guided students in interpreting a pediatric head trauma guideline by Kupperman et al.5, where an AI decision tree algorithm was implemented to determine if computed tomography scanning was beneficial based on a physician’s examination. We highlighted that this is a common example of AI providing predictive analytics for physicians to interpret, rather than a physician replacement.
In guided programming examples, available open-source (https://github.com/ubcaimed/ubcaimed.github.io/tree/master/programming_examples), we demonstrated how to conduct exploratory data analysis, dimensionality reduction, loading a standard model, training, and testing. We used Google Colaboratory notebooks (Google LLC, Mountain View, California), which allowed execution of Python code from web browsers. An example of a programming exercise is summarized in Fig. 2. The exercise involved predicting malignant tumors using the Wisconsin Breast Imaging Open Dataset6 with a decision tree algorithm.
We identified four main challenges during the training:
Heterogeneity of Prior Knowledge: Our participants varied in mathematical proficiency. For instance, students with advanced technical backgrounds sought in-depth content such as how to perform Fourier feature transformations. However, it was not feasible to discuss Fourier algorithms to the class as this required advanced signal processing knowledge.
Attendance Attrition: There was reduced attendance in subsequent sessions, particularly with the online format. A solution could be to track attendance and provide a certificate of completion. Medical schools have been known to provide recognition on student transcripts for extracurricular academic activities, which may incentivize completion.
Curricular Design: As AI spans numerous subfields, selecting core concepts at an appropriate depth and breadth was challenging. For instance, an important topic is the bench-to-bedside continuum for AI tools. Though we introduced data preprocessing, model construction, and validation, we did not include topics such as mining big data, interactive visualizations, or running an AI clinical trial7 in favor of focusing on concepts most unique to AI. Our guiding principle was to train literacy over proficiency. For instance, understanding how a model processes input features is important for interpretability and one method is with gradient activation maps, which visualize which region of data is predictive. However, this requires multivariate calculus and was not feasible to introduce8. Developing a shared terminology proved challenging as we struggled to explain how to manipulate data as vectors without mathematical formalism. We noticed different terms shared meanings, such as describing a “feature” as a “variable” or “attribute” in epidemiology.
Knowledge Retention: It remains to be seen how well participants retain knowledge as there are limited opportunities to apply AI. Medical school curriculums frequently rely on spaced repetition where knowledge is consolidated in practical rotations9, which may be applicable to AI education as well.
We observed four main successes:
Proficiency was targeted over literacy: The depth of material was designed without rigorous mathematics, which has been a perceived challenge in launching clinical AI curricula10. In programming examples, we used template programs to allow participants to fill in blanks and run software without requiring knowledge of setting up a full programming environment.
Concerns about AI were addressed: There is a common concern that AI might replace certain clinical duties3. To address this, we explained the limitations of AI, including that nearly all AI technologies approved by regulatory bodies require physician supervision11. We also emphasized the importance of bias, where algorithms are susceptible to systematic error, especially if the dataset is not diverse12. A certain subgroup may hence be modeled incorrectly, leading to inequitable clinical decisions.
Resources were open-access: We generated publicly available resources, including lecture slides and code. While access to synchronous content was limited due to time zones, the open-source content is a convenient, asynchronous method for learning as not all medical schools have readily available access to AI expertise.
Multidisciplinary Collaboration: The workshop was a joint venture initiated by medical students to plan curricula alongside engineers. This demonstrated collaborative opportunities and knowledge gaps in both domains for participants to understand potential roles they may contribute to in the future.
Based on our experience we have four recommendations for others implementing similar courses:
Identify Core AI Competencies: Defining a list of competencies provides a standardized structure that can be integrated into existing competency-based medical curricula. The workshop currently uses learning objectives levels 2 (understand), 3 (apply), and 4 (analyze) of Bloom’s Taxonomy. Having resources for higher taxonomic levels, such as creation of a project, can further consolidate knowledge. This requires collaboration with clinical experts to identify how AI topics can be applied to the clinical workflow and to prevent teaching redundant topics already included in standard medical curricula.
Create AI Case Studies: Similar to clinical vignettes, case-based instruction may consolidate abstract concepts by identifying relevance to clinical problems. For example, a study in the workshop analyzed Google’s AI-based diabetic retinopathy detection system13 to identify bench-to-bedside challenges such as external validation requirements and regulatory approval pathways.
Use experiential Learning: Technical skills require deliberate practice and repeated application to master, similar to the learning clinical trainees experience while on rotations. One potential solution is the flipped classroom model, which reported increased knowledge retention in engineering education14. In this model, students review theoretical material on their own and class time is used for problem-solving using case studies.
Expand to Multi-Disciplinary Participants: We envision the implementation of AI involving interaction from various disciplines, including physicians at different levels of training and allied health professionals. As such, curriculum-development in consultation with educators from different faculties may be needed to tailor content for different healthcare domains.
AI is highly technical, with foundational concepts involving mathematics and computer science. Training medical personnel to understand AI poses unique challenges relating to content selection, clinical relevance, and method used to teach the material. We hope that our insights gained from carrying out AI education workshops may assist future educators of innovative approaches to integrate AI into medical education.
The Google Colaboratory Python scripts are open-source and available at: https://github.com/ubcaimed/ubcaimed.github.io/tree/master/.
Prober, C. G. & Khan, S. Medical education reimagined: a call to action. Acad. Med. 88, 1407–1410 (2013).
McCoy, L. G. et al. What do medical students actually need to know about artificial intelligence? NPJ Digit. Med 3, 1–3 (2020).
Dos Santos, D. P. et al. Medical students’ attitude towards artificial intelligence: a multicentre survey. Eur. Radiol. 29, 1640–1646 (2019).
Fan, K. Y., Hu, R. & Singla, R. Introductory machine learning for medical students: A pilot. J. Med. Educ. 54, 1042–1043 (2020).
Kuppermann, N. et al. Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study. Lancet 374, 1160–1170 (2009).
Street, W. N., Wolberg, W. H. & Mangasarian, O. L. Nuclear feature extraction for breast tumor diagnosis. Biomed. Image Process. Biomed. Vis. 1905, 861–870 (1993).
Chen, P. H. C., Liu, Y. & Peng, L. How to develop machine learning models for healthcare. Nat. Mater. 18, 410–414 (2019).
Selvaraju, R. R. et al. Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision, 618–626 (2017).
Kumaravel, B., Stewart, C. & Ilic, D. Development and evaluation of a spiral model of assessing EBM competency using OSCEs in undergraduate medical education. BMC Med. Educ. 21, 1–9 (2021).
Kolachalama, V. B. & Garg, P. S. Machine learning and medical education. NPJ Digit. Med. 1, 1–3 (2018).
van Leeuwen, K. G., Schalekamp, S., Rutten, M. J., van Ginneken, B. & de Rooij, M. Artificial intelligence in radiology: 100 commercially available products and their scientific evidence. Eur. Radiol. 31, 3797–3804 (2021).
Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44–56 (2019).
Beede, E. et al. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020).
Kerr, B. The flipped classroom in engineering education: A survey of the research. Proceedings of the 2015 International Conference on Interactive Collaborative Learning (2015).
The authors thank Danielle Walker, Tim Salcudean and Peter Zandstra of the University of British Columbia’s Biomedical Imaging and Artificial Intelligence Research Cluster for support and funding.
The authors declare no competing interests.
Peer review information
Communications Medicine thanks Carolyn McGregor, Fabio Moraes and Aditya Borakati for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Hu, R., Fan, K.Y., Pandey, P. et al. Insights from teaching artificial intelligence to medical students in Canada. Commun Med 2, 63 (2022). https://doi.org/10.1038/s43856-022-00125-4
This article is cited by
Verso una leadership clinica dell’intelligenza artificiale per la salute