Introduction

Artificial intelligence (AI) in medicine can potentially create workplace efficiencies and aid in clinical decision making. To guide AI applications safely, clinicians need some understanding of AI. Numerous commentaries advocate for AI concepts to be taught1, such as interpreting AI models and validation processes2. However, few structured programs have been implemented, especially on national scales. Pinto Dos Santos et al3. surveyed 263 medical students and 71% agreed they needed AI training. Teaching AI to medical audiences requires nuanced design to balance technical and non-technical concepts for learners who typically have a broad range of prior knowledge. We describe our experiences delivering an AI workshop series to three cohorts of medical students and make recommendations for future AI medical education based on this.

Objectives, timeline, and methodology

Our five week “Introduction to Medical AI” workshop for medical students was delivered three times between February 2019 and April 2021. A timeline of each workshop summarizing curricular changes is shown in Fig. 1. We had three major learning objectives motivating our curriculum: For learners to understand how data is processed in an AI application, analyze clinical implications of AI literature, and apply opportunities to collaborate with engineers in developing AI.

Fig. 1: A visualization of the timeline for the three iterations of our workshop.
figure 1

In blue are lecture topics, with light blue dedicated time for interactive question and answer period. In grey are focused brief literature review sessions. In orange are the selected case studies outlining artificial intelligence models or techniques. In green are the guided programming sessions intended to teach how artificial intelligence can solve clinical problems and how to evaluate the model. The content and length with the workshop changed based on evaluation of student needs.

Full size image

The first workshop ran from February to April 2019 at the University of British Columbia and all 8 participants provided positive feedback4. Due to COVID-19, the second workshop was offered virtually from October to November 2020, with 222 medical students and 3 resident physicians from 8 Canadian medical schools registered. Presentation slides and code were uploaded to an open-access website (http://ubcaimed.github.io). Major feedback from the first iteration included lectures being dense and material being overly theoretical. There was the additional challenge to serve 6 different time zones in Canada. Hence, the second workshop reduced sessions to 1 h each, condensed didactic material, added more case studies, and created template programs to allow participants to complete segments of code with minimal debugging (Box 1). Major feedback from the second iteration included positive reception of programming exercises and requests to demonstrate planning a machine learning project. Hence, in the third workshop which ran from March to April 2021 virtually to 126 medical students we included more interactive programming exercises and a project feedback session to demonstrate critical evaluation of projects using concepts from the workshop.

Curriculum

The most recent curriculum, from April 2021, is summarized in Table 1 and includes the targeted learning objectives for each topic. The workshop was designed for a novice level of technical proficiency, with no mathematics beyond a first-year undergraduate medical course. The curriculum was designed by 6 medical students and 3 instructors with engineering graduate degrees. Engineers proposed AI theory for teaching and medical students filtered for clinically relevant material.

Table 1 A summary of concepts taught for each session of the final iteration of the workshop.
Full size table

The workshop consisted of lectures, case studies, and guided programming. In the first lecture, we reviewed select data analytics concepts from biostatistics including data visualization, logistic regression, and comparing descriptive versus inferential statistics. Although data analytics is fundamental to AI, we excluded topics such as data mining, significance tests, or interactive visualizations. This was due to time constraints and because several senior students had previous biostatistics training and were keen to cover more unique machine learning topics. The subsequent lectures presented current state-of-the-art methods and discussed AI problem formulation, strengths and limitations of AI models and model validation. Lectures were reinforced with case studies from the literature and from existing AI devices. We emphasized the skills needed to assess model performance and feasibility for a clinical problem, including understanding limitations of current AI devices. For example, we guided students in interpreting a pediatric head trauma guideline by Kupperman et al.5, where an AI decision tree algorithm was implemented to determine if computed tomography scanning was beneficial based on a physician’s examination. We highlighted that this is a common example of AI providing predictive analytics for physicians to interpret, rather than a physician replacement.

In guided programming examples, available open-source (https://github.com/ubcaimed/ubcaimed.github.io/tree/master/programming_examples), we demonstrated how to conduct exploratory data analysis, dimensionality reduction, loading a standard model, training, and testing. We used Google Colaboratory notebooks (Google LLC, Mountain View, California), which allowed execution of Python code from web browsers. An example of a programming exercise is summarized in Fig. 2. The exercise involved predicting malignant tumors using the Wisconsin Breast Imaging Open Dataset6 with a decision tree algorithm.

Fig. 2: A pipeline of the programming examples developed with specific concepts to be implemented.
figure 2

Programs are introduced during the week of the relevant topic and examples chosen were from published artificial intelligence applications. Programming elements were only included if deemed relevant to provide insight in future clinical practice, such as how to evaluate models to understand if they are ready to be applied in a clinical trial. The examples culminate in a full end-to-end application to classify tumors as benign or malignant based on medical image parameters.

Full size image

Challenges

We identified four main challenges during the training:

  1. 1.

    Heterogeneity of Prior Knowledge: Our participants varied in mathematical proficiency. For instance, students with advanced technical backgrounds sought in-depth content such as how to perform Fourier feature transformations. However, it was not feasible to discuss Fourier algorithms to the class as this required advanced signal processing knowledge.

  2. 2.

    Attendance Attrition: There was reduced attendance in subsequent sessions, particularly with the online format. A solution could be to track attendance and provide a certificate of completion. Medical schools have been known to provide recognition on student transcripts for extracurricular academic activities, which may incentivize completion.

  3. 3.

    Curricular Design: As AI spans numerous subfields, selecting core concepts at an appropriate depth and breadth was challenging. For instance, an important topic is the bench-to-bedside continuum for AI tools. Though we introduced data preprocessing, model construction, and validation, we did not include topics such as mining big data, interactive visualizations, or running an AI clinical trial7 in favor of focusing on concepts most unique to AI. Our guiding principle was to train literacy over proficiency. For instance, understanding how a model processes input features is important for interpretability and one method is with gradient activation maps, which visualize which region of data is predictive. However, this requires multivariate calculus and was not feasible to introduce8. Developing a shared terminology proved challenging as we struggled to explain how to manipulate data as vectors without mathematical formalism. We noticed different terms shared meanings, such as describing a “feature” as a “variable” or “attribute” in epidemiology.

  4. 4.

    Knowledge Retention: It remains to be seen how well participants retain knowledge as there are limited opportunities to apply AI. Medical school curriculums frequently rely on spaced repetition where knowledge is consolidated in practical rotations9, which may be applicable to AI education as well.

Successes

We observed four main successes:

  1. 1.

    Proficiency was targeted over literacy: The depth of material was designed without rigorous mathematics, which has been a perceived challenge in launching clinical AI curricula10. In programming examples, we used template programs to allow participants to fill in blanks and run software without requiring knowledge of setting up a full programming environment.

  2. 2.

    Concerns about AI were addressed: There is a common concern that AI might replace certain clinical duties3. To address this, we explained the limitations of AI, including that nearly all AI technologies approved by regulatory bodies require physician supervision11. We also emphasized the importance of bias, where algorithms are susceptible to systematic error, especially if the dataset is not diverse12. A certain subgroup may hence be modeled incorrectly, leading to inequitable clinical decisions.

  3. 3.

    Resources were open-access: We generated publicly available resources, including lecture slides and code. While access to synchronous content was limited due to time zones, the open-source content is a convenient, asynchronous method for learning as not all medical schools have readily available access to AI expertise.

  4. 4.

    Multidisciplinary Collaboration: The workshop was a joint venture initiated by medical students to plan curricula alongside engineers. This demonstrated collaborative opportunities and knowledge gaps in both domains for participants to understand potential roles they may contribute to in the future.

Recommendations

Based on our experience we have four recommendations for others implementing similar courses:

  1. 1.

    Identify Core AI Competencies: Defining a list of competencies provides a standardized structure that can be integrated into existing competency-based medical curricula. The workshop currently uses learning objectives levels 2 (understand), 3 (apply), and 4 (analyze) of Bloom’s Taxonomy. Having resources for higher taxonomic levels, such as creation of a project, can further consolidate knowledge. This requires collaboration with clinical experts to identify how AI topics can be applied to the clinical workflow and to prevent teaching redundant topics already included in standard medical curricula.

  2. 2.

    Create AI Case Studies: Similar to clinical vignettes, case-based instruction may consolidate abstract concepts by identifying relevance to clinical problems. For example, a study in the workshop analyzed Google’s AI-based diabetic retinopathy detection system13 to identify bench-to-bedside challenges such as external validation requirements and regulatory approval pathways.

  3. 3.

    Use experiential Learning: Technical skills require deliberate practice and repeated application to master, similar to the learning clinical trainees experience while on rotations. One potential solution is the flipped classroom model, which reported increased knowledge retention in engineering education14. In this model, students review theoretical material on their own and class time is used for problem-solving using case studies.

  4. 4.

    Expand to Multi-Disciplinary Participants: We envision the implementation of AI involving interaction from various disciplines, including physicians at different levels of training and allied health professionals. As such, curriculum-development in consultation with educators from different faculties may be needed to tailor content for different healthcare domains.

Conclusions

AI is highly technical, with foundational concepts involving mathematics and computer science. Training medical personnel to understand AI poses unique challenges relating to content selection, clinical relevance, and method used to teach the material. We hope that our insights gained from carrying out AI education workshops may assist future educators of innovative approaches to integrate AI into medical education.