Monitoring and managing resident workload is a cornerstone of policy in graduate medical education, and the duty hours metric is the backbone of current regulations. While the duty hours metric measures hours worked, it does not capture differences in intensity of work completed during those hours, which may independently contribute to fatigue and burnout. Few such metrics exist. Digital data streams generated during the usual course of hospital operations can serve as a novel source of insight into workload intensity by providing high-resolution, minute-by-minute data at the individual level; however, study and use of these data streams for workload monitoring has been limited to date. Paging data is one such data stream. In this work, we analyze over 500,000 pages—two full years of pages in an academic internal medicine residency program—to characterize paging patterns among housestaff. We demonstrate technical feasibility, validity, and utility of paging burden as a metric to provide insight into resident workload beyond duty hours alone, and illustrate a general framework for evaluation and incorporation of novel digital data streams into resident workload monitoring.
Resident workload has been in the public conscience since at least 1984, when the death of college student Libby Zion was proposed to be linked to resident fatigue.1 In subsequent years, additional studies have suggested that long work hours are associated with adverse events affecting both house officers and potentially their patients.2,3 To limit resident overwork, resident workload restrictions were recommended and ultimately mandated. The primary metric under restriction is hours worked, known as duty hours. Several notable studies have been unable to conclusively demonstrate improvement in outcomes with duty hours regulation.4,5,6
One hypothesis is that it is not just hours worked, but what happens during those hours, that contributes to workload and fatigue. The pager is inextricably intertwined with the daily routine of the medicine house officer. Analysis of large-scale paging data has the potential to provide high resolution, minute-by-minute information about not just the hours worked, but the nature and intensity of work completed during those hours. Most studies to date involve manual analysis of smaller-scale paging datasets7,8,9,10; no studies have characterized paging patterns in an academic medical residency program across multiple years, and even basic questions such as “How many pages does an internal medicine resident receive per year” remain unanswered.
Baseline paging characteristics
To demonstrate the utility of paging data in the discussion around resident workload, we first characterize baseline paging burden and frequency. We next explore whether paging burden is a metric that demonstrates variation even when duty hours are similar. We subsequently investigate whether observed variation is informative—that is, whether increased paging is associated with other metrics known to be related to workload.
In the two-year period of this study, residents of the internal medicine program received a total of 502,293 pages (Table 1). Interns (R1s) received an average of 2924 pages per year, while second-year residents (R2s) received an average of 1898 pages, and third-year residents (R3s) an average of 1492 pages. Over half of all pages were sent to interns, and on inpatient wards and subspecialty services, 57 to 87% of pages were fielded by interns.
Patterns of variation
Pages are not distributed uniformly across hours worked; patterns of variation emerge across services and year of training, and during time of day within a service. The most paged role in the training program is night-float intern covering the University hospital, where interns received a median of 30 pages per night (Fig. 1). Outpatient and elective rotations generated the least number of pages per intern. Trends within a service are consistent year-over-year.
Even within a given service, not all hours are created equal. Analysis of paging by year of trainee, clinical service, and hour of day sheds light on the cadence of various services (Fig. 2). For example, on call days, interns on the University inpatient medicine service are paged relatively infrequently during the morning, but paging begins to pick up around 1 p.m. and continues to increase all the way through the end of the admitting shift at 7 p.m. Pages to interns on the University night float service arrive in a bimodal distribution, peaking from 8 p.m. to midnight, and then again at 6 a.m., in the hour just before the shift ends, in what is known to many interns as the “morning labs” pages, when a new day’s worth of abnormalities must be acknowledged and acted upon.
Relationship between paging burden and operational strain
To investigate whether variation in paging burden is actually informative of increased workload conditions, we selected the service with greatest volume of pages, the University night float service, for further study. This service also exhibited the greatest shift-to-shift variation in paging burden (Fig. 1). We then explored whether increased paging was associated a known metric of increased workload or operational strain.
Emergency department length of stay (ED LOS), and the related measure emergency department boarding time (the component of length of stay corresponding to time spent in the ED after the decision to admit has been made), are metrics known to increase under conditions of systemwide operational strain. Increased length of stay and boarding time are associated with adverse outcomes.11,12
A linear model was used to relate total emergency department length of stay and ED boarding time in minutes to number of pages received between the start of the shift and the time the medicine service was consulted for admission. Each additional page was associated with increased boarding time of 1.06 min per page (95% CI 0.51–1.32, p-value < 0.001), and increased overall length of stay of 1.16 min (95% CI 0.73–1.59, p < 0.001) (Fig. 3a). Light paging conditions were associated with median ED LOS of ~7 h, while the heaviest paging conditions correspond to observed median LOS of ~12 h in the period under study (Fig. 3b).
Time since shift start was not a significant factor, either independently or in combination with paging burden.
Nights with increased paging were independently, linearly, and statistically significantly associated with increased ED length of stay and boarding time, supporting the case for paging burden as a marker informative of increased operational workload.
Analysis of large-scale paging data—which we term PAGEOMICS (Paging Analytics Guided Exploration of Medicine in Clinical Settings)—can serve as a novel source of high-resolution information informative of resident workload and hospital operations. Paging burden can vary widely by service, hour of day, and year of training, even when hours worked are similar, and this variation is related to other systemwide measures of operational strain. This data can enable a more comprehensive understanding of not only duration, but also intensity of resident work. For example, on the University night-float service, each intern and resident works the same 12-hour shift nightly, from 7 p.m. to 7 a.m.; all house officers report essentially identical duty hours on this service. However, some house officers receive less than 10 pages on a shift while others receive over 80! In the quest to monitor resident workload and fatigue, it does not make sense to treat these two extremes as identical workloads. Paging data can provide information not be captured by duty hours reporting alone.
While this work establishes that paging data varies even for similar hours worked and explores the relationship between paging burden and operational strain, it does not characterize all the determinants of variation in paging burden. Future work can elucidate additional hospital factors (for example, patient census, team census, number of admissions and discharges), patient factors (for example, patient complexity or acuity), page factors (for example, content of page), and resident factors (for example, age, gender, prior performance) that may play a role in predicting paging burden. Similarly, other proxies for work intensity may be studied, whether metrics that can be obtained from the EHR, or external instruments such as survey data relating perception of shift intensity to paging burden.
Paging is an older technology, which may eventually be replaced by modern forms of communication, such as cell phones, texting, HIPAA-compliant messaging apps, and others. In any case, house officers will continue to receive inbound digital communications from various team members throughout the hospital, and studying the type and frequency of those communications is likely to provide insight into the nature and intensity of work undertaken by residents.
Paging data represents but one type of large-scale digital data generated over the course of daily hospital operation. The EHR provides another source—order-entry, keystroke logging, number of clicks, and time spent on documentation have all been observed or extracted for various studies.13,14,15 And even beyond hospital systems, wearable devices such as motion trackers, step counters, heart-rate monitors, and sleep trackers can potentially provide information on resident actions and, in some cases, physiological markers of stress and well-being. It is unlikely that any one data stream, whether paging or otherwise, will be universally informative for all residents on all services in all institutions. The approach outlined in this work—to first obtain the data and ensure sufficient data quality and quantity, then characterize baseline distributions and patterns of variation, and finally to evaluate whether that variation is informative of workload intensity in a particular context-outlines a framework for evaluating and incorporating novel data streams (or combinations of data streams) for workload monitoring. We refer to this approach as volume, variation, and validity (VVV).
The true opportunity with these novel class of digital data streams is not just in their richness, but in their timelines. Digital data streams, combined with real-time analytics and alerts, can finally move us from post-hoc reporting to proactive interventions. For example, automated reporting of an unusually busy shift may trigger an alert to a program director and prompt calling in a resident from a backup pool. Or a resident with several days in a row of high-workload conditions may be asked to take a day off for load-management, as is now done in some professional sports.16 Or perhaps persistently unequal distribution of workload intensity may induce redistribution of patients across teams. Rich, validated, real-time metrics of workload intensity make the study of these types of interventions possible, and provide an opportunity to not just reflect on adverse outcomes, but to avoid them.
A technical and policy framework to encourage exploration, development, evaluation, and use of these data-driven approaches is needed.
Data and analyses
The Stanford University Internal Medicine Residency Program accepts up to 50 interns each year; ~40 are categorical internal medicine residents. Residents perform inpatient rotations primarily at Stanford Hospital, a 613-bed tertiary care center, and the neighboring Palo Alto VA Hospital, which has 272 beds for general medical and surgical disciplines.
All paging IDs associated with residents active from July 1, 2013 to June 24, 2015 were identified. This period represents two complete academic cycles.
Metadata for all pages sent to these IDs during the study period, including the time the page was sent, was retrospectively obtained from the University’s office of Paging and Messaging Services. By cross-referencing with electronic call and service calendars and residency rosters, each page was computationally annotated with additional metadata including the year of training of the recipient, the clinical service, and for inpatient services, whether the recipient was on call. Text of alphanumeric pages was not available for this study.
Emergency department length of stay and boarding time data for all general medicine admissions at the University hospital during the study period was obtained.17 Of the 9516 admissions to the service during the study period, we included only admissions to the night float service, and only those which had nonmissing, chronologically ordered timestamps for ED arrival, ED “Consult to Medicine” order, “Admit to Medicine” order, and ED departure. Two thousand three hundred and twenty five admissions (24.4%) met this inclusion criteria. A linear model was used to relate emergency department length of stay and boarding time in minutes to number of pages received between the start of the shift and the time the medicine service was consulted for admission.
This study was approved by the Stanford University Institutional Review Board (protocol #42093) with waived informed consent due to the retrospective nature of the study.
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
Data are available from the corresponding author (A.K.). The data are not publicly available due to them containing information that could compromise research participant privacy or consent, as well as patient privacy or consent.
Asch, D. A. & Parker, R. M. The Libby Zion case. One step forward or two steps backward? N. Engl. J. Med. 318, 771–775 (1988).
Barger, L. K. et al. Extended work shifts and the risk of motor vehicle crashes among interns. N. Engl. J. Med. 352, 125–134 (2005).
Barger, L. K. et al. Impact of extended-duration shifts on medical errors, adverse events, and attentional failures. PLoS Med. 3, e487 (2006).
Rajaram, R. et al. Association of the 2011 ACGME resident duty hour reform with general surgery patient outcomes and with resident examination performance. JAMA 312, 2374–2384 (2014).
Patel, M. S. et al. Association of the 2011 ACGME resident duty hour reforms with mortality and readmissions among hospitalized medicare patients. JAMA 312, 2364–2373 (2014).
Bilimoria, K. Y. et al. National cluster-randomized trial of duty-hour flexibility in surgical training. N. Engl. J. Med. 374, 713–727 (2016).
Katz, M. H. & Schroeder, S. A. The sounds of the hospital. Paging patterns in three teaching hospitals. N. Engl. J. Med. 319, 1585–1589 (1988).
Harvey, R., Jarrett, P. G. & Peltekian, K. M. Patterns of paging medical interns during night calls at two teaching hospitals. CMAJ 151, 307–311 (1994).
Fargen, K. M., O’Connor, T., Raymond, S., Sporrer, J. M. & Friedman, W. A. An observational study of hospital paging practices and workflow interruption among on-call junior neurological surgery residents. J. Grad. Med. Educ. 4, 467–471 (2012).
Luxenberg, A., Chan, B., Khanna, R. & Sarkar, U. Efficiency and interpretability of text paging communication for medical inpatients: a mixed-methods analysis. JAMA Intern. Med. 177, 1218–1220 (2017).
Singer, A. J., Thode, H. C., Viccellio, P. & Pines, J. M. The association between length of emergency department boarding and mortality. Acad. Emerg. Med. 18, 1324–1329 (2011).
Plunkett, P. K., Byrne, D. G., Breslin, T., Bennett, K. & Silke, B. Increasing wait times predict increasing mortality for emergency medical admissions. Eur. J. Emerg. Med. 18, 192–196 (2011).
Hill, R. G., Sears, L. M. & Melanson, S. W. 4000 clicks: a productivity analysis of electronic medical records in a community hospital ED. Am. J. Emerg. Med. 31, 1591–1594 (2013).
Sinsky, C. et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties. Ann. Intern. Med. 165, 753–760 (2016).
Ouyang, D., Chen, J. H., Hom, J. & Chi, J. Internal medicine resident computer usage: an electronic audit of an inpatient service. JAMA Intern. Med. 176, 252–254 (2016).
Bontemps, T. Kawhi: In-season rest key to availability in Finals. espn.com https://www.espn.com/nba/story/_/id/26922607/kawhi-season-rest-key-availability-finals. (Accessed 18 July 2019).
Lowe, H. J., Ferris, T. A., Hernandez, P. M. & Weber, S. C. STRIDE–An integrated standards-based translational research informatics platform. AMIA Annu. Symp. Proc. 2009, 391–395 (2009).
The authors wish to acknowledge Lee Merrick and Stanford University IT Paging and Messaging Services for provision of paging data; Gomathi Krishnan and the Stanford Translational Research Integrated Database Environment (STRIDE)17 for provision of admissions data; the Stanford University Department of Medicine and the Stanford University Internal Medicine Residency Program for support, and Dr. Nigam Shah, Dr. Russ Altman, and Dr. Arul Thangavel for thoughtful feedback and support.
The authors declare no competing interests.
Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Kaushal, A., Katznelson, L. & Harrington, R.A. Beyond duty hours: leveraging large-scale paging data to monitor resident workload. npj Digit. Med. 2, 87 (2019). https://doi.org/10.1038/s41746-019-0165-2