Main

Poor medication adherence is a major healthcare problem, contributing to 10% of hospitalizations, 125,000 deaths per year and up to $290 billion in annual cost in the United States alone1,2. A crucial step toward achieving medication adherence is ensuring proper MSA—that is, ensuring that patients take medications at the prescribed time and use the proper self-administration technique. When patients fail to perform MSA properly, the net result is that the medication is not delivered to its intended action site, causing failures in managing the condition3. Unfortunately, MSA errors are common, particularly in chronic diseases where up to 50% of patients do not take medications as prescribed4,5. The problem is exacerbated when medication delivery requires devices such as insulin pens or inhalers. MSA errors associated with medication delivery devices (for example, not shaking the inhaler before use or not priming the insulin dose) result in administration failures, subsequent high levels of non-adherence, reduced disease control and unscheduled use of healthcare resources6,7,8,9. Physicians report that up to 70% of their patients do not take their insulin as prescribed6,7. Similarly, over 50% of patients who use inhalers do so erroneously regardless of the inhalation device used8. These statistics are alarming given that hundreds of millions of patients worldwide depend on these devices for their medication administration9,10.

Addressing the above problems requires adequately assessing patients’ MSA and detecting MSA errors. Existing solutions to assess MSA typically require direct observation by health professionals—that is, a clinician or pharmacist watches the patient as she uses her inhaler or insulin pen and guides her through the proper administration technique. For example, the clinician would explain to the patient that she needs to shake the inhaler, fully exhale, inhale a dose and hold her breath for 10 s before exhaling. The clinician would also watch the patient performing these steps and alert her if she fails to follow the proper administration technique. Although this approach can be used in the clinic, most MSA errors occur at home and away from the observation of a clinician. Furthermore, patients’ performance in front of a clinician might be unrepresentative of their actual MSA, as patients tend to perform better when assessed by a clinician, a phenomenon known as ‘white-coat compliance’11. Even when patients receive initial training on their devices by a clinician, MSA errors occur over time due to forgetfulness or recklessness in adhering to the prescribed administration time, frequency or technique12. As a result, many MSA errors end up undetected until they manifest as serious health problems or admissions to the emergency room12.

This study was motivated by the question of whether we could use AI to assist with the observation of patients at home and provide a continuous assessment of their MSA. We present an AI-based solution that achieves this goal in an accurate, efficient and cost-effective manner. Our solution avoids cameras, which many patients find to be intrusive when deployed in their homes13. Instead, our solution uses a Wi-Fi-like sensor deployed in patient homes. (The sensor transmits signals around the Wi-Fi frequency range using the frequency-modulated continuous-wave (FMCW) technique. A detailed description of the radio sensor can be found in the Methods section.) The sensor transmits a very low-power wireless signal (1,000 times lower power than standard Wi-Fi), and our system analyzes the reflections of the signal from the environment using AI techniques. Because up to 60% of the human body is water, it reflects the surrounding radio signals and modulates them with the person’s movements14. Past work has shown that such radio reflections can be used to capture breathing and heart rate, detect falls and monitor sleep15,16,17. In this study, we focused on movements associated with MSA events. Our AI system, embedded in the sensor, analyses the radio reflections from the environment to track the specific movements associated with MSA and to detect when a patient administers her medication using an inhaler or insulin pen. It further examines the wireless reflections to detect whether the patient has followed the required steps of using the medication device and generates an alert if the patient fails to follow the proper technique (for example, forgot to prime her insulin pen or shake her inhaler). This AI-based solution works in a contactless and passive manner, introducing no burden on the patient, caregiver or health personnel.

Figure 1 illustrates a use case of our AI system at home, where it assesses the individual’s MSA with an inhaler. The wireless sensor is mounted on the wall like a Wi-Fi box (Fig. 1a). There is no need for cameras, wearable sensors or any additional smart devices. The wall-mounted sensor analyzes the surrounding radio signal using AI methods. In this case, it would detect an instance of MSA using an inhaler and document the corresponding time. The AI solution also tracks the motion of the person and detects that the person shook the device, exhaled before use and, finally, inhaled a dose (Fig. 1b–d), which are required steps for MSA with an inhaler. An example output of our system is available in the Supplementary Video.

Fig. 1: A use case illustration of the wireless AI-based system to monitor individual MSA with an inhaler device.
figure 1

a, The wireless sensor is mounted on the wall, analyzing the surrounding radio signals using AI. The AI solution would detect when the person started to use an inhaler. b–d, Our AI solution also tracks the motion during the MSA event and detects that the person shook the device, exhaled before use and, finally, inhaled a dose. (We obtained informed consent from the participant for the use of his photographs.)

To build the AI-based solution, we designed a study where health professionals and the AI-based solution simultaneously observe MSA events with insulin pens and inhaler devices. To emulate real-world standard practice for first-time users of these medications, participants were trained by a pharmacist to perform MSA according to current guidelines and recommendations18,19 and were then asked to demonstrate their MSA technique. During the experiment, human observers provided an MSA assessment that included the time window of each MSA event and the errors made during each event (if any). We used the MSA assessment provided by human observers to train and evaluate our AI algorithm. In total, we collected a large dataset that consists of 47,788 examples, where each example is a 2-min recording of radio signals. This dataset has 1,203 positive examples of MSA events with insulin pens and inhaler devices; about half of these MSA events are performed with no errors, whereas the other half includes some errors (for example, not shaking the inhaler before use or not holding one’s breath after inhaling the dose). The dataset also includes 46,585 negative MSA examples corresponding to common home activities that do not involve MSA, such as cooking, eating, typing and interacting with objects such as glasses, clothes, microwaves and hairdryers. The MSA examples in the dataset were performed by 107 healthy individuals whose ages varied from 18 to 72 years. The dataset was divided into training and testing sets that we used to train and evaluate the AI system, respectively.

Extensive experimental results (that are detailed in the Results section) demonstrate that our AI-based solution can reliably detect the occurrence of MSA events. Specifically, the AUC was 0.992 for detecting the use of an inhaler and 0.967 for detecting the use of an insulin pen. These results indicate that an AI system could be used at home to monitor whether patients use their inhalers and insulin pens following the prescribed time and frequency.

The experimental results also show that the AI solution can accurately evaluate whether the individual correctly followed the required steps for administering her medication using an inhaler or insulin pen. Adherence to the proper steps while performing MSA is crucial for disease management and therapeutic effectiveness20. For example, failure to follow the correct steps when using an insulin pen can lead to hyperglycemia or severe hypoglycemic episodes for patients with diabetes21,22. Similarly, failure to follow the recommended steps during inhaler administration contributes to symptom exacerbations and subsequent reduced quality of life for patients with asthma and patients with chronic obstructive pulmonary disease (COPD)23,24,25. Our results show that the AI system reliably detects both 1) missing key steps during the administration process (for example, not shaking the inhaler before use or not priming the insulin pen) and 2) patients not following duration-based requirements (for example, not holding the insulin pen after injection for 10 s).

Figure 2 illustrates how we envision such an AI-based solution that could be used in patient homes to help detect and address MSA errors. Our wireless sensor would be deployed in the patient’s home. The AI system would continuously and automatically analyze the radio signals and document MSA assessment results, which are uploaded over the internet and appended to the patient’s digital health record. Reminders will be sent to the patient if she fails to take the medication at the prescribed time. Authorized health professionals will also be able to access these records via a web portal to learn which patients have difficulties with their MSA and the types of errors they experience. The health professionals can then reach out to the patient to corroborate these results and make a clinical judgment (for example, whether more training on medication device administration is needed for the patient).

Fig. 2: Potential integration of our system into care management.
figure 2

Our wireless sensor with AI will continuously and automatically analyze the radio signals and document MSA assessment results in the cloud. The patient will receive reminders if she fails to take the medication at the prescribed time. Authorized health professionals can also access these records via a web portal to learn which patients have difficulties with their MSA and the types of errors they experience. The health professionals can reach out to the patient to corroborate these results and make a clinical judgment if necessary.

The recent outbreak of Coronavirus Disease 2019 (COVID-19) emphasizes the need for an automated and contactless solution for assessing MSA at home. The stay-at-home orders make it even more difficult to assess MSA through direct observation by health professionals. At the same time, individuals suffering from asthma, COPD and diabetes are at higher risk for severe illness from COVID-19 (refs. 26,27); hence, it is even more critical to ensure that they take their medications with the proper administration technique. Our automated and contactless AI-based MSA assessment solution could help these vulnerable populations to control their chronic conditions. It also enables health professionals to remotely monitor the MSA of their patients, without risks of contagion.

Our work shows how advances in AI can address an important unmet need in healthcare6,7,8,9,10, by continuously monitoring the MSA of patients in their homes, detecting when patients fail to use their medication devices as prescribed and providing patients with feedback on their medication administration technique and whether it follows the required steps. More generally, the work opens the door to the integration of AI-based solutions in care management through in-home passive, unobtrusive and contactless patient monitoring. Such integration could improve outcomes for patients and reduce the cost of healthcare.

Results

Our system was built and validated on a large dataset that consists of 47,788 examples of MSA events and home activities that do not involve MSA from 107 participants (53 females and 54 males), where each example is a recording of radio signals up to 2 min. A total of 40 different locations (such as offices, lounges, seminar rooms, kitchens and halls) were chosen to allow for variation between environmental conditions. Participants were allowed to move freely in the space and perform the MSA at a location of their choice, within 10 m from the wireless sensor. They were either standing or sitting when performing MSA and were allowed to pick any orientation with respect to the wireless sensor except for having their back to the sensor. For the MSA events with insulin pens, 150 events missed a common step, and 155 of them failed to comply with a specific duration requirement. For the MSA events with inhaler devices, 149 of them missed a required step, and 168 of them failed to comply with duration requirements.

We trained a neural network model that takes a stream of radio signals as input. It first detects and tracks the location of each individual in the environment. It then zooms in on the radio signals pertaining to each individual and predicts the occurrence of an MSA event. Instead of detecting the whole MSA event directly, our model detects the steps involved during administration and only claims an occurrence of an MSA event when multiple steps happen one after another. Detecting MSA events in this way improves our model’s robustness to variations among different people and enables the detection of MSA errors. Details of the model are described in the Methods section.

We performed K-fold cross-validation (K = 5) to evaluate our model. Specifically, the dataset was randomly split into five equal-sized subsets. A single fold was retained as test data, and the remaining four folds were used for training. This process was repeated five times, with each of the five folds used exactly once as the test data. The folds were divided such that participants who appear in the training data do not appear in the testing data and vice versa. To balance the number of participants across folds, we imposed no constraints on allocating sites to folds. We note that the cross-validation was not used for hyperparameter tuning.

Detection of MSA events

Our model detects MSA events in a sliding-window fashion. Specifically, it detects whether an MSA has happened for each 2-min window. To evaluate the performance of our model, we compared its predictions with ground truth provided by human annotations. Figure 3a,b show the receiver operating characteristic (ROC) curves for detecting MSA events with insulin pens and inhaler devices, respectively. When computing sensitivity and specificity, positive examples indicate MSA events, whereas negative examples indicate non-MSA events. Our system detected the occurrence of an insulin pen administration event with a sensitivity of 87.58% (95% confidence interval (CI), 84.7–90.0%) and a specificity of 96.06% (95% CI, 95.9–96.2%) and an AUC of 0.967. Similarly, inhaler administration events were detected with a sensitivity of 91.08% (95% CI, 88.4–93.2%) and a specificity of 99.22% (95% CI, 99.1–99.3%) and an AUC of 0.992. We note that the specificity or the false-positive ratio is computed for windows of non-MSA events such as eating, drinking or putting on clothes, not just any window of radio frequency (RF) signals. The number of false positives when considering any window of RF signals is significantly smaller in real-world deployment. Specifically, we leveraged a dataset in which the radio was used to monitor patients with Parkinson’s disease and control individualss (that is, healthy individuals) for over 1 month28. Because none of the individuals in this dataset used inhalers or insulin pens, all detected MSA events could be considered false positives. We considered five homes from the study and used one full month of RF signals from each home. On average, the number of false positives over a whole month was 2.2 for insulin pens and 6.6 for inhalers.

Fig. 3: Evaluation results for the detection of MSA events with insulin pens and inhaler devices.
figure 3

a,b, ROC curves for detecting insulin (n = 47,205) and inhaler (n = 47,168) administration. ROC curves demonstrate the tradeoff between sensitivity and specificity as the detection thresholds are varied. The AUC is an aggregate measure of detection performance (a model whose predictions are 100% correct will have an AUC of 1.0). c, Distribution of the errors for start time and end time estimation (n = 620 for insulin pens and n = 583 for inhalers). On each box plot, the central line indicates the median, and the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to 1.5 times the interquartile range. Points beyond the whiskers are plotted individually using the ‘+’ symbol. d,e, Cumulative distribution functions (CDFs) of the absolute error for start time and end time estimation (n = 620 for insulin pens and n = 583 for inhalers).

We also looked at errors that our model made when estimating the start time and end time of an MSA event. Figure 3c shows the box plots for the estimation errors, and Fig. 3d,e shows the cumulative distribution functions of the absolute estimation errors. Our system made an unbiased (that is, median-unbiased) estimation of the start time and the end time for both devices. For the start time estimation, the 50th percentile error was 0.6 s and 0.4 s, whereas the 90th percentile error was 2.0 s and 1.3 s for insulin pens and inhaler devices, respectively. Similarly, for the end time estimation, the 50th percentile error was 0.4 s and 0.3 s, whereas the 90th percentile error was 1.4 s and 0.9 s for insulin pens and inhaler devices, respectively. To put these errors in context, the average duration of MSA events based on human annotations was 65.27 ± 13.22 s for insulin pens and 34.30 ± 7.12 s for inhalers.

Evaluation of MSA techniques

To evaluate the MSA technique, we partitioned an MSA event into constituent key steps based on recommendations pertaining to insulin pen and inhaler device administration18,19. Figure 4 illustrates the details of these steps. Besides detecting the occurrence of an MSA event, our model also predicts the time window for each of the individual steps involved during the administration. To evaluate its performance, we compared the predicted duration of each individual step with human annotations. Figure 5a shows the box plots of the duration estimation errors for eight different steps during MSA events with insulin pens. Similarly, Fig. 5b plots the duration estimation errors for six different steps during MSA events with inhaler devices. Our model made an unbiased duration estimation for all the steps, and the interquartile range was smaller than 1.5 s for all the steps.

Fig. 4: Partitioning of key constituent steps of insulin pen and inhaler device self-administration enabling evaluation of administration technique.
figure 4

Based on recommendations pertaining to insulin pen and inhaler device administration, we partitioned an MSA event into eight steps and six steps for insulin and inhaler administration, respectively. (We obtained informed consent from the participant for the use of his photographs).

Fig. 5: Evaluation results for estimating the duration for constituent key steps and detecting MSA errors during MSA events with insulin pens and inhaler devices.
figure 5

a,b, Distribution of the step duration estimation errors for insulin (n = 620) and inhaler (n = 583) administration, respectively. On each box plot, the central line indicates the median, and the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to 1.5 times the interquartile range. Points beyond the whiskers are plotted individually using the ‘+’ symbol. c,d, ROC curves for detecting MSA errors of missing a key step during insulin (n = 620) and inhaler (n = 583) administration. ROC curves demonstrate the tradeoff between sensitivity and specificity as the detection thresholds are varied. The shaded AUC is an aggregate measure of detection performance. e,f, ROC curves for detecting MSA errors of failing to comply with duration requirements during insulin (n = 620) and inhaler (n = 583) administration.

Based on the detection of individual steps and estimation of their duration, we further looked at two common errors of MSA: 1) missing a key step during the administration process and 2) failure to comply with the duration requirements of device administration. The above MSA errors occur frequently and are associated with poor disease control outcomes3,20,29. For example, studies have reported that 37% of patients fail to shake their inhalers29, and patients not holding their breath after inhalation was a prevalent (53%) error during inhalation device administration29.

MSA errors with a missing step considered in this study were not priming the insulin pen (Step 4) and not shaking the inhaler device before use (Step 2). These steps are crucial to make sure these devices deliver the medication at the right dose. Specifically, priming the insulin pen ensures an unobstructed and free flow of insulin20, and shaking the inhaler ensures proper mixture of particles and consistent dose delivery23. Figure 5c,d show the ROC curves for detecting such errors during insulin and inhaler administration, respectively. Our system detected not priming the insulin pen with a sensitivity of 84.00% (95% CI, 76.9–89.2%) and a specificity of 92.55% (95% CI, 89.7–94.7%) and an AUC of 0.905. Similarly, our system detected not shaking the inhaler device before use with a sensitivity of 96.64% (95% CI, 91.9–98.7%) and a specificity 94.47% (95%, CI 91.8–96.3%) and an AUC of 0.967.

To evaluate our system’s performance in detecting errors of failing to comply with duration requirements, we considered two duration-related common steps—namely, holding the insulin pen still for 10 s after injection (Step 6 of insulin administration) and holding one’s breath for 10 s after inhaling a dose (Step 4 of inhaler administration). We use the cutoff of 10 s base on clinical recommendations20,23,30. Specifically, holding the insulin pen for 10 s after injection ensures no insulin leakage or dribbling20, and holding the breath after dose inhalation ensures adequate lung deposition, which occurs through sedimentation of particles23,30. Figure 5e and Fig. 5f show the ROC curves for detecting MSA errors when individuals failed to comply with the duration requirements during insulin and inhaler administration, respectively. Our system detected not holding the insulin pen still for 10 s after injection with a sensitivity of 94.19% (95% CI, 88.9–97.1%) and a specificity of 95.48% (95% CI, 93.1–97.1%) and an AUC of 0.981. Similarly, our system detected not holding breath after inhaling a dose with a sensitivity of 89.88% (95% CI, 84.0–93.8%) and a specificity of 92.04% (95% CI, 88.9–94.4%) and an AUC of 0.953. This performance rose further for detecting significant deviations from the recommendations (for example, failing to hold breath/pen even for a few seconds). For the insulin pen, the AUC increased to 0.986 and 0.993 for detecting durations shorter than 5 s and 3 s, respectively. Similarly, the AUC increased to 0.983 and 0.988 in the case of inhalers.

In Fig. 6 we show example outputs from our system. Figure 6a shows an example MSA event with the insulin pen. The top panel plots the predictions of our system on the time axis—that is, a detected MSA event with multiple boxes corresponding to all eight steps when using an insulin pen. The bottom panel shows the human annotation during the corresponding MSA event. Figure 6b shows an example when the individual missed a key step during the administration process. Note that the box corresponding to the step of priming the insulin dose (Step 4) is missing, and this error was successfully detected by our AI model. Figure 6c shows another insulin pen example where the individual failed to comply with administration duration requirements—namely, the individual failed to hold the pen for 10 s after injecting the dose. Again, our model was able to detect this error, as the detected step of holding the insulin pen after injection (Step 6) was much shorter than 10 s. Similarly, Fig. 6d–f show example outputs with inhalers.

Fig. 6: Example outputs from our AI-based system.
figure 6

We compared the prediction of the AI-based solution and the ground truth provided by human annotation. a, An insulin administration event without any error. b, An insulin administration event that missed the step of priming the insulin pen (Step 4). c, An insulin administration event where the individual did not hold the pen for 10 s after injection (Step 6). d, An inhaler administration event without any error. e, An inhaler administration event that missed the step of shaking the inhaler device before use (Step 2). f, An inhaler administration event where the individual did not hold their breath after inhaling a dose (Step 4).

Discussion

Here we described an AI-based solution for contactless at-home assessment of patient MSA using inhalers and insulin pens. Our solution is characterized by three properties: low overhead, informative and accurate. It is low overhead because it works in a passive and contactless manner without requiring patients or health professionals to observe, report or measure any parameters. It is informative because, in addition to detecting patient medication administration, it also assesses the patient’s self-administration technique and informs her of errors and omissions of required steps. It is also accurate as demonstrated through our empirical results.

We think that the above three properties are important for the success of an MSA assessment solution. Past solutions for assessing MSA at home fall short of delivering all three properties. In particular, solutions that attach sensors to medication devices to monitor MSA31,32,33 can impose a new burden on the patient, as they require the patient to regularly charge or replace their battery and bring the devices in the vicinity of a smartphone so they can upload their data. Although such solutions can detect dose release, they lack information on whether the patient followed the proper MSA technique to ensure adequate dose delivery—that is, the sensor captures the actuation and movements of the medication device itself but cannot capture the patient’s actions and their sequence, which are crucial for correct MSA. To our knowledge, this paper is the first to introduce an automated solution for assessing an individual’s MSA technique and whether it follows the proper steps. Being able to assess MSA techniques is essential because failures to follow the proper techniques are common and have been associated with high non-adherence levels and subsequent poor disease outcomes3,20,28,34.

Another feature of our approach is its ability to generalize to different types of insulin pens and inhalers. The neural network models used in this study support both reusable and disposable insulin pens and the widely used metered-dose inhalers, as the constituent key steps that our system learned are similar regardless of the type of insulin or the medication delivered by the inhaler. Specifically, in the case of insulin pens, our model is trained to support both reusable and disposable pens by considering the cartridge-loading step as optional. In the case of inhalers, our model is trained on the MSA steps recommended for the widely used metered-dose inhalers. Because the main difference between different types of metered-dose inhalers is the actual drug administered (for example, salbutamol, ipratropium ot fluticasone) rather than the administration technique itself, our system works with all such inhalers. Furthermore, our model can be extended to work with dry powder inhalers, which do not require shaking before use. This can be done by using a flag to indicate that the patient uses a dry powder inhaler and, therefore, not declaring an MSA error when the shaking step is missed.

We think that the clinical implications of our system could be significant. We envision that this system will be able to provide continuous feedback for clinicians on their patients’ MSA. Based on the feedback from our system, health professionals can then make a clinical judgment (for example, whether more training and education on medication device administration techniques is needed for the patient). Additionally, this system could contribute to patient empowerment and engagement in their health by giving them feedback about their MSA technique and allowing them to avoid common MSA errors.

Although our AI-based solution provides an important improvement over the status quo, we also note that it has several limitations. First, our system was developed and tested with healthy individuals in laboratory conditions. We designed the experiment in this study to emulate the real-world scenarios of how patients use medication delivery devices after the initial training from pharmacists or other health professionals. Thus, we chose individuals without prior experience with the insulin pen and inhaler and had them trained by a pharmacist to use these devices. This also ensured that our participants’ level of education and training in using medication devices was standardized, therefore mitigating reported barriers associated with individuals’ lack of training and education when using an insulin pen and inhaler device3,35. We think that this study provides an important first step toward enabling automatic MSA assessment at home. We envision that future work would validate the system with actual patients in their homes and study the effect of having such a system for automatic at-home MSA assessment on medication adherence. Future work could also evaluate potential confounding factors that might affect MSA errors, such as the patient’s chronic conditions, dexterity issues, health literacy and education level.

Second, we focused on insulin pens and metered-dose inhalers and their common errors, but there are many other MSA devices and potential technique errors. Although this is a limitation of the specific neural network that we trained, the AI approach that we propose is general and can be adapted to other MSA devices and MSA errors.

Third, the ability of our system to detect MSA events in various locations in the home is limited by the coverage area of the radio. The radio device that we used in this study can assess MSA events in locations up to 10 m from the device. This is usually enough to cover several rooms in a home. If desirable, however, the whole home can be covered by deploying multiple radio devices. Still, patients might take their medications outside the home (for example, at work), leading to some MSA events being missed. Even when MSA detection is incomplete, the system continues to be useful. Specifically, it would provide health professionals with a list of missing MSA events, which allows them to discuss this information with patients to clarify whether the missing MSA events are due to incomplete information or the patients indeed did not take their medication. Furthermore, it would detect MSA technique errors, which are typically repeated by patients, and today often go undetected until direct observation from clinicians or poor disease outcomes11,36.

Fourth, the system does not detect MSA events if the person has his back to the radio, because most of the RF signals are blocked by the person’s own body. Similarly to the previous limitation, this issue can be addressed by deploying a second radio in the environment with a different orientation.

Additionally, the exposition in this paper focused on scenarios where the house has a single person who uses an inhaler and/or insulin pen. For homes with multiple patients who use inhalers or insulin pens, a user identification system based on RF reflections37,38,39,40 can be employed to resolve the ambiguity. Such systems use RF signals to accurately identify a person from a small set of people—for example, other residents in a home or co-workers in an office scenario. Because we only require identification from others at home, their methods apply to this scenario.

In summary, we developed an AI system that can successfully detect MSA events and assess a patient’s MSA technique. Our system demonstrates how AI can be applied to ensure medication safety, specifically with device-based administration, in a manner that has minimal potential overhead for patients and health professionals.

Methods

Experiment design

When designing the experiments in this study, we aimed to emulate the real-world scenarios of how patients use medication delivery devices. Patients typically receive training from pharmacists or other health professionals on how to use their medication delivery device when they are prescribed such medication for the first time41. To emulate the real-world scenarios, we chose individuals without prior experience with insulin pens and inhalers and had them trained by a pharmacist to use those devices. The pharmacist followed a standard procedure where he first taught the individual the MSA process and then asked the individual to demonstrate their MSA technique and ensured that the individual correctly simulated the administration of their insulin pen and inhaler device. After the initial training session, the individual performed MSA in front of a wireless sensor and a camera that recorded videos for annotation purposes. In addition to performing MSA, individuals were instructed to perform other activities, such as typing, cooking, eating and interacting with surrounding objects. We annotated the exact time window for every step involved in each MSA event and the types of errors that were made. To mitigate the imbalance between MSA events with and without errors and facilitate the development of AI models, we asked the individuals to purposely simulate errors during the experiment sessions. Note that, during both the initial training session and the later experiment session, all the MSA events were performed using placebo devices, and no medication dose was actually administered.

Individuals and dataset

A total of 107 healthy individuals (18–72 years of age) were recruited for this study. The individuals performed 1,203 positive examples of MSA events with insulin pens and inhaler devices at 40 different locations (offices, lounges, seminar rooms, kitchens, halls, etc). Positive MSA events were compared against a total of 46,585 instances of negative MSA examples corresponding to common activities that do not involve MSA. Of the 1,203 MSA events, 620 used insulin pens, and 583 used inhalers for administration. For the MSA events with insulin pens, 150 of them missed a common step, and 155 of them failed to comply with duration requirements. For the MSA events with inhaler devices, 149 of them missed a common step, and 168 of them failed to comply with duration requirements. None of the MSA events simultaneously missed a step and failed to comply with duration requirements.

During the experiments, the individuals were allowed to move freely in the space and perform the MSA at a location of their choice, within 10 m from the wireless sensor. The individuals were recorded both in sitting and standing positions via the wireless sensor. They were allowed to pick any orientation with respect to the wireless sensor (that is, face the sensor or show their sides to it) except for having their back facing the sensor.

RF sensing technology

Recent advances in RF sensing have developed systems that can capture human motion and infer biometric information, such as respiration, heart rate, gait speed, mobility, sleep stages and human pose14,15,16,17,37,42,43,44. Similarly to this past work, we used a radio sensor that employs FMCW and antenna arrays. The system works by transmitting low-power RF signals (1,000 times weaker than Wi-Fi) and receiving reflections from nearby people. Because up to 60% of the human body is water, it reflects the radio signals and modulates them with the person’s movements, capturing important information about the person’s MSA technique. The radio is equipped with vertical and horizontal antenna arrays, each of which has 12 antennas. It transmits an FMCW chirp sweeping the frequencies from 5.4 to 7.2 GHz. The combination of FMCW and antenna arrays allows the radio to separate RF reflections from different areas based on their distance (that is, range) and spatial direction (that is, angle of arrival) with respect to the radio sensor14,43. This property allows the system to separate reflections from different people and process them independently.

We processed the RF signal into three-dimensional (3D) tensors indicating the amount of RF reflection from each point in the 3D space43. We generated 30 such tensors (that is, frames) every second.

Building the AI-based model

Our AI-based model processes RF signals through three stages to detect and assess MSA events. The first stage uses a neural network model to localize and track people in the environment, zooming in on each individual while eliminating noise and interference from other people and objects in the environment. The second stage uses another neural network model to perform frame-wise prediction of MSA steps, where each frame is a snapshot of the 3D RF tensor at one point in time. Finally, the third stage decodes the frame-wise predictions into start time and end time of each MSA step and analyzes the sequence of steps to determine whether an MSA event has occurred. Below, we describe all three stages in detail.

Stage 1: The first stage of the processing takes a stream of radio signals as input and outputs bounding boxes43 representing the spatial locations of each individual. By focusing on RF reflections from the spatial locations indicated by the bounding boxes, our model zooms in on each individual while eliminating noise and interference from other people and objects in the environment. We used the same neural network as previous work43 to localize and track people in the environment. This neural network model uses a 12-layer ResNet to extract features from RF signals together with a region proposal network that outputs bounding boxes43.

Stage 2: The second stage takes the RF frames focused on a specific individual from the previous stage as input and outputs for each RF frame a probability score of the frame belonging to each of the MSA steps. The neural network used in this stage has a UNet structure45 with 3D convolutional layers. Specifically, it has eight residual blocks, each of which consists of three convolutional layers, along with group normalization layers and exponential linear unit layers46,47. It also interleaves four long short-term memory layers48 within the last four residual blocks to capture temporal information. This sub-network is trained using human annotations of MSA steps. The model is implemented in PyTorch. During training, the weights of the model are randomly initialized, and we use cross-entropy loss computed for each RF frame. Adam optimizer is used with a learning rate of 3 × 10−4. We use a batch size of 4 on four NVIDIA TITAN Xp graphical processing units with distributed data parallelization. The model is trained for 100,000 iterations with a 10× learning rate decay after 20,000 and 50,000 iterations.

Stage 3: The third stage of the processing decodes the frame-wise MSA step probabilities to estimate the start time and end time of each MSA step and determine whether an MSA event has occurred. We adopt beam search decoding, which is widely used in speech and handwritten text recognition for decoding the output of neural network models49. At a high level, there is an analogy between recognizing a spoken word by detecting the sequence of its phonemes and detecting an MSA event by detecting the sequence of its steps (and their corresponding RF frames). The beam search decoding algorithm considers all the frames jointly and uses language models as prior knowledge to output a coherent sequence of characters/words as opposed to a greedy decoding scheme that decodes each frame independently. Similarly, the beam search decoding algorithm in our model uses priors of the transition probability between MSA steps and the step duration, which are based on the statistics of the training data. The beam search decoding also computes a score (that is, log likelihood) for the decoded results. The score is normalized by the duration of the detected MSA event, and our model rejects all MSA events if their final score is less than a threshold. (The threshold is set to 0.4, which balances sensitivity with specificity).

Statistical methods

To evaluate the performance of MSA event detection, we used the following metrics: sensitivity, specificity, ROC curves, AUC and the estimation error of start time and end time, with sample sizes as given. To evaluate the performance of MSA error detection, we used the following metrics: estimation error of step duration, sensitivity and specificity of MSA error detection, ROC curves and AUC.

Sensitivity and specificity are calculated as (TP: true positive; FN: false negative; TN: true negative; FP: false positive):

$${\mathrm{Sensitivity}} = \frac{\mathrm{TP}}{{\mathrm{TP}} + {\mathrm{FN}}},$$
$${\mathrm{Specificity}} = \frac{{\mathrm{TN}}}{{\mathrm{TN}} + {\mathrm{FP}}}.$$

We plotted ROC curves that demonstrate the tradeoff between sensitivity and specificity, as the detection thresholds are varied. When reporting the sensitivity and specificity, we used a detection threshold of 0.6 for MSA event detecting, a detection threshold of 0.25 s for detecting missing steps and a detection threshold of 8 s for detecting the error of failing to comply with duration-based requirements. We followed standard procedures to calculate the 95% CI for sensitivity and specificity50. We also reported AUC, which is the area under the corresponding ROC curves showing an aggregate measure of detection performance.

We computed the error between the predicted start (or end) time and the ground truth start (or end) time of events as (ts: ground truth start time; te: ground truth end time; \(\widehat {t_s}\): predicted start time; \(\widehat {t_e}\): predicted end time):

$$error_s = \left| {\widehat {t_s} - t_s} \right|,$$
$$error_e = \left| {\widehat {t_e} - t_e} \right|.$$

The error of step duration estimation is computed as (d: ground truth duration; \(\hat d\): predicted duration):

$$error_d = \left| {\hat d - d} \right|.$$

We reported the error of start/end time estimation and step duration estimation with box plots. For each box plot, the central line indicates the median, and the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to 1.5 times the interquartile range. Points beyond the whiskers are plotted individually using the ‘+’ symbol.

Human subject data

Our study was approved by the institutional review board of the Massachusetts Institute of Technology (protocol no. 1509173592). All individuals reviewed and signed consent forms before participating in our study.

Reporting Summary

Further information on research design is available in the Nature Research Reporting Summary linked to this article.