Abstract
Errors in medication self-administration (MSA) lead to poor treatment adherence, increased hospitalizations and higher healthcare costs. These errors are particularly common when medication delivery involves devices such as inhalers or insulin pens. We present a contactless and unobtrusive artificial intelligence (AI) framework that can detect and monitor MSA errors by analyzing the wireless signals in the patient’s home, without the need for physical contact. The system was developed by observing self-administration conducted by volunteers and evaluated by comparing its prediction with human annotations. Findings from this study demonstrate that our approach can automatically detect when patients use their inhalers (area under the curve (AUC) = 0.992) or insulin pens (AUC = 0.967), and assess whether patients follow the appropriate steps for using these devices (AUC = 0.952). The work shows the potential of leveraging AI-based solutions to improve medication safety with minimal overhead for patients and health professionals.
Similar content being viewed by others
Main
Poor medication adherence is a major healthcare problem, contributing to 10% of hospitalizations, 125,000 deaths per year and up to $290 billion in annual cost in the United States alone1,2. A crucial step toward achieving medication adherence is ensuring proper MSA—that is, ensuring that patients take medications at the prescribed time and use the proper self-administration technique. When patients fail to perform MSA properly, the net result is that the medication is not delivered to its intended action site, causing failures in managing the condition3. Unfortunately, MSA errors are common, particularly in chronic diseases where up to 50% of patients do not take medications as prescribed4,5. The problem is exacerbated when medication delivery requires devices such as insulin pens or inhalers. MSA errors associated with medication delivery devices (for example, not shaking the inhaler before use or not priming the insulin dose) result in administration failures, subsequent high levels of non-adherence, reduced disease control and unscheduled use of healthcare resources6,7,8,9. Physicians report that up to 70% of their patients do not take their insulin as prescribed6,7. Similarly, over 50% of patients who use inhalers do so erroneously regardless of the inhalation device used8. These statistics are alarming given that hundreds of millions of patients worldwide depend on these devices for their medication administration9,10.
Addressing the above problems requires adequately assessing patients’ MSA and detecting MSA errors. Existing solutions to assess MSA typically require direct observation by health professionals—that is, a clinician or pharmacist watches the patient as she uses her inhaler or insulin pen and guides her through the proper administration technique. For example, the clinician would explain to the patient that she needs to shake the inhaler, fully exhale, inhale a dose and hold her breath for 10 s before exhaling. The clinician would also watch the patient performing these steps and alert her if she fails to follow the proper administration technique. Although this approach can be used in the clinic, most MSA errors occur at home and away from the observation of a clinician. Furthermore, patients’ performance in front of a clinician might be unrepresentative of their actual MSA, as patients tend to perform better when assessed by a clinician, a phenomenon known as ‘white-coat compliance’11. Even when patients receive initial training on their devices by a clinician, MSA errors occur over time due to forgetfulness or recklessness in adhering to the prescribed administration time, frequency or technique12. As a result, many MSA errors end up undetected until they manifest as serious health problems or admissions to the emergency room12.
This study was motivated by the question of whether we could use AI to assist with the observation of patients at home and provide a continuous assessment of their MSA. We present an AI-based solution that achieves this goal in an accurate, efficient and cost-effective manner. Our solution avoids cameras, which many patients find to be intrusive when deployed in their homes13. Instead, our solution uses a Wi-Fi-like sensor deployed in patient homes. (The sensor transmits signals around the Wi-Fi frequency range using the frequency-modulated continuous-wave (FMCW) technique. A detailed description of the radio sensor can be found in the Methods section.) The sensor transmits a very low-power wireless signal (1,000 times lower power than standard Wi-Fi), and our system analyzes the reflections of the signal from the environment using AI techniques. Because up to 60% of the human body is water, it reflects the surrounding radio signals and modulates them with the person’s movements14. Past work has shown that such radio reflections can be used to capture breathing and heart rate, detect falls and monitor sleep15,16,17. In this study, we focused on movements associated with MSA events. Our AI system, embedded in the sensor, analyses the radio reflections from the environment to track the specific movements associated with MSA and to detect when a patient administers her medication using an inhaler or insulin pen. It further examines the wireless reflections to detect whether the patient has followed the required steps of using the medication device and generates an alert if the patient fails to follow the proper technique (for example, forgot to prime her insulin pen or shake her inhaler). This AI-based solution works in a contactless and passive manner, introducing no burden on the patient, caregiver or health personnel.
Figure 1 illustrates a use case of our AI system at home, where it assesses the individual’s MSA with an inhaler. The wireless sensor is mounted on the wall like a Wi-Fi box (Fig. 1a). There is no need for cameras, wearable sensors or any additional smart devices. The wall-mounted sensor analyzes the surrounding radio signal using AI methods. In this case, it would detect an instance of MSA using an inhaler and document the corresponding time. The AI solution also tracks the motion of the person and detects that the person shook the device, exhaled before use and, finally, inhaled a dose (Fig. 1b–d), which are required steps for MSA with an inhaler. An example output of our system is available in the Supplementary Video.
To build the AI-based solution, we designed a study where health professionals and the AI-based solution simultaneously observe MSA events with insulin pens and inhaler devices. To emulate real-world standard practice for first-time users of these medications, participants were trained by a pharmacist to perform MSA according to current guidelines and recommendations18,19 and were then asked to demonstrate their MSA technique. During the experiment, human observers provided an MSA assessment that included the time window of each MSA event and the errors made during each event (if any). We used the MSA assessment provided by human observers to train and evaluate our AI algorithm. In total, we collected a large dataset that consists of 47,788 examples, where each example is a 2-min recording of radio signals. This dataset has 1,203 positive examples of MSA events with insulin pens and inhaler devices; about half of these MSA events are performed with no errors, whereas the other half includes some errors (for example, not shaking the inhaler before use or not holding one’s breath after inhaling the dose). The dataset also includes 46,585 negative MSA examples corresponding to common home activities that do not involve MSA, such as cooking, eating, typing and interacting with objects such as glasses, clothes, microwaves and hairdryers. The MSA examples in the dataset were performed by 107 healthy individuals whose ages varied from 18 to 72 years. The dataset was divided into training and testing sets that we used to train and evaluate the AI system, respectively.
Extensive experimental results (that are detailed in the Results section) demonstrate that our AI-based solution can reliably detect the occurrence of MSA events. Specifically, the AUC was 0.992 for detecting the use of an inhaler and 0.967 for detecting the use of an insulin pen. These results indicate that an AI system could be used at home to monitor whether patients use their inhalers and insulin pens following the prescribed time and frequency.
The experimental results also show that the AI solution can accurately evaluate whether the individual correctly followed the required steps for administering her medication using an inhaler or insulin pen. Adherence to the proper steps while performing MSA is crucial for disease management and therapeutic effectiveness20. For example, failure to follow the correct steps when using an insulin pen can lead to hyperglycemia or severe hypoglycemic episodes for patients with diabetes21,22. Similarly, failure to follow the recommended steps during inhaler administration contributes to symptom exacerbations and subsequent reduced quality of life for patients with asthma and patients with chronic obstructive pulmonary disease (COPD)23,24,25. Our results show that the AI system reliably detects both 1) missing key steps during the administration process (for example, not shaking the inhaler before use or not priming the insulin pen) and 2) patients not following duration-based requirements (for example, not holding the insulin pen after injection for 10 s).
Figure 2 illustrates how we envision such an AI-based solution that could be used in patient homes to help detect and address MSA errors. Our wireless sensor would be deployed in the patient’s home. The AI system would continuously and automatically analyze the radio signals and document MSA assessment results, which are uploaded over the internet and appended to the patient’s digital health record. Reminders will be sent to the patient if she fails to take the medication at the prescribed time. Authorized health professionals will also be able to access these records via a web portal to learn which patients have difficulties with their MSA and the types of errors they experience. The health professionals can then reach out to the patient to corroborate these results and make a clinical judgment (for example, whether more training on medication device administration is needed for the patient).
The recent outbreak of Coronavirus Disease 2019 (COVID-19) emphasizes the need for an automated and contactless solution for assessing MSA at home. The stay-at-home orders make it even more difficult to assess MSA through direct observation by health professionals. At the same time, individuals suffering from asthma, COPD and diabetes are at higher risk for severe illness from COVID-19 (refs. 26,27); hence, it is even more critical to ensure that they take their medications with the proper administration technique. Our automated and contactless AI-based MSA assessment solution could help these vulnerable populations to control their chronic conditions. It also enables health professionals to remotely monitor the MSA of their patients, without risks of contagion.
Our work shows how advances in AI can address an important unmet need in healthcare6,7,8,9,10, by continuously monitoring the MSA of patients in their homes, detecting when patients fail to use their medication devices as prescribed and providing patients with feedback on their medication administration technique and whether it follows the required steps. More generally, the work opens the door to the integration of AI-based solutions in care management through in-home passive, unobtrusive and contactless patient monitoring. Such integration could improve outcomes for patients and reduce the cost of healthcare.
Results
Our system was built and validated on a large dataset that consists of 47,788 examples of MSA events and home activities that do not involve MSA from 107 participants (53 females and 54 males), where each example is a recording of radio signals up to 2 min. A total of 40 different locations (such as offices, lounges, seminar rooms, kitchens and halls) were chosen to allow for variation between environmental conditions. Participants were allowed to move freely in the space and perform the MSA at a location of their choice, within 10 m from the wireless sensor. They were either standing or sitting when performing MSA and were allowed to pick any orientation with respect to the wireless sensor except for having their back to the sensor. For the MSA events with insulin pens, 150 events missed a common step, and 155 of them failed to comply with a specific duration requirement. For the MSA events with inhaler devices, 149 of them missed a required step, and 168 of them failed to comply with duration requirements.
We trained a neural network model that takes a stream of radio signals as input. It first detects and tracks the location of each individual in the environment. It then zooms in on the radio signals pertaining to each individual and predicts the occurrence of an MSA event. Instead of detecting the whole MSA event directly, our model detects the steps involved during administration and only claims an occurrence of an MSA event when multiple steps happen one after another. Detecting MSA events in this way improves our model’s robustness to variations among different people and enables the detection of MSA errors. Details of the model are described in the Methods section.
We performed K-fold cross-validation (K = 5) to evaluate our model. Specifically, the dataset was randomly split into five equal-sized subsets. A single fold was retained as test data, and the remaining four folds were used for training. This process was repeated five times, with each of the five folds used exactly once as the test data. The folds were divided such that participants who appear in the training data do not appear in the testing data and vice versa. To balance the number of participants across folds, we imposed no constraints on allocating sites to folds. We note that the cross-validation was not used for hyperparameter tuning.
Detection of MSA events
Our model detects MSA events in a sliding-window fashion. Specifically, it detects whether an MSA has happened for each 2-min window. To evaluate the performance of our model, we compared its predictions with ground truth provided by human annotations. Figure 3a,b show the receiver operating characteristic (ROC) curves for detecting MSA events with insulin pens and inhaler devices, respectively. When computing sensitivity and specificity, positive examples indicate MSA events, whereas negative examples indicate non-MSA events. Our system detected the occurrence of an insulin pen administration event with a sensitivity of 87.58% (95% confidence interval (CI), 84.7–90.0%) and a specificity of 96.06% (95% CI, 95.9–96.2%) and an AUC of 0.967. Similarly, inhaler administration events were detected with a sensitivity of 91.08% (95% CI, 88.4–93.2%) and a specificity of 99.22% (95% CI, 99.1–99.3%) and an AUC of 0.992. We note that the specificity or the false-positive ratio is computed for windows of non-MSA events such as eating, drinking or putting on clothes, not just any window of radio frequency (RF) signals. The number of false positives when considering any window of RF signals is significantly smaller in real-world deployment. Specifically, we leveraged a dataset in which the radio was used to monitor patients with Parkinson’s disease and control individualss (that is, healthy individuals) for over 1 month28. Because none of the individuals in this dataset used inhalers or insulin pens, all detected MSA events could be considered false positives. We considered five homes from the study and used one full month of RF signals from each home. On average, the number of false positives over a whole month was 2.2 for insulin pens and 6.6 for inhalers.
We also looked at errors that our model made when estimating the start time and end time of an MSA event. Figure 3c shows the box plots for the estimation errors, and Fig. 3d,e shows the cumulative distribution functions of the absolute estimation errors. Our system made an unbiased (that is, median-unbiased) estimation of the start time and the end time for both devices. For the start time estimation, the 50th percentile error was 0.6 s and 0.4 s, whereas the 90th percentile error was 2.0 s and 1.3 s for insulin pens and inhaler devices, respectively. Similarly, for the end time estimation, the 50th percentile error was 0.4 s and 0.3 s, whereas the 90th percentile error was 1.4 s and 0.9 s for insulin pens and inhaler devices, respectively. To put these errors in context, the average duration of MSA events based on human annotations was 65.27 ± 13.22 s for insulin pens and 34.30 ± 7.12 s for inhalers.
Evaluation of MSA techniques
To evaluate the MSA technique, we partitioned an MSA event into constituent key steps based on recommendations pertaining to insulin pen and inhaler device administration18,19. Figure 4 illustrates the details of these steps. Besides detecting the occurrence of an MSA event, our model also predicts the time window for each of the individual steps involved during the administration. To evaluate its performance, we compared the predicted duration of each individual step with human annotations. Figure 5a shows the box plots of the duration estimation errors for eight different steps during MSA events with insulin pens. Similarly, Fig. 5b plots the duration estimation errors for six different steps during MSA events with inhaler devices. Our model made an unbiased duration estimation for all the steps, and the interquartile range was smaller than 1.5 s for all the steps.
Based on the detection of individual steps and estimation of their duration, we further looked at two common errors of MSA: 1) missing a key step during the administration process and 2) failure to comply with the duration requirements of device administration. The above MSA errors occur frequently and are associated with poor disease control outcomes3,20,29. For example, studies have reported that 37% of patients fail to shake their inhalers29, and patients not holding their breath after inhalation was a prevalent (53%) error during inhalation device administration29.
MSA errors with a missing step considered in this study were not priming the insulin pen (Step 4) and not shaking the inhaler device before use (Step 2). These steps are crucial to make sure these devices deliver the medication at the right dose. Specifically, priming the insulin pen ensures an unobstructed and free flow of insulin20, and shaking the inhaler ensures proper mixture of particles and consistent dose delivery23. Figure 5c,d show the ROC curves for detecting such errors during insulin and inhaler administration, respectively. Our system detected not priming the insulin pen with a sensitivity of 84.00% (95% CI, 76.9–89.2%) and a specificity of 92.55% (95% CI, 89.7–94.7%) and an AUC of 0.905. Similarly, our system detected not shaking the inhaler device before use with a sensitivity of 96.64% (95% CI, 91.9–98.7%) and a specificity 94.47% (95%, CI 91.8–96.3%) and an AUC of 0.967.
To evaluate our system’s performance in detecting errors of failing to comply with duration requirements, we considered two duration-related common steps—namely, holding the insulin pen still for 10 s after injection (Step 6 of insulin administration) and holding one’s breath for 10 s after inhaling a dose (Step 4 of inhaler administration). We use the cutoff of 10 s base on clinical recommendations20,23,30. Specifically, holding the insulin pen for 10 s after injection ensures no insulin leakage or dribbling20, and holding the breath after dose inhalation ensures adequate lung deposition, which occurs through sedimentation of particles23,30. Figure 5e and Fig. 5f show the ROC curves for detecting MSA errors when individuals failed to comply with the duration requirements during insulin and inhaler administration, respectively. Our system detected not holding the insulin pen still for 10 s after injection with a sensitivity of 94.19% (95% CI, 88.9–97.1%) and a specificity of 95.48% (95% CI, 93.1–97.1%) and an AUC of 0.981. Similarly, our system detected not holding breath after inhaling a dose with a sensitivity of 89.88% (95% CI, 84.0–93.8%) and a specificity of 92.04% (95% CI, 88.9–94.4%) and an AUC of 0.953. This performance rose further for detecting significant deviations from the recommendations (for example, failing to hold breath/pen even for a few seconds). For the insulin pen, the AUC increased to 0.986 and 0.993 for detecting durations shorter than 5 s and 3 s, respectively. Similarly, the AUC increased to 0.983 and 0.988 in the case of inhalers.
In Fig. 6 we show example outputs from our system. Figure 6a shows an example MSA event with the insulin pen. The top panel plots the predictions of our system on the time axis—that is, a detected MSA event with multiple boxes corresponding to all eight steps when using an insulin pen. The bottom panel shows the human annotation during the corresponding MSA event. Figure 6b shows an example when the individual missed a key step during the administration process. Note that the box corresponding to the step of priming the insulin dose (Step 4) is missing, and this error was successfully detected by our AI model. Figure 6c shows another insulin pen example where the individual failed to comply with administration duration requirements—namely, the individual failed to hold the pen for 10 s after injecting the dose. Again, our model was able to detect this error, as the detected step of holding the insulin pen after injection (Step 6) was much shorter than 10 s. Similarly, Fig. 6d–f show example outputs with inhalers.
Discussion
Here we described an AI-based solution for contactless at-home assessment of patient MSA using inhalers and insulin pens. Our solution is characterized by three properties: low overhead, informative and accurate. It is low overhead because it works in a passive and contactless manner without requiring patients or health professionals to observe, report or measure any parameters. It is informative because, in addition to detecting patient medication administration, it also assesses the patient’s self-administration technique and informs her of errors and omissions of required steps. It is also accurate as demonstrated through our empirical results.
We think that the above three properties are important for the success of an MSA assessment solution. Past solutions for assessing MSA at home fall short of delivering all three properties. In particular, solutions that attach sensors to medication devices to monitor MSA31,32,33 can impose a new burden on the patient, as they require the patient to regularly charge or replace their battery and bring the devices in the vicinity of a smartphone so they can upload their data. Although such solutions can detect dose release, they lack information on whether the patient followed the proper MSA technique to ensure adequate dose delivery—that is, the sensor captures the actuation and movements of the medication device itself but cannot capture the patient’s actions and their sequence, which are crucial for correct MSA. To our knowledge, this paper is the first to introduce an automated solution for assessing an individual’s MSA technique and whether it follows the proper steps. Being able to assess MSA techniques is essential because failures to follow the proper techniques are common and have been associated with high non-adherence levels and subsequent poor disease outcomes3,20,28,34.
Another feature of our approach is its ability to generalize to different types of insulin pens and inhalers. The neural network models used in this study support both reusable and disposable insulin pens and the widely used metered-dose inhalers, as the constituent key steps that our system learned are similar regardless of the type of insulin or the medication delivered by the inhaler. Specifically, in the case of insulin pens, our model is trained to support both reusable and disposable pens by considering the cartridge-loading step as optional. In the case of inhalers, our model is trained on the MSA steps recommended for the widely used metered-dose inhalers. Because the main difference between different types of metered-dose inhalers is the actual drug administered (for example, salbutamol, ipratropium ot fluticasone) rather than the administration technique itself, our system works with all such inhalers. Furthermore, our model can be extended to work with dry powder inhalers, which do not require shaking before use. This can be done by using a flag to indicate that the patient uses a dry powder inhaler and, therefore, not declaring an MSA error when the shaking step is missed.
We think that the clinical implications of our system could be significant. We envision that this system will be able to provide continuous feedback for clinicians on their patients’ MSA. Based on the feedback from our system, health professionals can then make a clinical judgment (for example, whether more training and education on medication device administration techniques is needed for the patient). Additionally, this system could contribute to patient empowerment and engagement in their health by giving them feedback about their MSA technique and allowing them to avoid common MSA errors.
Although our AI-based solution provides an important improvement over the status quo, we also note that it has several limitations. First, our system was developed and tested with healthy individuals in laboratory conditions. We designed the experiment in this study to emulate the real-world scenarios of how patients use medication delivery devices after the initial training from pharmacists or other health professionals. Thus, we chose individuals without prior experience with the insulin pen and inhaler and had them trained by a pharmacist to use these devices. This also ensured that our participants’ level of education and training in using medication devices was standardized, therefore mitigating reported barriers associated with individuals’ lack of training and education when using an insulin pen and inhaler device3,35. We think that this study provides an important first step toward enabling automatic MSA assessment at home. We envision that future work would validate the system with actual patients in their homes and study the effect of having such a system for automatic at-home MSA assessment on medication adherence. Future work could also evaluate potential confounding factors that might affect MSA errors, such as the patient’s chronic conditions, dexterity issues, health literacy and education level.
Second, we focused on insulin pens and metered-dose inhalers and their common errors, but there are many other MSA devices and potential technique errors. Although this is a limitation of the specific neural network that we trained, the AI approach that we propose is general and can be adapted to other MSA devices and MSA errors.
Third, the ability of our system to detect MSA events in various locations in the home is limited by the coverage area of the radio. The radio device that we used in this study can assess MSA events in locations up to 10 m from the device. This is usually enough to cover several rooms in a home. If desirable, however, the whole home can be covered by deploying multiple radio devices. Still, patients might take their medications outside the home (for example, at work), leading to some MSA events being missed. Even when MSA detection is incomplete, the system continues to be useful. Specifically, it would provide health professionals with a list of missing MSA events, which allows them to discuss this information with patients to clarify whether the missing MSA events are due to incomplete information or the patients indeed did not take their medication. Furthermore, it would detect MSA technique errors, which are typically repeated by patients, and today often go undetected until direct observation from clinicians or poor disease outcomes11,36.
Fourth, the system does not detect MSA events if the person has his back to the radio, because most of the RF signals are blocked by the person’s own body. Similarly to the previous limitation, this issue can be addressed by deploying a second radio in the environment with a different orientation.
Additionally, the exposition in this paper focused on scenarios where the house has a single person who uses an inhaler and/or insulin pen. For homes with multiple patients who use inhalers or insulin pens, a user identification system based on RF reflections37,38,39,40 can be employed to resolve the ambiguity. Such systems use RF signals to accurately identify a person from a small set of people—for example, other residents in a home or co-workers in an office scenario. Because we only require identification from others at home, their methods apply to this scenario.
In summary, we developed an AI system that can successfully detect MSA events and assess a patient’s MSA technique. Our system demonstrates how AI can be applied to ensure medication safety, specifically with device-based administration, in a manner that has minimal potential overhead for patients and health professionals.
Methods
Experiment design
When designing the experiments in this study, we aimed to emulate the real-world scenarios of how patients use medication delivery devices. Patients typically receive training from pharmacists or other health professionals on how to use their medication delivery device when they are prescribed such medication for the first time41. To emulate the real-world scenarios, we chose individuals without prior experience with insulin pens and inhalers and had them trained by a pharmacist to use those devices. The pharmacist followed a standard procedure where he first taught the individual the MSA process and then asked the individual to demonstrate their MSA technique and ensured that the individual correctly simulated the administration of their insulin pen and inhaler device. After the initial training session, the individual performed MSA in front of a wireless sensor and a camera that recorded videos for annotation purposes. In addition to performing MSA, individuals were instructed to perform other activities, such as typing, cooking, eating and interacting with surrounding objects. We annotated the exact time window for every step involved in each MSA event and the types of errors that were made. To mitigate the imbalance between MSA events with and without errors and facilitate the development of AI models, we asked the individuals to purposely simulate errors during the experiment sessions. Note that, during both the initial training session and the later experiment session, all the MSA events were performed using placebo devices, and no medication dose was actually administered.
Individuals and dataset
A total of 107 healthy individuals (18–72 years of age) were recruited for this study. The individuals performed 1,203 positive examples of MSA events with insulin pens and inhaler devices at 40 different locations (offices, lounges, seminar rooms, kitchens, halls, etc). Positive MSA events were compared against a total of 46,585 instances of negative MSA examples corresponding to common activities that do not involve MSA. Of the 1,203 MSA events, 620 used insulin pens, and 583 used inhalers for administration. For the MSA events with insulin pens, 150 of them missed a common step, and 155 of them failed to comply with duration requirements. For the MSA events with inhaler devices, 149 of them missed a common step, and 168 of them failed to comply with duration requirements. None of the MSA events simultaneously missed a step and failed to comply with duration requirements.
During the experiments, the individuals were allowed to move freely in the space and perform the MSA at a location of their choice, within 10 m from the wireless sensor. The individuals were recorded both in sitting and standing positions via the wireless sensor. They were allowed to pick any orientation with respect to the wireless sensor (that is, face the sensor or show their sides to it) except for having their back facing the sensor.
RF sensing technology
Recent advances in RF sensing have developed systems that can capture human motion and infer biometric information, such as respiration, heart rate, gait speed, mobility, sleep stages and human pose14,15,16,17,37,42,43,44. Similarly to this past work, we used a radio sensor that employs FMCW and antenna arrays. The system works by transmitting low-power RF signals (1,000 times weaker than Wi-Fi) and receiving reflections from nearby people. Because up to 60% of the human body is water, it reflects the radio signals and modulates them with the person’s movements, capturing important information about the person’s MSA technique. The radio is equipped with vertical and horizontal antenna arrays, each of which has 12 antennas. It transmits an FMCW chirp sweeping the frequencies from 5.4 to 7.2 GHz. The combination of FMCW and antenna arrays allows the radio to separate RF reflections from different areas based on their distance (that is, range) and spatial direction (that is, angle of arrival) with respect to the radio sensor14,43. This property allows the system to separate reflections from different people and process them independently.
We processed the RF signal into three-dimensional (3D) tensors indicating the amount of RF reflection from each point in the 3D space43. We generated 30 such tensors (that is, frames) every second.
Building the AI-based model
Our AI-based model processes RF signals through three stages to detect and assess MSA events. The first stage uses a neural network model to localize and track people in the environment, zooming in on each individual while eliminating noise and interference from other people and objects in the environment. The second stage uses another neural network model to perform frame-wise prediction of MSA steps, where each frame is a snapshot of the 3D RF tensor at one point in time. Finally, the third stage decodes the frame-wise predictions into start time and end time of each MSA step and analyzes the sequence of steps to determine whether an MSA event has occurred. Below, we describe all three stages in detail.
Stage 1: The first stage of the processing takes a stream of radio signals as input and outputs bounding boxes43 representing the spatial locations of each individual. By focusing on RF reflections from the spatial locations indicated by the bounding boxes, our model zooms in on each individual while eliminating noise and interference from other people and objects in the environment. We used the same neural network as previous work43 to localize and track people in the environment. This neural network model uses a 12-layer ResNet to extract features from RF signals together with a region proposal network that outputs bounding boxes43.
Stage 2: The second stage takes the RF frames focused on a specific individual from the previous stage as input and outputs for each RF frame a probability score of the frame belonging to each of the MSA steps. The neural network used in this stage has a UNet structure45 with 3D convolutional layers. Specifically, it has eight residual blocks, each of which consists of three convolutional layers, along with group normalization layers and exponential linear unit layers46,47. It also interleaves four long short-term memory layers48 within the last four residual blocks to capture temporal information. This sub-network is trained using human annotations of MSA steps. The model is implemented in PyTorch. During training, the weights of the model are randomly initialized, and we use cross-entropy loss computed for each RF frame. Adam optimizer is used with a learning rate of 3 × 10−4. We use a batch size of 4 on four NVIDIA TITAN Xp graphical processing units with distributed data parallelization. The model is trained for 100,000 iterations with a 10× learning rate decay after 20,000 and 50,000 iterations.
Stage 3: The third stage of the processing decodes the frame-wise MSA step probabilities to estimate the start time and end time of each MSA step and determine whether an MSA event has occurred. We adopt beam search decoding, which is widely used in speech and handwritten text recognition for decoding the output of neural network models49. At a high level, there is an analogy between recognizing a spoken word by detecting the sequence of its phonemes and detecting an MSA event by detecting the sequence of its steps (and their corresponding RF frames). The beam search decoding algorithm considers all the frames jointly and uses language models as prior knowledge to output a coherent sequence of characters/words as opposed to a greedy decoding scheme that decodes each frame independently. Similarly, the beam search decoding algorithm in our model uses priors of the transition probability between MSA steps and the step duration, which are based on the statistics of the training data. The beam search decoding also computes a score (that is, log likelihood) for the decoded results. The score is normalized by the duration of the detected MSA event, and our model rejects all MSA events if their final score is less than a threshold. (The threshold is set to 0.4, which balances sensitivity with specificity).
Statistical methods
To evaluate the performance of MSA event detection, we used the following metrics: sensitivity, specificity, ROC curves, AUC and the estimation error of start time and end time, with sample sizes as given. To evaluate the performance of MSA error detection, we used the following metrics: estimation error of step duration, sensitivity and specificity of MSA error detection, ROC curves and AUC.
Sensitivity and specificity are calculated as (TP: true positive; FN: false negative; TN: true negative; FP: false positive):
We plotted ROC curves that demonstrate the tradeoff between sensitivity and specificity, as the detection thresholds are varied. When reporting the sensitivity and specificity, we used a detection threshold of 0.6 for MSA event detecting, a detection threshold of 0.25 s for detecting missing steps and a detection threshold of 8 s for detecting the error of failing to comply with duration-based requirements. We followed standard procedures to calculate the 95% CI for sensitivity and specificity50. We also reported AUC, which is the area under the corresponding ROC curves showing an aggregate measure of detection performance.
We computed the error between the predicted start (or end) time and the ground truth start (or end) time of events as (ts: ground truth start time; te: ground truth end time; \(\widehat {t_s}\): predicted start time; \(\widehat {t_e}\): predicted end time):
The error of step duration estimation is computed as (d: ground truth duration; \(\hat d\): predicted duration):
We reported the error of start/end time estimation and step duration estimation with box plots. For each box plot, the central line indicates the median, and the bottom and top edges of the box indicate the 25th and 75th percentiles, respectively. The whiskers extend to 1.5 times the interquartile range. Points beyond the whiskers are plotted individually using the ‘+’ symbol.
Human subject data
Our study was approved by the institutional review board of the Massachusetts Institute of Technology (protocol no. 1509173592). All individuals reviewed and signed consent forms before participating in our study.
Reporting Summary
Further information on research design is available in the Nature Research Reporting Summary linked to this article.
Data availability
De-identified data that support the findings of this study are available from the authors upon reasonable request and with a signed data use agreement. By signing the agreement, the requester agrees that the data 1) will be used only for research purposes and will not be used for any product-related effort and 2) will not be shared with a third party. Please contact rf-msa@csail.mit.edu for access.
Code availability
Code that supports the findings of this study is available from the authors upon reasonable request and with a signed code use agreement. By signing the agreement, the requester agrees that the code 1) will be used only for research purposes and will not be used for any product-related effort and 2) will not be shared with a third party. Please contact rf-msa@csail.mit.edu for access.
References
Viswanathan, M. et al. Interventions to improve adherence to self-administered medications for chronic diseases in the United States. Ann. Intern. Med. 157, 785–795 (2012).
Iuga, A. O. & McGuire, M. J. Adherence and health care costs. Risk Manag. Healthc. Policy 7, 35–44 (2014).
Cohen, J. L. et al. Assessing the validity of self-reported medication adherence among inner-city asthmatic adults: the Medication Adherence Report Scale for Asthma. Ann. Allergy Asthma Immunol. 103, 325–331 (2009).
Zullig, L. L. et al. Improving diabetes medication adherence: successful, scalable interventions. Patient Prefer. Adherence 9, 139 (2015).
Lam, W. Y. & Fresco, P. Medication adherence measures: an overview. Biomed. Res. Int. 2015, 217047 (2015).
Cooke, C. E., Lee, H. Y., Tong, Y. P. & Haines, S. T. Persistence with injectable antidiabetic agents in members with type 2 diabetes in a commercial managed care organization. Curr. Med. Res. Opin. 26, 231–238 (2010).
Peyrot, M., Barnett, A. H., Meneghini, L. F. & Schumm-Draeger, P.-M. Insulin adherence behaviours and barriers in the multinational Global Attitudes of Patients and Physicians in Insulin Therapy study. Diabet. Med. 29, 682–689 (2012).
Molimard, M. et al. Chronic obstructive pulmonary disease exacerbation and inhaler device handling: real-life assessment of 2935 patients. Eur. Respir. J. 49, 1601794 (2017).
Selvin, E., Parrinello, C. M., Daya, N. & Bergenstal, R. M. Trends in insulin use and diabetes control in the US: 1988–1994 and 1999–2012. Diabetes Care 39, e33–e35 (2016).
Lavorini, F. et al. Effect of incorrect use of dry powder inhalers on management of patients with asthma and COPD. Respir. Med. 102, 593–604 (2008).
Urquhart, J. Role of patient compliance in clinical pharmacokinetics. Clin. Pharmacokinet. 27, 202–215 (1994).
Samaranayake, N. R. & Cheung, B. M. Y. Medication safety in hospitals: avoiding medication errors in the medication use process. Adv. Pharmacoepidem. Drug Safety 2, 1–7 (2013).
Kelly, P. et al. An ethical framework for automated, wearable cameras in health behavior research. Am. J. Prev. Med. 44, 314–319 (2013).
Adib, F., Kabelac, Z., Katabi, D. & Miller, R. C. 3D tracking via body radio reflections. in 11th USENIX Symposium on Networked Systems Design and Implementation 317–329 (2014).
Adib, F., Mao, H., Kabelac, Z., Katabi, D. & Miller, R. C. Smart homes that monitor breathing and heart rate. in Proc. of the 33rd Annual ACM Conference on Human Factors in Computing Systems 837–846 (2015).
Tian, Y., Lee, G.-H., He, H., Hsu, C.-Y. & Katabi, D. RF-based fall monitoring using convolutional neural networks. in Proc. of the ACM on Interactive, Mobile and Wearable Ubiquitous Technologies 2, 1–24 (2018).
Zhao, M., Yue, S., Katabi, D., Jaakkola, T. S. & Bianchi, M. T. Learning sleep stages from radio signals: a conditional adversarial architecture. in Proc. of the 34th International Conference on Machine Learning 4100–4109 (2017).
National Asthma Council Australia. Inhaler Technique for People with Asthma or COPD. https://www.nationalasthma.org.au/living-with-asthma/resources/health-professionals/information-paper/hp-inhaler-technique-for-people-with-asthma-or-copd (2018).
American Diabetes Association. Insulin administration. Diabetes Care 27, s106–s107 (2004).
Frid, A. H. et al. New insulin delivery recommendations. Mayo Clin. Proc. 91, 1231–1255 (2016).
Truong, T. H., Nguyen, T. T., Armor, B. L. & Farley, J. R. Errors in the administration technique of insulin pen devices: a result of insufficient education. Diabetes Ther. 8, 221–226 (2017).
Spollett, G., Edelman, S. V., Mehner, P., Walter, C. & Penfornis, A. Improvement of insulin injection technique: examination of current issues and recommendations. Diabetes Educ. 42, 379–394 (2016).
Murphy, A. How to help patients optimise their inhaler technique. Evaluation 14, 19 (2020).
Giraud, V., Allaert, F.-A. & Roche, N. Inhaler technique and asthma: feasability and acceptability of training by pharmacists. Respir. Med. 105, 1815–1822 (2011).
Göriş, S., Taşci, S. & Elmali, F. The effects of training on inhaler technique and quality of life in patients with COPD. J. Aerosol Med. Pulm. Drug Deliv. 26, 336–344 (2013).
Lippi, G. & Henry, B. M. Chronic obstructive pulmonary disease is associated with severe coronavirus disease 2019 (COVID-19). Respir. Med. 167, 105941 (2020).
Guo, W. et al. Diabetes is a risk factor for the progression and prognosis of COVID-19. Diabetes Metab. Res. Rev. 31, e3319 (2020).
Kabelac, Z. et al. Passive monitoring at home: a pilot study in Parkinson disease. Digit. Biomark. 3, 22–30 (2019).
Melani, A. S. et al. Inhaler mishandling remains common in real life and is associated with reduced disease control. Respir. Med. 105, 930–938 (2011).
Laube, B. L. et al. What the pulmonary specialist should know about the new inhalation therapies. Eur. Respir. J. 37, 1308–1417 (2011).
Stegemann, S. et al. Adherence measurement systems and technology for medications in older patient populations. Eur. Geriatr. Med. 3, 254–260 (2012).
Lavorini, F., Janson, C., Braido, F., Stratelis, G. & Løkke, A. What to consider before prescribing inhaled medications: a pragmatic approach for evaluating the current inhaler landscape. Ther. Adv. Respir. Dis. 13, 1753466619884532 (2019).
Aldeer, M., Javanmard, M. & Martin, R. P. A review of medication adherence monitoring technologies. Appl. Syst. Innov. 1, 14 (2018).
Trief, P. M., Cibula, D., Rodriguez, E., Akel, B. & Weinstock, R. S. Incorrect insulin administration: a problem that warrants attention. Clin. Diabetes 34, 25–33 (2016).
Spain, C. V., Wright, J. J., Hahn, R. M., Wivel, A. & Martin, A. A. Self-reported barriers to adherence and persistence to treatment with injectable medications for type 2 diabetes. Clin. Ther. 38, 1653–1664 (2016).
Price, D. et al. Inhaler competence in asthma: common errors, barriers to use and recommended solutions. Respir. Med. 107, 37–46 (2013).
Zhao, M. et al. Through-wall human pose estimation using radio signals. in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition 7356–7365 (2018).
Hsu, C.-Y., Hristov, R., Lee, G.-H., Zhao, M. & Katabi, D. Enabling identification and behavioral sensing in homes using radio reflections. in Proc. of the 2019 CHI Conference on Human Factors in Computing Systems 1–13 (2019).
Wang, W., Liu, A. X. & Shahzad, M. Gait recognition using wifi signals. in Proc. of the ACM International Joint Conference on Pervasive and Ubiquitous Computing 363–373 (2016).
Fan, L. et al. Learning longterm representations for person re-identification using radio signals. in Proc. of the IEEE Conference on Computer Vision and Pattern Recognition 10699–10709 (2020).
Ruud, K. W., Rønningen, S. W., Faksvåg, P. K., Ariansen, H. & Hovland, R. Evaluation of a structured pharmacist-led inhalation technique assessment service for patients with asthma and COPD in Norwegian pharmacies. Patient Educ. Couns. 101, 1828–1837 (2018).
Hsu, C.-Y. et al. Extracting gait velocity and stride length from surrounding radio signals. in Proc. of the 2017 CHI Conference on Human Factors in Computing Systems 2116–2126 (2017).
Zhao, M. et al. RF-based 3D skeletons. in Proc. of the 2018 Conference of the ACM Special Interest Group on Data Communication 267–281 (2018).
Zhao, M., Adib, F. & Katabi, D. Emotion recognition using wireless signals. in Proc. of the 22nd Annual International Conference on Mobile Computing and Networking 95–108 (2016).
Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical image segmentation. in Medical Image Computing and Computer-Assisted Intervention 234–241 (2015).
Wu, Y. & He, K. Group normalization. in Proc. of the European Conference on Computer Vision 3–19 (2018).
Clevert, D.-A., Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). in International Conference on Learning Representations (2016).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Graves, A., Fernández, S., Gomez, F. & Schmidhuber, J. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. in Proc. of the 23rd International Conference on Machine Learning 369–376 (2006).
Newcombe, R. G. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat. Med. 17, 857–872 (1998).
Acknowledgements
We thank the editor and the reviewers for their insightful comments and feedback. We are grateful to S. Desselle, L. Celi, L. Xing, X. Li, H. Hsu, J. Stokes, Z. Hussain, U. Munir, D. Vasisht, C.-Y. Hsu, G. Zhang, S. Yue, H. He, L. Fan, Y. Yang, T. Li, Y. Liu, H. Rahul, Z. Kabelac and R. Hristov for their comments on our manuscript. We also thank all the individuals who participated in our study. D.K., M.Z., H.W. and A.R. are funded by Delta Electronics (6934407). K.H. is funded by a Fulbright scholarship.
Author information
Authors and Affiliations
Contributions
M.Z., K.H. and D.K. conceived the study. M.Z., H.W., A.R. and K.H. designed and conducted the experiment. M.Z. processed the data. K.H. trained the participants and provided substantial clinical input on the study. M.Z., H.W. and A.R. developed the technical approach. M.Z. and H.W. conducted the analysis and generated the figures. D.K. supervised the work. All authors contributed to and approved the manuscript.
Corresponding authors
Ethics declarations
Competing interests
D.K. is a co-founder of Emerald Innovations, which uses wireless sensors similar to the one used in the study to analyze physiological signals. The remaining authors declare no competing interests.
Additional information
Peer review information: Javier Carmona was the primary editor on this article and managed its editorial process and peer review in collaboration with the rest of the editorial team.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Supplementary Video
The device setup and an example output from our system
Rights and permissions
About this article
Cite this article
Zhao, M., Hoti, K., Wang, H. et al. Assessment of medication self-administration using artificial intelligence. Nat Med 27, 727–735 (2021). https://doi.org/10.1038/s41591-021-01273-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41591-021-01273-1
This article is cited by
-
An intriguing vision for transatlantic collaborative health data use and artificial intelligence development
npj Digital Medicine (2024)
-
AI-Enhanced Healthcare: Not a new Paradigm for Informed Consent
Journal of Bioethical Inquiry (2024)
-
Deep-learning-enabled brain hemodynamic mapping using resting-state fMRI
npj Digital Medicine (2023)