In-sensor human gait analysis with machine learning in a wearable microfabricated accelerometer

In-sensor computing could become a fundamentally new approach to the deployment of machine learning in small devices that must operate securely with limited energy resources, such as wearable medical devices and devices for the Internet of Things. Progress in this field has been slowed by the difficulty to find appropriate computing devices that operate using physical degrees of freedom that can be coupled directly to degrees of freedom that perform sensing. Here we leverage reservoir computing as a natural framework to do machine learning with the degrees of freedom of a physical system, to show that a micro-electromechanical system can implement computing and the sensing of accelerations by coupling the displacement of suspended microstructures. We present a complete wearable system that can be attached to the foot to identify the gait patterns of human subjects in real-time. The computing efficiency and the power consumption of this in-sensor computing system is then compared to a conventional system with a separate sensor and digital computer. For similar computing capabilities, a much better power efficiency can be expected for the highly-integrated in-sensor computing devices, thus providing a path for the ubiquitous deployment of machine learning in edge computing devices.

could be observed.
For driving voltage amplitudes over 110 V (outside of the range displayed in figures 2a and 2b), there is a drive frequency within the resonance peak of the clamped beam where the inertial mass, which is polarized at the driving voltage, electrostatically pulls in, in plane and towards the beam, until it rests on the stoppers attached to the anchors.This is due to a constant attractive term (in the parallel plates approximation) of  0  2 0 /4 2 in the electrostatic force which is proportional to the square of the driving voltage amplitude, that overcomes the mechanical restoring force.With a capacitor surface  = 130 m × 50 m, a driving voltage amplitude  0 = 110 V, a fixed gap  = 8 m and  0 the vacuum permittivity, the calculated constant electrostatic force is 2.7 N.With the inertial mass collapsed on its stoppers after pull-in, the gap is reduced to 3 m and the electrostatic force is 19.3 N, larger than the calculated restoring force of 17 N (using an inertial mass spring constant of 3.4 N/m).By reducing the air gap between the beam and inertial mass (which acts as the driving electrode), this pull-in reduces the quality factor of the beam due to increased squeeze-film damping, and sets the upper limit of possible driving voltages since a pulled-in inertial mass cannot be used to sense accelerations.
The acceleration sensitivity of the inertial mass (fig.2d) was characterized by fixing the beam driving voltage amplitude and frequency at 60 V, 246.5 kHz, and sweeping an acceleration signal amplitude and frequency.For this purpose, the MEMS device was installed on an electrodynamic shaker (SinoCeramics JZK-2) powered by a The Modal Shop 2100E21 amplifier and controlled by a Stanford Research Systems SIM960 PID with its input signal set by the National Instruments PCIe 6374 DAQ, using an Analog Devices ADXL1002z accelerometer board installed on the shaker table as a reference.The PID target for these acceleration sweeps consisted of successive Tukey windowed sinusoids of increasing frequency, from 40 Hz to 1.1 kHz, in 400 frequency steps distributed evenly on a logarithmic scale.
As a result of the sinusoidal displacement of the inertial mass, the oscillation amplitude of the beam was modulated at the same frequency.This latter oscillation amplitude, sampled by the DAQ at 20 kS/s, was then measured by averaging the magnitude of its Hilbert transform over all samples contained in the flat section of the Tukey window (taper parameter  = 0.1).This windowing procedure was used to mitigate spurious high frequency signals which were otherwise introduced at the beginning of the acceleration signals, when the acceleration frequency was changed.Since the acceleration response of our electrodynamic shaker system was not perfectly flat and presented peaks between 300 and 500 Hz, the measured MEMS response was normalized, at each measurement frequency, by the reference acceleration signal amplitude (also obtained through the Hilbert transform) produced by the ADXL 1002z accelerometer and measured by the DAQ, to yield the sensitivity in units of volts of beam oscillation signal per g of applied acceleration.While it helped smooth the curves, this normalization was not perfect as the reference accelerometer and our MEMS device were not mounted at exactly the same point on the shaker table, such that there could be small relative displacements between the two (e.g., due to vibration modes of the shaker table or imperfect mounting on the latter).
The phase of the inertial mass oscillations relative to that of the acceleration signal (fig.2d, red) was measured by converting the lag (index of the maximum of the cross-correlation between the reference acceleration and the beam oscillation envelope signal) into a phase shift using the nominal acceleration signal frequency and the sample rate.With the resistors of the wheatstone bridge adjusted to correct for the imbalance between the two piezoresistive strain gauge connections, which ensured proper cancellation of the drive signal feedthrough at the differential amplifier stage (diff.amp in Supplementary Figure 2), the instantaneous oscillation signal of the beam was bandpass filtered in a 160 kHz band around its natural frequency before being demodulated by a diode envelope detector (demod in Supplementary Figure 2), amplified and vertically shifted in order to be sampled by the analog-to-digital converter (ADC) of the microcontroller.
The ATSAM4S microcontroller was used to signal the detected walking pattern to the user by enabling a light-emitting diode (LED).It was also used to log and communicate data wirelessly through an external Feather M0 module interfaced by a serial peripheral interface (SPI) using the microcontroller programmable input/output (PIO), although this feature could be disabled to preserve battery life once the system had been trained.Finally, the microcontroller implemented the output layer of the reservoir computer and its feedback loop.The delayed feedback signal was obtained by delaying the output of the leaky integrator (described in section Training the MEMS system) by 100 samples and scaling it by the feedback gain  (see Supplementary Table 1).This signal was superposed to the random binary mask signal and the result was output by the microcontroller onboard digital-to-analog converter (DAC) at 14285 Hz.
A voltage-controlled oscillator (VCO) generated a 50% duty square voltage signal at a frequency chosen between 200 kHz and 300 kHz by the microcontroller.This signal was bandpass filtered in a bandwidth of 100 kHz around 250 kHz and was multiplied by the vertically shifted feedback and mask signal, which yielded the amplitude-modulated beam drive signal ready to be amplified by the high-voltage module, thus closing the in-sensor computing loop.
with the hardware limitations and the timescales of the gait classification task, and from experience with other biomechanical tasks.The other hyperparameters were adjusted with one subject (who was not part of the dataset used in this study) walking on the treadmill at 0.54 m/s, using a bootstrapping approach to estimate the expected value and standard deviation of the mean ROC AUC for each hyperparameter set with a relatively small amount of data.
A segmentation of the recordings by gait cycle was required for the bootstrapping tests, in order to resample complete gait cycles.This segmentation was performed based on the Teager-Kaiser energy operator (TKEO), 3 which outputs the energy signal of the acceleration recordings.The convolution of the low pass filtered TKEO energy signal with a 50 points ramp yielded a signal which peaked at the beginning of the swing phase of each gait cycle, when toes lifted from the ground.This peak, detected with a peak finder, was used to mark the start of each gait cycle.Segmented cycles with a duration under or over 2 standard deviations of the mean cycle duration were discarded.
For each investigated hyperparameter set, the subject walked in the N, TO and TL patterns for 40 cycles each.From the valid segments of the 30 first cycles, 30 cycles were resampled with replacement and used as training data, while 10 cycles were resampled with replacement from the last 10 cycles and used as validation data.This procedure was repeated 150 times in order to estimate the mean and variance of the ROC AUC for a given hyperparameter set.This whole procedure was repeated for 11 random hyperparameter sets which explored different random values of the drive voltage levels  1 and  2 , feedback gain  and leaking rate .The beam driving signal frequency was fixed at 249.4 kHz based on the beam characterization curves.This frequency was chosen to be within the broad resonance peak seen in fig.2a and to be above the linear resonant frequency (frequency of maximum oscillation amplitude at the lowest forcing amplitude) of 492.5 kHz, to ensure the existence of a driving voltage around which the amplitude response (fig.2b) was sufficiently nonlinear but without hysteresis.Due to its Duffing nonlinearity and the resulting multi-stability of its dynamics, the oscillation amplitude of our MEMS beam can exhibit jumps (up and down) and hysteresis. 2 For the results presented in this paper, we chose a combination of drive amplitude and drive frequency to operate the MEMS beam in a nonlinear regime without hysteresis.The ESN demonstrated in this paper also used an hyperbolic tangent activation function, that was nonlinear and without hysteresis.The beam driving signal frequency was not systematically optimized, as the aforementioned choice gave satisfactory results,

Supplementary Figure 1 |
Power spectral density of the acceleration signals sampled by the ADXL326 reference accelerometer, averaged over all four walking patterns and all participants, for a single walking speed of 0.63 m/s.double-tuned transformer amplifier module composed of two impedance-adapted series transformers (TX) driven by the two phases of a differential output preamplifier (preamp).The high-voltage amplifier could output signals up to 120 V in amplitude at 200 kHz to 300 kHz, which allowed the driving of the doubly clamped silicon beam into its nonlinear regime.