Magneto-oscillatory localization for small-scale robots

Abstract

Magnetism is widely used for the wireless localization and actuation of robots and devices for medical procedures. However, current static magnetic localization methods suffer from large required magnets and are limited to only five degrees of freedom due to a fundamental constraint of the rotational symmetry around the magnetic axis. We present the small-scale magneto-oscillatory localization (SMOL) method, which is capable of wirelessly localizing a millimeter-scale tracker with full six degrees of freedom in deep biological tissues. The SMOL device uses the temporal oscillation of a mechanically resonant cantilever with a magnetic dipole to break the rotational symmetry, and exploits the frequency-response to achieve a high signal-to-noise ratio with sub-millimeter accuracy over a large distance of up to 12 centimeters and quasi-continuous refresh rates up to 200 Hz. Integration into real-time closed-loop controlled robots and minimally-invasive surgical tools are demonstrated to reveal the vast potential of the SMOL method.

Introduction

Small-scale robots hold enormous potential for minimally-invasive medicine, for example, they may allow targeted drug delivery, in vivo sensing, and new minimally-invasive surgical procedures^1,2. Recently, significant progress in the field has been made to wirelessly power and actuate nano- to millimeter robots in biological environments^3,4. Controlled propulsion has been shown in in vivo or ex vivo experiments, such as microrobots swimming in mouse intestines using magnetic fields⁵ and chemical fields^6,7. We demonstrated that magnetically-actuated helical nanorobots can propel through the vitreous humor of the eye for drug delivery to the retina⁸ and it was shown that enzymatically active micropropellers can move through gastric mucin gels⁹. Millimeter-sized robots were reported to move through various biological tissues for minimally invasive surgery. For example, screw-shaped magnetic robots are able to penetrate muscle tissue^10,11 and to navigate through vascular systems¹². Dragging, tumbling, and other means of actuation have also been reported to actuate small-scale robots for biomedical applications^13,14,15,16. Despite the progress in actuation, a commonly overlooked challenge for small-scale robotics is the real-time localization of the small robots in deep biological tissues. Localization requires the detection of the robot’s position and its orientation, which are both crucial for feedback control of the robot in complex unknown environments inside the body and thus greatly affect the efficacy to perform biomedical tasks.

Localization of small-scale robots poses various technical challenges¹⁷ that traditional medical imaging and device tracking techniques cannot fulfill. First, to continuously control the locomotion, real-time tracking is required, however, established medical imaging techniques, such as magnetic resonance imaging (MRI), have limited refresh rates. Radiation-based imaging methods, such as computer tomography (CT), positron emission tomography (PET), and fluoroscopy, are unsuitable for the continuous tracking of moving robots due to undesired harmful radiation. Magnetic particle imaging (MPI) is a promising new imaging method for magnetic nanoparticles, which has been used for combined tracking and actuation with improved imaging rates^18,19. Second, intracorporeal localization of small-scale robots requires high spatial resolution as well as large tissue penetration depth. Ultrasound (US) imaging was commonly applied as a real-time imaging method for such devices^5,20,21,22. However, intracorporeal ultrasonic waves suffer from multiple reflections and strong scattering that limit the imaging contrast and the localization resolution. Moreover, US imaging is often two-dimensional (2D), making it difficult to localize and track a moving robot in the 3D space. In our previous work, we demonstrated that optical coherence imaging (OCT) can be used to localize nanorobots in the vitreous⁸, however, opacity of most biological tissues poses a common limitation to near-optical wavelengths. Photoacoustic imaging^23,24 and fluorescent imaging²⁵ were also reported to track robots’ motion in vivo, however, their penetration depth is limited. Lastly, as small-scale robots aim for wireless functionality, tethered localization techniques, such as electromagnetic tracking systems with tethered magnetic sensors for endoscopes or catheters^26,27, cannot be applied.

Wireless localization and tracking methods were developed for biomedical devices, such as endoscopic capsules and implantable tissue markers. Electromagnetic (EM) fields were used to track prostate movement with multiple embedded coils to a spatial resolution below 2 mm and a temporal resolution of 10 Hz²⁸, however, each coil is 8 mm long and at least two coils are required for localization. Using a complex chip design, an integrated sensor circuit to measure the 3D magnetic field gradients to localize the device with a sub-millimeter resolution at 10 Hz was reported²⁹. Magnetic fields, which are safe to use in the human body, were generated from embedded permanent magnets and measured by external sensor arrays to localize medical devices^30,31. However, the static permanent magnets need to be relatively large (>6 mm³) in order to generate a measurable field in deep tissues (>5 cm) and are heavily influenced by nearby magnetic surgical tools³². Overall, most mentioned trackers have a large size (at least one dimension surpasses 7 mm) making them unsuitable for the integration on small-scale robots.

Traditional localization method based on single permanent magnets can maximally achieve 5 degrees of freedom (DoF). As the magnetic field of the device exhibits rotational symmetry around the axis of the magnetic moment, it is fundamentally impossible to measure the rotation angle of a robot around this axis. However, the missing sixth DoF for a static magnet can play a crucial role for the feedback control. For instance, magnetic helical robots often have a magnetic dipole moment perpendicular to the propulsion (helical) axis. Without localizing the rotational DoF around the magnetic dipole, the pitch (or yaw) angle of the robot is missing. Currently, their propulsion direction can only be interpolated by relating the current location to a previous location³³ or using additional onboard sensors^34,35. When the initial robot’s propulsion axis is unknown, or when the movement of the robot is disturbed, the sixth DoF is lost, which leads to loss of control over the robot. Another example is the capsule endoscope, where full 6 DoF information is very useful to improve the stitching of endoscopic images and relate them to a global reference frame³⁶. Full 6 DoF localization has recently been achieved using a magneto-mechanical torsional resonator that consists of two opposing magnets³⁷, however, the device does not exhibit a net magnetic moment in the far field and hence cannot be actuated by external magnetic fields.

Here, we demonstrate small-scale magneto-oscillatory localization (SMOL) with 6 DoF for magnetic robots to enable real-time closed-loop control. SMOL combines the resonance properties of EM devices and the miniaturized footprint of a permanent magnet. The tracker consists of a finite magnetic moment attached to a cantilever, which oscillates at a designed frequency near 100 Hz about a rotational axis perpendicular to the magnetic moment to break the rotational symmetry of the permanent magnet (Fig. 1). The magnetic field generated by the 0.8 mm³ oscillating magnet is measured by an array of 10 sensors and fitted to a magnetic field model by an optimization algorithm, so that all 3 translational DoF and 3 rotational DoF of the device are accurately determined. We show that the SMOL tracker can be readily integrated in a miniature robot (Fig. 1A) or onto surgical tools, and the same magnet can be used for closed-loop localization and actuation under different magnetic field excitations in biological environments. The results show that full 6 DoF tracking is achieved with an adjustable spatial and angular resolution from sub-millimeter to sub-100-micrometer for all translational DoF and down to sub-degree for all rotational DoF at distances up to 12 cm. Real-time localization at around 5 Hz is achieved, with the possibility for an increase beyond 50 Hz using the segmentation of a single oscillation. The SMOL method requires simple instrumentation and exploits a unique frequency response of the device to maintain high signal-to-noise (SNR) ratios in a magnetically unshielded environment, thus it will open up promising opportunities to localize small-scale robots or medical tools deep inside the human body for clinical applications.

Fig. 1: Overview of the SMOL method.

A Schematic of a SMOL system with an excitation coil, an embedded SMOL device and a sensing array. B Oscillating cantilever model. The magnetic moment m at position r oscillates in time on a circular path upon excitation by an external B-field B_ext. The deflection angle θ is limited by the housing. C High-speed optical analysis of the magnets center point (orange) and orientation (red) at two times during the oscillation. D Schematic illustration of the excitation and signal phase. During the excitation phase, a current I_coil is applied to the excitation coil. The resulting B-field leads to a continuous increase of the deflection angle θ and to the saturation of the sensor signals B₁ to B_i. After I_coil is shut down, θ slowly decays in an underdamped harmonic oscillation and the sensors measure the signal emitted from the SMOL device. E Raw signal measured at 80 mm distance with a resonance frequency ${f}_{{{{\rm{res}}}}}=103.5$ Hz. F Resulting signal after filtering. G Full oscillation period. The corresponding cantilever deflection is illustrated below. The fully sampled data (orange) is down-sampled to 5 points per half-period (blue) for insertion into the optimization model. The fitting result (red) for the physical model yields an excellent fit with R² = 0.99.

Results

Localization principle

The SMOL method is based on the mechanical resonance of a cantilever structure with an attached finite magnetic moment m, as shown in Fig. 1B. The magnetic moment, i.e. a permanent magnet, acts as both a magnetic actuator to excite the oscillation of the cantilever and as an emitter of a varying magnetic field for sensing. A thorough derivation of the physical model and measured signal is presented in Supplementary Materials. Upon application of an external magnetic field B_ext perpendicular to the magnetic moment direction and cantilever direction, a torque τ (Eq. S1) acts on the magnet, resulting in a deflection of the cantilever by the angle θ up to the maximum angle ${\theta }_{\max }$. Using an alternating current (AC) magnetic field at the resonance frequency ${f}_{{{{\rm{res}}}}}$ of the mechanical structure as excitation, a periodic deflection can be induced which decays as an underdamped harmonic oscillation (Eq. S2-S4). Depending on the rotation and translation of m in time, the magnetic field emitted from the magnet, approximated as ideal magnetic dipole (Eq. S5), produces a non-trivial signal (Eq. S10) of the magnetic flux density B with respect to θ at a nearby magnetic sensor at position r.

The sensed magnetic field from a SMOL device exhibits, most dominantly, the fundamental frequency ${f}_{{{{\rm{res}}}}}$ of the mechanical oscillator but also multiples of this frequency (higher harmonics) due to the highly non-homogeneous shape of the dipole field. The main components of this signal are the first and second harmonics due to the in-plane oscillation of the magnetic moment vector (x-z-plane in Fig. 1B). By recording with multiple sensors simultaneously, i.e. in a sensor array (Fig. 1A), the multitude of different signal shapes can be used to decode the 3D position and 3D orientation by fitting the physical model (Eq. S2 to S8) to recorded data. Effectively, the oscillating magnet breaks the rotational symmetry of a static permanent magnet, resulting in full 6 DoF information of the SMOL device.

Three main components are necessary for a complete SMOL system, as shown in Fig. S1 (details in the “Materials and methods” section). First, an excitation unit, consisting of a set of coils, is driven at the known resonance frequency of the SMOL device. Here, the excitation unit consists of two perpendicular flat coils to generate any planar B-field direction above the coils. Second, a sensing unit in the form of a magnetic sensor array picks up AC magnetic signals of the SMOL device. Here, ten fluxgate magnetometers with sub-nanotesla resolution are arranged in a 15 × 15  cm² plane. Third, the SMOL device itself, shown in Fig. 1A, is the key component and the device to be tracked. Its mechanical resonance frequency is tuneable by change of cantilever properties, such as elasticity or dimensions of the cantilever and the magnetic mass, which is crucial to achieve spectral separation from magnetic noise frequencies. For the following characterization, a steel cantilever (length ~ 3.5 mm, width 0.2 mm, thickness 20 μm) and a N52 NdFeB magnet of dimensions ⌀ 1 mm × 1 mm are used (see also Materials and Methods). In order to avoid rapid damping of the oscillatory motion in direct contact to the environment, an enclosure surrounds the cantilever and magnet. Besides the protective function, it also limits the maximum deflection angle ${\theta }_{\max }$, which is of importance for a precise localization.

Oscillation of a SMOL device

A high-speed oscillation of a SMOL device with ${f}_{{{{\rm{res}}}}}=135$ Hz is presented in Movie S1 and shown at two frames with the highest deflection angles of approximately −20° to +20° in Fig. 1C. The corresponding image analysis is shown in Fig. S2, while a schematic representation of a full time sequence of a single measurement is shown in Fig. 1D. The measurement can be divided into an excitation phase and a signal phase. In the excitation phase, a current I_coil in an unbiased sine-wave form is applied to the excitation coil at the oscillating system’s resonance frequency. Multiple cycles are used to gradually increase the deflection angle θ to the maximum angle. Since the excitation field’s magnitude is beyond the measurement range of the magnetic sensors, the sensors B₁ to B_i saturate periodically during the excitation phase. In the signal phase, the excitation field is shut off and the stored energy in the cantilever is released in an underdamped oscillatory movement. During this period, θ gradually decreases and meanwhile the sensors pick up the magnetic signal.

A recorded raw signal is presented in Fig. 1E. The presence of magnetic noise in combination with a highly complex magnetic signal from the SMOL device lead to the necessity of signal filtering. The full signal filtering process, together with discrete Fourier transformations (DFT), is presented in Fig. S3 and further explained in Supplementary Materials. Using a spatial difference, low-pass filtering and a time derivative, high fidelity reconstruction of the oscillating signal (Fig. 1F) with a 20-fold increase of the signal-to-noise ratio (SNR, with respect to 50 Hz noise) is achieved. Regarding only the first signal periods, the SNR amounts to approximately 61.

All necessary signal features for localization can be extracted by utilizing an integer multiple number N of a half period of a filtered signal, i.e. a half period of the magnets’ oscillation. In Fig. 1G, a full period (N = 2) is shown, which lasts 10 ms and the corresponding magnet deflection is illustrated below. Per half period, the signal is down-sampled to n = 4N + 1 = 5 equidistant discrete points, which allow a detection up to the fourth harmonic for the 8 non-redundant points per full period, according to the Nyquist-Shannon sampling theorem. These data points, for all magnetic sensors, and their respective times are fed into an unweighted Levenberg-Marquardt optimization algorithm³⁸ using Eqs. S2-S8 as a physical model with the 3D position x, y, z and 3D orientation, given in quaternions q₁ to q₄, as fitting parameters. The sum squared error SSE (Eq. S20) between real data and the model fit is minimized during the process and coefficients of determination R² (Eq. S23) of over 0.99 are achieved, indicating a satisfactory fitting results while also validating the physical model. Since the localization result is dependent on good estimations of ${\theta }_{\max }$ and damping coefficient η, a calibration is performed when first using or changing the physical boundaries of the SMOL device by adding both variables as optimization parameters and fixing the z-component to the true value (see Supplementary Materials).

Spatial and angular accuracy

Experiments and numerical simulations were carried out to determine and validate the localization accuracy of the SMOL method. Since a visual ground truth with respect to the measurement plane (x-y-plane) is difficult to establish with an accuracy below 1 mm, differential measurements were performed.

Figure 2A–C show the localization accuracy for two translation directions, x and z, and the rotation axis ϕ_z, which cannot be determined for a static magnet, i.e. the intrinsic z-axis (see Fig. 1B). A summary table of all 6 DoF accuracies is given in Fig. 2D. The localization accuracy is distinguished for two modes: The speed mode uses two half-periods (N = 2) for faster localization rates owing to the low amount of data for optimization, whereas the precision mode (shown as inset) uses twenty half-periods (N = 20) for a more precise localization but lower refresh rates. The complete translation and rotation measurements are presented in Figs. S4 and S5, respectively.

Fig. 2: Characterization of the 6 DoF accuracy and localization rate.

A, B Translational accuracy in x- and z-direction for the speed mode (number of evaluated half periods N = 2) and precision mode (N = 20) as inset, respectively. Experiments (blue) and simulations (red) accurately represent the ground truths, as the data points span on the 45° line in the graphs. C Rotational accuracy around the z-axis, which is the missing rotational DoF for a static magnet (see Fig. 1B). D Summary table for the accuracies in presented ranges. The translation axes correspond to the systems extrinsic axes with respect to the sensor array (Fig. 1A), while the rotation axes are the intrinsic axes of the SMOL device according to Fig. 1B. E Maximum localization distance ${z}_{\max }$ versus magnet size a of an equivalent cube. F Standard deviation σ_x,y,z and localization rate f_loc versus N. G σ_x,y,z versus damping coefficient η (Eq. S4) for both modes with linear trend lines. A larger η indicates a faster decay of the signal. H Superfast localization demonstration with 51.7 Hz refresh rate for a linear stepping motion. The original signal is segmented into N_seg = 4 half period segments.

For the speed mode, over the large investigated in-plane translation ranges in x- and y-direction at 80 mm distance from the sensing array, as well as for translation in z-direction up to 120 mm distance, sub-millimeter accuracy is achieved, indicated by the perfect match of values to the 45° ground truth line. For the precision mode, measured in 200 μm steps, accuracies significantly below 100 μm are achieved, which is less than one-tenth of the magnet’s length at a distance of approximately 100 times of the magnet’s length. In z-direction (Fig. 2B), an increasing scattering of the points with increasing distance is observed, which suggests a decrease in the measurement precision at large depths. The main reason is the magnetic field decay over the cube of the distance according to Eq. S5.

Full 360° rotations are performed for all three rotation axes in speed mode. Accuracies below 4° are achieved for all axis, whereby the intrinsic z- and y-axis perform, on average, slightly better than the x-axis. This discrepancy, which is also prominent in simulation, is likely caused by a minor amplitude decrease from low-pass filtering or direction-dependent noise. In precision mode, measured in increments of 1°, accuracies below 0.5° are achieved for all axes.

Simulation results are also shown in Fig. 2A–D and match very well to the experimental results for all accuracy measurements. It reveals that the numerical model (see Supplementary Materials) clearly represents important features of the SMOL method, as it considers real magnetic noise in all directions. The agreement between simulation and experiments validates the numerical model, and thus the latter offers a valuable way to predict and optimize the performance of SMOL devices. A detailed explanation of accuracy measurements and the underlying statistical analysis (Eqs. S24–S30) can be found in Supplementary Materials.

Using the numerical model, SMOL devices with a scaled cubic magnet of side length a (volume V = a³) are simulated and shown in Fig. 2E, while keeping the resonance frequency, deflection angle, and damping constant. The localization depth where sub-millimeter precision (mean standard deviation in all directions σ_x,y,z < 1 mm) can still be achieved, defined as ${z}_{\max }$, is between 110 mm and 115 mm for the experimentally characterized SMOL device. A clear linear trend between ${z}_{\max }$ and a can be observed which can be ascribed to the scaling behavior of B-field according to Eqs. S5 and S8 where B ~ V/z³ ~ (a/z)³ for a SMOL device in the center of the array. At large distances >100 mm, the sensor array is additionally scaled proportional to the magnet size to compensate the diminishing signal difference of adjacent sensors. An optimized, realistic SMOL device achieves a localization depth of 156 mm (details in Supplementary Materials).

Precision and localization rate

The highly tuneable range of the precision, given as standard deviation, across all directions σ_x,y,z at z = 80 mm, in addition to the localization rate f_loc (including excitation and coil ringdown time), is analyzed for different numbers of evaluated half periods N in Fig. 2F. With increasing N, meaning that a larger portion of the signal is evaluated, σ_x,y,z decreases from 0.5 mm for N = 1 to 0.1 mm for N = 20. The decay can be attributed to an averaging effect and is most significant between N = 1 and N = 6. Random noise or noise at frequencies distant from the resonance frequency of the SMOL signal is simply averaged out, as it does not reoccur with the same magnitude and in phase with the signal over multiple half-periods. In contrast, noise, appearing in very close spectral proximity to the resonance frequency, adds an unknown contribution to the signal in phase with the SMOL signal, creating an uncorrectable error in all half-periods which amounts here to approximately 0.1 mm. While N is increased, the localization rate automatically decreases due to a higher number of data points n. For the presented system, the total number of points, which are fed into the optimization algorithm, for 10 sensor signals with 5 sampling points each for N is 50 (10 × (4N + 1)), whereas for N = 20 it is 810. Correspondingly, with the current system, f_loc of up to 8 Hz can be achieved for the low precision setting (σ_x,y,z < 1 mm) and 2.5 Hz for the high precision setting (σ_x,y,z < 200 μm). Computation times required per localization are currently 40–60 ms for the speed mode and ~360 ms for the precision mode. Overall, the upper limit of the localization rate is mostly defined by the hardware system and could be further reduced using, for example, single-pulse excitation, better current amplifiers and higher computational power. The best trade-off between precision and localization rate is found between N = 4 and 6.

Since SMOL is based on the mechanical response of the cantilever, the precision is influenced by the mechanical coupling between the outside environment to the housing³⁹. In Fig. 2G, a wide damping spectrum is simulated, ranging from no damping (η = 0) to very strong damping (η > 25 s⁻¹) assuming exponential damping (Eq. S4). For reference, experimentally determined damping coefficients for a solid boundary and a viscous liquid boundary (Glycerol) are 1.6 s⁻¹ and 31.7 s⁻¹, respectively, highlighting the extremely diverse environments in which SMOL is generally applicable. A linear increase of σ_x,y,z in dependence of η can be observed for both speed mode and precision mode, the respective slopes of a linear regression fit are 10.3 μms (R² = 0.90) and 6.8 μms (R² = 0.92). This means that the SNR is better for a SMOL device in more rigid environments. Even though the signal is quickly decayed to 5% of the original magnitude after the first 120 ms of a signal for high damping of η = 25 s⁻¹, in precision mode σ_x,y,z is still very high with ~ 200 μm. Beyond this damping coefficient, N has to be reduced to obtain a sufficient SNR for localization.

Superfast localization

Excitation and coil ringdown times, which are contributing adversely to f_loc, are both inherently limiting the maximally achievable rates. Significantly higher f_loc can be obtained by directly segmenting a signal into smaller partitions, each with the same number of half periods N_seg. Reducing N_seg to 1 means that every half period is independently evaluated, resulting in a superfast localization rate limit f_loc,limit of twice the resonance frequency. This means that a SMOL device with ${f}_{{{{\rm{res}}}}}=103.5$ Hz can be temporarily localized with a 207 Hz refresh rate, which is beyond the necessary rate for real-time medical applications. Figure 2H and S6 demonstrate the capabilities of this method to localize a fast linear stepping motion with a speed up to 200 mm/s. High accuracies are already obtained with a refresh rate of 51.7 Hz for a total duration of 2.4 s (Fig. 2H) and the precision is better than 1 mm in the first 1.5 s of the signal (Fig. S6F), and then it requires re-excitation. The re-excitation of the SMOL oscillation takes approximately 80 ms, for both the excitation and the coil ringdown with the current system, which corresponds to the total downtime of the localization. Therefore, a quasi-continuous localization with superfast rates can be achieved. The superfast localization approach pushes the SMOL method to the physical limit and can be used for tracking of very fast movements, given the necessary computation speed and low damping of the oscillation.

Integration of SMOL and actuation in millirobots

The single magnet in the SMOL device can also be used for the actuation of small-scale robots¹⁴. However, since the magnet is fixed to a delicate elastic component, special care has to be taken when combining it with the large forces needed for magnetic actuation. Two kinds of millirobots, as shown in Fig. 3, demonstrate not only the compatibility of SMOL with magnetic actuation but also its versatility of implementation.

Fig. 3: Integration of SMOL and actuation in millirobots.

A R-shaped actuation path of a millirobot in viscous fluid determined by optical tracking (orange) and SMOL tracking (red). The robot is controlled in a closed-loop by a magnetic gradient setup (Fig. S8A and B) and localized using SMOL at an average refresh rate of 3.5 Hz. Note that optical tracking was not used as feedback and is only presented for reference. B S-shaped actuation path of a helical millirobot in viscoelastic gel. Red arrows indicate the main axis of the robot determined by SMOL. An external rotating magnet (see Fig. S8C and D) induces a torque τ along the main axis of the robot due to the perpendicular alignment between the cantilever (green) and the magnet (blue-red).

Figure 3 A presents a magnetic gradient force actuated millirobot being dragged through a viscous environment in an R-shaped path. This SMOL robot has a magnetic moment axis aligned with the cantilever, resulting in a pulling force F upon application of an appropriate magnetic gradient field. The force is applied away from the cantilever to avoid damage from sudden compression. External B-fields are generated using four independently controlled electromagnetic coils (Fig. S8A) for arbitrary in-plane movements. SMOL and actuation are performed sequentially with an average refresh rate of 3.5 Hz, enabling real-time closed-loop control of a wireless magnetic robot without any optical feedback. The optical path aligns very well to the SMOL path, as also shown in Movie S2 and analyzed in Fig. S9, with a tracking error below 0.3 mm.

Figure 3B presents a torque-actuated helical millirobot penetrating a viscoelastic gel in an S-shaped path. During actuation, a magnetic torque around the cantilever axis is applied to the magnet and the cantilever is able to transfer the torsion to the housing, which leads to rotation of the SMOL device. The robot has a helical shape on the surface to couple the rotation to the translation, and thus propels in soft viscoelastic materials^10,21. The actuation of the helical millirobot is performed wirelessly using an external rotating magnetic field generated by a rotating permanent magnet with a steerable rotating axis (Fig. S8C, D). The complete motion is shown in Movie S3. As analyzed in Fig. S9, SMOL results perfectly match to the optical ground truth with an average error below 0.2 mm. Propulsion directions (red arrows) at the respective locations deviate less than 5^∘ from the optically determined direction.

A strong magnetic field, necessary for actuation, disrupts the SMOL method, as it saturates the magnetic sensors and also acts as additional damping torque to force the cantilever’s oscillation to stop. Therefore, the actuation and localization have to be performed alternatingly. Since the torque-based robot (Fig. 3B) utilizes a rotating permanent magnet for actuation, the sample container holding the robot and the gel needs to be transferred between the actuation and the localization setups. For the gradient-based robot (Fig. 3A), the actuation field can be shut off quickly, resulting in high refresh rates for closed-loop control in a single setup. These results show the accurate localization possibility with the SMOL method. Moreover, they also demonstrate the feasibility of using the same magnetic moment on the miniaturized robot for both actuation and localization purposes in various media.

Integration of SMOL for biomedical applications

Accurate navigation of surgical tools in the human body is essential for the success of medical procedures. Low-frequency magnetic fields (<100 kHz) are able to penetrate biological tissues with no noticeable attenuation due to the negligible difference of the magnetic permeability between air and water⁴⁰. Additionally, magnetic fields are well-known for their bio-compatibility even beyond a field-strength of 1 T for MRI machines⁴¹. In comparison, the magnetic field generated by the SMOL device’s magnet is in the micro- to nanotesla range, and the external field for excitation is in the millitesla range.

For the demonstration of surgical tool localization and magnetic tool compatibility, a SMOL device is attached to the front of a flexible endoscope that is manually navigated through a transparent in vitro kidney organ phantom⁴² as a commonly performed procedure in retrograde minimally-invasive intra-renal surgeries⁴³, shown in Fig. 4A, B. The endoscope itself is weakly magnetic and produces a surface magnetic field of maximally 120 μT, which makes it compatible with the sensors at distances larger than 2 cm. The endoscope tip first navigates towards the lower calyx, then towards the upper calyx and is finally retracted from the kidney. Meanwhile, the tip of the tool is tracked with an average 4.2 Hz refresh rate using the attached SMOL device (Movie S4). Since the orientation of the SMOL tracker changes during sharp bending of the endoscope tip (over 180^∘), the excitation field orientation is adapted according to the last known orientation of the device. Localization of the endoscope within the whole kidney is achieved in real-time with full position and rotation information, without the use of radiation or optical methods. A further demonstration of the high tolerance of SMOL to magnetic interference, e.g. by a surgical tool, is shown in Fig. S10, where additionally a direct comparison with static permanent magnet localization is performed.

Fig. 4: Integration of SMOL for biomedical applications.

A Schematic illustration of the urinary tract. B Navigation of a flexible endoscope with an attached SMOL device in an in vitro kidney organ phantom. The localization results of the endoscopic tip by SMOL are shown as red dots, and yellow arrows indicate the movement of the endoscope. An average localization rate of 4.2 Hz was achieved (see Movie S4). C Implantation of a SMOL device into an ex vivo pig brain. D Ultrasound (US) image of the implanted SMOL device. A distinction of the tracker to the background is difficult due to biological inhomogeneity. E The overlay of the SMOL result on the enlarged US image. The position and orientation (red) are precisely determined by SMOL.

Besides the tracking of mobile millirobots and surgical tools, a SMOL device can also be used as a static biomedical marker. Strong mechanical damping behavior is expected to impede mechanical oscillations in soft biological tissues, which creates an additional challenge to obtain a sustained oscillating signal of the SMOL device for an accurate localization. Brain is known as one of the softest tissues in the human body⁴⁴, therefore, this tissue is chosen as a realistic and strict testing environment for the SMOL method. The results of the SMOL method working in an ex vivo porcine brain are shown in Fig. 4C–E. The SMOL device is implanted into the gray matter in the cerebrum (Fig. 4C) and successive US imaging (Fig. 4D) is used to obtain planar information of the location of the device relative to the rigid boundaries of the sample container. As shown in the image, the overall US imaging resolution and contrast are poor due to multiple reflections and scattering of the US beam in inhomogeneous biological tissues. The tracker at a distance of ~40 mm to the US probe is barely distinguishable from the background noise. On the contrary to US imaging, the SMOL method accurately detects the position and orientation of the robot in the brain. When the localization information is overlaid with the US image (Fig. 4E), a very good correlation is found between the two localization methods and the tracker’s major axis (red arrow) aligns very well with the estimated orientation by US imaging. At 45 mm z-distance from the magnetic sensors, the standard deviation of 10 independent measurements is in the sub-millimeter region, which reveals the high usability of the SMOL method in real biological soft tissues.

Discussion

Advantages of the SMOL method

SMOL offers a completely wireless localization method of small-scale devices in magnetically noisy and highly-damped biological environments. It outperforms state-of-the-art magnetic localization techniques in many aspects, and the use of a single magnetic moment and a cantilever for the restoring force brings unique benefits.

First, the small footprint of the SMOL device enables its integration into small-scale robots and minimally-invasive surgical tools, e.g. endoscopes and possibly catheters and needles in the future. It requires no on-board power, which makes it easier to be integrated with wireless medical devices, such as capsule endoscopes and implants.

Second, the unique frequency response of a SMOL device facilitates the isolation from DC and low-frequency magnetic noise, i.e. surrounding magnetic and electronic devices or moving surgical tools. The resonance frequency is readily tunable by the material and the geometry of the cantilever. Precise manufacturing techniques are required for frequency tuning with the environmental noise in consideration.

Third, the high accuracy for all 6 DoF over large distances in addition to its small size are beyond the possibilities of other wireless tracking methods. Especially, most state-of-the-art methods based on static permanent magnets, allowing actuation and 5 DoF localization with the same magnet, do not achieve sub-millimeter accuracy at lower or equal localization depths, and are using multiple orders of magnitude larger magnetic volumes^{30,31,32,33,45,46,47}. Owing to the physical principle behind the evaluation method, the accuracy is also highly tuneable by over one order of magnitude from 0.47 mm/2.14° down to 40 μm/0.15° (see Fig. 2D), allowing a localization accuracy of less than one-tenth of a millimeter at large depths for very demanding localization applications.

Fourth, SMOL is proven to work in various boundary conditions. Although physical boundaries have large impact on the attenuation of the mechanical oscillation, experiments show the principle works with solid boundaries, soft viscoelastic materials, and even liquid boundaries. The stronger coupling to the surrounding environment is also beneficial for material sensing applications³⁹. Moreover, SMOL requires no physical contact of the external device to soft tissues, which will benefit minimally-invasive and robotic surgeries, where a direct contact of the imaging probe to the internal organs is often not possible.

Integration considerations for robotic applications

The movement freedom required for the magnet’s oscillation necessitates careful considerations with respect to the desired application. The helical robot (Fig. 3B) is actuated by converting torque about the propulsion axis into linear translation by the screw-shaped housing. The torque from an external field, however, is only applied to the magnet, which is attached to the delicate cantilever. If the magnet is free to rotate without angular restriction, the strong magnetic torque will keep twisting the cantilever and exceed the strength limit of the cantilever material, leading to permanent plastic deformation or fracture of the beam. Hence, geometric constrains are added inside the housing to transmit the torque by the direct contact to the housing. Furthermore, depending on the angular speed of the external actuation B-field, mechanical instabilities, appearing in the form of path deviations or wobbling, in helical robots can occur⁴⁸. This behavior may be amplified in the SMOL helical robot, since the magnet freely deflects according to the external fields, leading to misalignment of the magnet with the robot’s central axis.

Most non-magnetic localization methods cannot provide more than three spatial coordinates, which is very limiting for robotic control, whereas localization method based on single permanent magnets can maximally achieve 5 DoF. As the magnetic field exhibits rotational symmetry around the axis of the magnetic moment, it is fundamentally impossible to measure the rotation angle of a robot around this axis. Lack of information about this axis can lead to control difficulties, for example when using magnetic helical robots or wireless endoscopy capsules, since the propulsion axis is perpendicular to the magnetic moment vector and therefore unknown. Other means of determining the sixth DoF require knowledge of previous locations³³, which are vulnerable to interruptions or sudden turning, or additional embedded sensors^34,35, which drastically increase the device size. The integration of magneto-oscillatory methods such as SMOL, as shown in Fig. 3, circumvents the need of additional implementations by directly providing all 6 DoF. Besides the need for 6 DoF localization for closed-loop actuation, it can also provide crucial information for the navigation of endoscopes. The tip of an endoscope is equipped with a camera, which provides optical feedback for the surgeon; hence the navigation precision and interpretation of the endoscopic images can be significantly enhanced by full rotation information.

Closed-loop control in real-time is desired for most robotic systems. In Fig. 3A, the capabilities of SMOL to fulfill this demand are shown. Current limitations of the localization rate lie in the saturation of the magnetic sensors during the excitation, meaning that excitation and sensing cannot be performed simultaneously. A 5-cycle excitation for a SMOL device with ${f}_{{{{\rm{res}}}}}=103.5$ Hz, in addition to the coil ringdown, requires around 80 ms. For a single cycle excitation, it could be reduced to below 40 ms. Realistically, by reducing the coil ringdown and computation time, a localization refresh rate above 25 Hz can be expected for the SMOL method, making it very suitable for closed-loop robotic applications. Exploiting the physical principle allows superfast localization rates even beyond 50 Hz, revealing the full potential of the SMOL method for real-time biomedical and robotic applications.

The magnet for the used SMOL devices occupies a volume of 0.8 mm³, and the device as a whole (Figs. 1 and S7) requires ~14 mm³ of space, owing to the volume needed for cantilever deflection. The minimal footprint makes it ideal for a direct integration into existing applications such as endoscopic capsules, surgical tools or as a stand-alone minimally-invasive fiducial marker. The magnetic signal strength, and therefore the localization distance, linearly scales with the magnets size over the distance. Hence, depending on the applications’ needs, the device could be scaled to achieve larger distances or smaller sizes. A shorter cantilever, for example, could allow a larger deflection angle and a stronger signal, and the resulting resonance frequency increase could be compensated by thinning of the beam. In general, up-scaling requires a spread-out sensing array, while down-scaling poses additional challenges to manufacturing which could be overcome with MEMS fabrication methods.

The working volume of the SMOL method is dominated by the excitation coils. Magnetic fields in all directions and at all positions within the desired volume have to be generated for excitation of an arbitrarily oriented SMOL device. Currently, two perpendicular excitation coils are used (Fig. S1A), which can achieve excitation for all in-plane orientations, as demonstrated by the closed-loop controlled robot in Fig. 3A. The excitation coil can be extended to an optimized 3-axis multi-coil for excitation of any orientation of the SMOL device. For extension of the actuation system from 2D to 3D, which has been demonstrated for many small-scale magnetic robots in the past⁴⁹, additional coils above and below the working plane are needed to overcome buoyancy and gravitational forces, and their interference with the sensing array has to be minimized.

In summary, the SMOL method can be readily integrated with minimal volume occupation into many existing biomedical and robotic applications to offer high spatial and angular 6 DoF localization as well as real-time temporal resolution.

Materials and methods

SMOL system components

A wireless SMOL device consists of a housing and a mechanically resonant structure with an attached magnet. The housing was designed and 3D printed to allow a sufficiently large deflection angle for the magnet (⌀1 mm × 1 mm, grade N52). The magnet is attached to the tip of a steel stripe (C1095 spring steel, 20 μm thick, 0.2 mm wide, 3–5 mm long) to form the resonant cantilever structure, which is fixed to the housing. Details on the SMOL device fabrication can be found in Supplementary Materials.

System control and data processing were performed in MATLAB (R2022b, The MathWorks, US) and Python (V3.9.12, Python Software Foundation, US) on a i9-12900K 3.20 GHz processor with 64 GB RAM and an NVIDIA GeForce RTX 3060 GPU. For analog control and analog signal conversion, a data acquisition board (PCIe-6363, NI, US) with an input range of ± 11 V, 16 bit resolution (0.33 mV resolution) and a sampling rate of 50 kS/s was used. The magnetic fields were measured with a 10-channel fluxgate sensor unit (FL1-10-10-AUTO, Stefan Mayer Instruments, Germany) with the individual sensors arranged in a 2D plane with a laser-cut (Beambox Pro, FLUX, US) polymethylmethacrylate (PMMA) plate. The sensor unit was equipped with a manual offset compensation function and a sensitivity level of 1 V/μT, a range of ± 10 μT, a resolution of 0.1 nT and an inherent noise of 20 pT/$\sqrt{{{{\rm{Hz}}}}}$ at 1 Hz. For the excitation of the SMOL device, two perpendicular customized electromagnetic coils (0.56 mm-diameter enameled copper wire, ~150 turns on a 60 mm × 50 mm × 10 mm and ~ 150 turns on a 70 mm × 70 mm × 15 mm 3D printed bobbin) were built to generate a magnetic field above the coil ( ~1 mT at a distance of 30 mm) with full planar field-of-view. They were powered by power amplifiers (TSA 4000, the t.amp, Germany) with a current amplitude of up to 10 A. In order to prevent overheating, a customized water cooling reservoir was 3D printed which was filled with double-distilled water. For fast ringdown times and removal of current-leakage, solid state relays (D2425-10, Sensata-Crydom, US) were used. Between the end of the excitation phase and the start of the evaluation phase, a buffer time of 20 ms was applied to avoid the interference of the magnetic signal by the coil ringdown. Statistical analysis was performed in MATLAB. All experiments were performed at room temperature (~22 °C) in an unshielded environment without any magnetic or electric shielding.

Closed-loop control of the gradient-actuated millirobot

For the gradient-actuated millirobot, a planar array of electromagnetic coils was used (Fig. S8A). Four commercial coils (LSIP-330, Monacor, Germany) were arranged in a square with 80 mm side length, resulting in an effective distance of 84 mm between opposing coils. They were powered by a 4-channel DC power supply (NGP800 with NGP K107 module, Rohde & Schwarz, US), and controlled with four solid state relays (TC-GSR-1-25DD, Tru Components, Germany) in addition to the data acquisition unit. The tank was filled with glycerol (≥99% purity, Honeywell, US). The millirobot partially submerged due to the buoyancy. An additional external coil (EM-6723A, Pasco, US) operated with 400 mV and ~20 mA DC (AFG31022, Tektronix, US) was placed at 150 mm distance to roughly compensate the earth’s magnetic field in the working volume to avoid undesired alignment of the SMOL device with the earth’s magnetic field. A simplified closed-loop control scheme is shown in Fig. S8B and further details on the control as well as the fabrication procedure of SMOL millirobots, the numerical simulations, the accuracy characterization and the biomedical application demonstrations are presented in Supplementary Materials.

Helical millirobot actuation

For the actuation of the helical millirobot, an external rotating magnetic field with an average magnetic field strength of ~100 mT was used. A cubic NdFeB magnet (side length 50.8 mm, N40, Supermagnete, Germany) was mechanically fixed to a stepper motor (23HS30-2804S, Stepperonline, US) with the magnetic axis perpendicular to the rotational axis of the motor. The assembly was placed on a rotational stage (GFV5G50, Orientalmotor, Japan) to steer the rotational axis of the magnet in a 2D plane (see Fig. S8C and D). The propulsion of the millirobot was restricted to a planar motion in the gel in-between a PMMA plate and the bottom of the container with a gap of 6 mm and the propulsion path was pre-defined in the gel for improved steerability. As a viscoelastic testing environment to mimic the mechanical properties of a porcine brain⁵⁰, hydrogel with 3 wt.-% gelatin and 0.2 wt.-% agarose was used. Both components were stirred together in double-distilled water at 80 °C for 30 min, filled into rectangular plastic containers (50 mm × 60 mm × 15 mm) and cooled to 22 °C for at least 4 h before use. The rotation frequency of 0.25 Hz and the direction of the rotating magnetic field were controlled by an Arduino board (Ardunio Uno Rev3 SMD, Arduino), and the direction of the rotational axis was steered manually. The propulsion was paused 8 times for the localization, and in each pause, the sample box (with the robot embedded statically inside) was removed from the actuation setup and put in the localization setup at the same location. A camera system (EOS RP with a RF35mm F1.8 lens at 25 fps, Canon, Japan) was used to image the propulsion from the top view. Eight videos were linked and analyzed by a customized code (MATLAB) to recognize the robot center-point and orientation in each frame and draw the trajectory to overlay on the original videos.