A localization method for wireless capsule endoscopy using side wall cameras and IMU sensor

Localizing the endoscopy capsule inside gastrointestinal (GI) system provides key information which leads to GI abnormality tracking and precision medical delivery. In this paper, we have proposed a new method to localize the capsule inside human GI track. We propose to equip the capsule with four side wall cameras and an Inertial Measurement Unit (IMU), that consists of 9 Degree-Of-Freedom (DOF) including a gyroscope, an accelerometer and a magnetometer to monitor the capsule’s orientation and direction of travel. The low resolution mono-chromatic cameras, installed along the wide wall, are responsible to measure the actual capsule movement, not the involuntary motion of the small intestine. Finally, a fusion algorithm is used to combine all data to derive the traveled path and plot the trajectory. Compared to other methods, the presented system is resistive to surrounding conditions, such as GI nonhomogeneous structure and involuntary small bowel movements. In addition, it does not require external antenna or arrays. Therefore, GI tracking can be achieved without disturbing patients’ daily activities.

114 mm, respectively. The accuracy is not enough to be considered as a general solution; technical problem such as line-of-sight put these methods into a challenge. On top of that, in practical experiments, the antennas are mounted on patient's body which used as a reference for localization, while the reference body is also in motion.
Some works have focused on other external localization techniques based on magnetic field. The idea is to integrate a small magnet inside the capsule, then an external magnetometer will measure the magnetic field 14 ; finally, methods such as triangulation locate the capsule in 3D space. The main advantage of these approaches over other localization methodologies is that low-frequency magnetic signals can pass through the human tissues without degradation, and it is an advantage over RF approaches which are depending on the RF signal's strength. Another advantage is that, the magnetic sensors do not need the line-of-sight vision to detect the capsule. All in all, the computational complexity and overall accuracy of magnetic-based techniques are comparative to RF localization methods. Taddese et al. 15 make use of magnetic-based localization on their paper and achieved position and orientation accuracies lower than 5 mm and 6°mm, respectively. A recent study by Shao et al. 16 , they have enclosed a magnet with bio-tissue and tried to localize it. They showed that the non-ferromagnetic biotissue has minor effect on the magnetic fields, so positioning accuracy will not be influenced by human tissues. In addition, their proposed method obtains the initial guess of position through the variance-based algorithm which leads to reduce the iterations and achieve higher localization accuracy. Shao reported the positioning error up to 10 mm and average orientation error of 12°mm. External positioning also is facing significant challenges, such as interference between multiple magnetic sources and uncertain initial guess for optimization algorithm which leads to drift in estimating the location. Moreover, the need for fixed position external reference is also limiting the application of magnetic localization methods for clinical and experimental usage.
Internal localization could be performed via image processing techniques. Lee et al. 17 have equipped a traditional endoscopy system with an IMU sensor to get the camera's heading in order to compensate the rotation. Lee's system gives the rotation angles; however, the position is still unknown. Turan et al. 18 have proposed a monocular visual odometry for WCE. Utilizing the advantage of deep Recurrent Convolutional Neural Networks (RCNNs), the system is trained in an end-to-end manner, so there is no need to fine-tune the network parameters. Similar to previous methods, image processing localization algorithms encounter with some critical issues, such as inappropriate accuracy due the lack of reference point inside the body. Considering occasional movements inside the body and limited number of frames captured and transmitted by the WCE, it will increase the positioning error.
Some other types of localization methods are also available, for example, Kalantar-Zadeh et al. 19 developed an electronic capsule, capable of sensing various gases in the gut. This capsule can sense oxygen, hydrogen, and carbon dioxide. They performed a human pilot trial and sensed gas concentration profile with respect to GI organs. Their research showed that, carbon dioxide concentration profile is a promising option to localize the capsule inside the body. In a recent article by Jang et al. 20 , an endoscopy capsule has been developed with two cameras located at capsule's sides and it paves the path for taking pictures of the internal GI's walls. Their system has the ability to localize the capsule inside the GI by RSSI technique. They have claimed an average localization error of 1 cm.
Medical and radiological imaging, such as Magnetic Resonance Imaging (MRI), Computerized Tomography (CT), ultrasound, X-ray, and gamma ray techniques or hybrid methods are also considered for capsule localization. But they are not easily combined with WCE because of the necessity for continuous imaging over all examination process which may long as 8 hours.
Till now several available WCE localization methods have been described. There are a number of factors that need to be considered while performing a comparative assessment among different technologies. The accuracy of the location is important, but not the only index. For example, although radiation-based methods such as MRI and X-ray offer a high level of accuracy, it is not feasible to perform those methods for long time duration, and not desirable for the risk of radiation exposure. Robustness is another factor that indicates whether the method is capable of tracking multiple motions including the most challenging involuntary motion of the small bowel. The intestine has inherent motion inside the body. Hence, the capsule's location is recorded according to the current state of the intestine. After a while, it may take a new orientation. The new orientation of the small bowel and the position of the capsule should be considered at the time of surgery or drug delivery. Another factor like patient's comfort is important as well. Methods that are intrusive and disturb a person's daily activity are not suitable. The need for external reference point may require a box antenna surround the patient's body and limit his/ her activities. The availability or need of hospital facilities is another contributing factor. Some methods require technicians to constantly monitor the patient during examination. All in all, there are considerable tradeoffs between different techniques of WCE localization.
Above all, most localization techniques suffer from non-predictable involuntary movement of the small bowel itself, which disrupts the relative movement between the human body and the sensor array. For example, the body may move according to an external coordinates system, while the GI organ itself may move or rotate within the body coordinates in a different direction. As of today, no method is able to address this problem. In the next section, we propose a new approach for WCE localization that addresses this unique issue and results in a better accuracy.

Methods
Human GI tract is a tubular pathway, using this feature we have developed a device for WCE localization. To shed some light on the method, we started this section by an example. Thinking of a Metro which is travelling in a tunnel, a man in the wagon has a compass in his hand and he is looking toward the tunnel's wall. The compass gives the wagon's heading direction, in addition, that person by looking toward the tunnel's wall knows how far it has traveled. Similarly, we have proposed a system that is composed of an IMU sensors, side wall cameras, www.nature.com/scientificreports/ processing unit and transceiver unit. The orientation of the device comes from an IMU sensor and capsule's displacement derives from side wall cameras. On top of that, a fusion algorithm is introduced to combine the orientation data and the displacement. Finally, the system is able to plot the traveled trajectory in a 3D space.
Capsule prototype. In this section an overview of the proposed device and necessary modules and components is provided. The system block diagram is shown on Fig. 1. The processing unit is a bridge between different parts of the system, and it controls all modules including IMU sensor, side wall cameras and RF transceiver unit. The capsule is equipped with four cameras, one for each side; however, to make it easy for presentation, only two of them is depicted in the schematic. The final prototype is shown in Fig. 2a,b. The capsule size is 3.5 cm × 3.5 cm × 4 cm. Different parts of the capsule prototype are shown in Fig. 2c. The capsule is responsible to capture the data and sends them in a raw format to the data-logger outside of the body; therefore, the capsule does not require any data processing which significantly reduces the design complexity. After receiving the data, a computer-based program is responsible to apply algorithms and calculate the 3D result.
Processing unit. A microcontroller (µC) is responsible to connect all components together to work as a system. Individual sensors are connected to the µC through specific protocols. For this project we have used ATmega32U4 as a processing unit, this µC has built-in I 2 C and Serial Peripheral Interface (SPI) which make it simpler to use for different applications.  The µC reads all sensors then sends the raw data using a wireless link. The capsule reads 2 Bytes from each side wall camera. Considering four side wall cameras, the µC reads 8 Bytes in total. The IMU sensor produces 18 Bytes. Finally, the µC creates a frame of 28 Bytes with all data and necessary headers, then sends it to a datalogger outside of the body. In order to reduce the transmission power, the µC only sends the frame when it detects a motion.
Side wall camera. Visual odometry is a technology that measures the displacement using optical flow of the scene. In principle, a camera captures the pictures and sends the frames out of the body for processing, then an image processing algorithm calculates the displacement. However, adding four cameras on capsule's side and sending frames require huge power resources and transmission band width. While, processing images inside the capsule adds complexity to the design.
Motion measurement camera is an Integrated Circuit (IC) that has a low-resolution camera with on-board Digital Signal Processing (DSP) unit to run the image processing algorithms and communication interface, all on a small silicon dye. Figure 3 illustrates the side wall camera, the actual size of this IC is 1 mm × 2 mm. The light sensitive part is a Charged Coupled Device (CCD) camera that has 18 × 18 pixels sensor array. A chip with part number YS8008B is selected for the design. This IC works with 3 V power supply and it has Inter-Integrated Circuit (I 2 C) interface protocol.
The technology used inside wall camera is similar to the one used in optical mouse today. It consists of three basic components: motion sensor (monochromatic camera), optical lens and lighting source. Figure 4a illustrates the camera and related optical system. The optical system consists of two parts, a prism and a focus lens. The prism guides the light source so that it lights up the camera view then the lens focuses the reflected light from surface to the CCD. The light guider is positioned so that it could receive the light rays from the right side and guides the rays with a correct angle to be emitted to the surface and reflected precisely to the CCD. In addition, a Light Emitting Diode (LED) with red color is selected to have the highest reflection rate in the GI. A magnifier (lens) is placed between the side wall camera and surface. This lens magnifies the surface as shown on Fig. 4b and  www.nature.com/scientificreports/ makes it possible to see several tiny markers. The raw image has only 18 × 18 pixels of a microscopic view of the surface. Hence, the motion sensing does not depend on larger markers on GI's walls, instead it depends on global motion. The processor inside the side wall camera is responsible to measure the global motion vectors. The side wall camera takes pictures every 20 ms and by looking at global motion of pixels frame to frame, it can detect the size and direction of motions. A separate source of light is used for side wall camera to adjust the brightness. In addition, due to the configuration of side wall cameras around the capsule, at least one of them always is attached to the GI's wall. By tuning the correct focal length (h), pixels are less likely to get blurred. However, in such cases, the entire figure also moves, and processor still is be able to derive the motions. Figure 4b shows how the motion estimation algorithm is working. h is the fixed distance between the camera and intestine's wall. θ is the angular position of each pixel, and − → D is a vector that shows the displacement of individual pixels compared to the previous frame. The displacement comes from four cameras which are located at the capsule's side wall. As a rule of thumb at least one camera should stick to the intestine's wall. The camera takes pictures every 10 ms. Then, pictures are compared and distinct points at each frame will be selected. Then, the algorithm looks for the same point at adjacent frames. The algorithm is able to calculate motion vectors for individual pixels ( Inertial measurement unit. An Inertial Measurement Unit, commonly known as an IMU, is an electronic device that measures and reports orientation, velocity, and gravitational forces using accelerometers and gyroscopes and often magnetometers. For this project we have used an IMU sensor MPU9250 with size of less than 5 mm × 5 mm. Gyroscope measures angular velocity, in other words gyroscope reports how fast the device is spinning about an axis. Rotation about different axes is illustrated in Fig. 5, which named as roll, pitch and yaw. The gyroscope gives the rate of change of the angular position over time with a unit of deg s , according to Eq. (1).
In order to derive angular position, we integrate the angular velocity in a period of time, it can be mathematically shown as Eq. (2) where, T s is the sampling time and it represents the time interval between samples. The gyroscope data is reliable only on the short term, as it starts to drift on the long term. Hence, accelerometer data is used to compensate the gyroscope data.
The accelerometer measures all forces that are working on the object and sends them as A x , A y and A z . To obtain the angular position with the accelerometer, we are going to determine the position of the earth gravity vector which is always visible on the accelerometer. This can be done by using an atan function. In addition, the yaw derived from magnetometer; so, a fusing algorithm is required to combine the sensors all together.
IMU sensor and side wall cameras have different coordinate systems. The IMU's z axis is placed along the front to back of the capsule as depicted on Fig. 5. The heading vector ( − → V heading ) shows the capsule's motion direction (IMU's z axis) in a 3D space.
− → V heading is defined based on the earth gravity and earth north magnetic pole. The IMU sensor must be calibrated before use to compensate the surrounding conditions and noises. In order to produce an accurate orientation estimation, the magnetometer should see the earth's magnetic field, and any ignore conditions that affect the earth's magnetic properties. The magnetic measurements will be subjected to two types of distortion-hard iron and soft iron distortions. Objects that produce a magnetic field cause the hard iron distortion, which is a permanent bias in magnetometer output. Soft iron distortion, on the other hand,  Sensor fusion algorithm. Generally, in order to get 3D orientation of the capsule, we have to fuse all three sensors of accelerometer, gyroscope and magnetometer. Kalman filters have been demonstrating its usefulness in various applications, it not only works well in practice, but is theoretically attractive because it can be shown that of all possible filters, it is the one that minimizes the variance of the estimation error. Kalman filtering is an algorithm that provides estimates of some unknown variables given the measurements observed over time. Figure 6a demonstrates the IMU sensor fusing schematic as described by Roetenberg et al. 21 . The accelerometer is designed based on a MEMS technology and suffers from noises on its output. Hence, Filter_1 is used to filter the noise. However, Kalman Filter_2 is part of the fusion algorithm and fuses the IMU data to find the orientation of the capsule. Figure 6b shows the 3D orientation result. The output provides an accurate orientation estimation. The next step toward capsule localization is fusing the orientation with displacement data. The capsule's orientation is derived via IMU fusion algorithm; and the heading vector can be calculated using roll, pitch and  where, ψ is yaw, ϕ pitch and θ roll angles. The X, Y and Z are the unit vectors in Cartesian coordinate system. Side wall camera sensor provides the displacement along the heading vector, then multiplying the displacement to heading vector represents the 3D projection of the capsule's movement. This is shown in Eq. (5) where, − → D shows the 3D displacement vector, and D mx is the X component of the − → D m . Finally, to get the 3D position, we have used the following equation: where, P t+1 is the new position and P t is the previous position. Thus, we are able to calculate the new position of the capsule from the previous position. Finally, a 3D trajectory is generated.
Since the capsule is confined in a tubular shape track, like GI tract, peristalsis motion pushes the capsule through the cylindric structure and it moves only along − → V heading (IMU's z axis, and not along x or y direction). Capsule can move either forward or backward (± IMU's z axis) depending on how it is entered into the small bowel. As shown in Fig. 6b the red vector defines the − → V heading , which pointing out the direction of capsule. The algorithm is designed in a way that other motions except − → V heading will be ignored, as a result, intestinal motions inside body will not interfere the capsule tracking. An overview of heading vector and fusion algorithm is depicted on Fig. 6c.

Results and discussion
Surface pattern. In order to evaluate the motion sensor's performance, basic patterns were introduced as depicted in Figure 7a to understand the preliminary movement of the device against surfaces with various patterns. Patterns are expanded on horizontal columns and printed on white paper. We placed the wall camera on top of that line and made multiple movements. The camera sensors were able to capture the dx displacement. The experiment has two outcomes. First of all, YS8008B could measure the displacement in all patterns except two of the parallel patterns. Secondly, the sensor's sensitivity and lens' focal length are important properties for correctly capturing and analyzing the surface. We expanded the experiments to WCE like patterns as shown in Fig. 7b. In this way, we make sure that wall camera will work on small intestine's texture, as well.
The next objective is see if the camera can track the motion on a soft surface (like animal skin or meat). The surface's material defines the optical reflection properties of the surface, several surface materials, such as porcine intestine, grounded meat and human skin, are selected. By considering a curved surface, side wall camera works as long as the top of the surface sticks to the motion sensor's focal length. Elastic properties of the intestine will ensure that the surface is kept at the right distance from the lens. The initial experiment shows that the selected side wall camera is able to measure the traveled distance on soft surfaces like porcine intestine.
Test setup. In order to have a precise ground truth for verification, a robotic arm with five axis of freedom is used in all our experiments that holds the capsule prototype (Fig. 8a). The device sends raw data to a computer which then stored in an excel format for further processing.
The following types of motions were simulated using the setup to evaluate the performance of the proposed prototype and fusion method.
1D tracking. This experiment is designed to evaluate the accuracy of the side wall camera for distance measurements. A robotic arm moves the capsule toward the x direction and side wall camera measures the distances. Figure 9 presents the 1D tracking data, in which Fig. 9a is the raw data from side wall camera. The slop of traveled distance plot indicates the velocity, Fig. 9b is the IMU data and Fig. 9c is the output of our fusion algorithm (estimated trajectory). Figure 9b shows minimal changes in Gyroscope and Magnetometer outputs which indicates that moving the capsule in a line does not include rotation. Figure 10 illustrates another experiment to evaluate the motion sensors' accuracy. In this experiment, the capsule travels three straight line path with size of 15, 30 and 60 cm. Each experiment repeated three times. Figure 10a shows the experiment results and Fig. 10b illustrates the error in estimating the distance. It is shown that the maximum error of ± 3.71% can be achieved in 60 cm path.
Leap and peristalsis motions. Leap and peristalsis motions are two common motions inside the GI system. Leap motion consists of series of fast and small movements. The side wall camera measures these motions. Hence, the frame rate plays a key role. The YS8008B measures the displacement between two sampling time (T s ) then sends the information to the host, hence, T s limits the maximum speeds. Shorter T s tends to overflow the buffer, so, the error rises in displacement measurement, on the other side, longer T s results higher drift error. This experiment is performed to understand whether the side wall camera can detect surge motions. Four different speeds are evaluated including 0.5 cm/s, 2 cm/s, 5 cm/s, and 7 cm/s. As shown on Fig. 11, the side wall camera is able to successfully track velocity up to 7 cm/s. Note that the typical speed of capsule inside the small bowel is 2-5 cm/s 22 .  www.nature.com/scientificreports/ In another experiment, the peristalsis motion is investigated. Peristalsis motion is an involuntary contraction and relaxation of muscles which leads to push food ahead of the wave. Muscle's contraction behind the food to keep it from moving backward, then longitudinal contraction to push it forward. Commonly, peristaltic waves exist in the small intestine at irregular intervals and travel for different distances, some ways travel only a few inches, others travel longer. Tracking an object which is under peristalsis motion could be difficult, due to the fact that the object might go forward and slightly backward.
In this experiment, the robotic arm moves forward and backward in x direction to simulate the GI's peristalsis motion. Figure 12 illustrates a peristalsis motion. In Fig. 12c, the 3D trajectory, fused by our system is shown for the forward and backward motions. The results show a relationship between the direction of travel and acceleration, as shown on Fig. 12b. The blue curve and yellow curve show accelerations across x and z axis, respectively. According to the plots, acceleration variation is reflected only on heading vector which is the IMU's z axis.
The peristalsis motion's sequence includes.    Fig. 12. The size of acceleration depends on the rate of velocity changes. Figure 13a shows how peristalsis motion is working, due to the contraction of muscles, we could ensure that side wall cameras will stick to the GI wall. An experiment has performed in Two Dimensional (2D) tracking to monitor the side wall camera behaviors' in such movement, as well. Figure 13b illustrates the effect of peristalsis motion in the experiment. The capsule is traveled a 15 cm path and during the path, it moves forward and backward several times. The results show that under this condition, the error in estimated distance is around 7.4%.

Involuntary motion.
Involuntary motion is the movement of small bowel inside the body coordinate. At this type of motion, capsule has a stationary location compared to the global GI system. However, its relative position within body coordinate will change. To simulate the involuntary motion, the capsule is attached to the intestine and placed on a tray, and a Robotic Arm moves the tray.
During the involuntary motion, side wall camera will not detect any motion (Fig. 14a). Meanwhile, the simulated body movements are captured by the IMU sensor (Fig. 14b). Since, during involuntary motion, the capsule is stationary, but the small bowel is moving, the 3D trajectory is drawn as a single point showing no motion (Fig. 14c). Therefore, our method is able to detect the involuntary motion successfully.

3D tracking.
A robotic arm moves the capsule based on a predefined trajectory (ground truth). Several tracks are examined, but only three of them are shown on Fig. 15. Then, the proposed algorithm fuses information from side wall camera and IMU sensor to estimate the 3D trajectory.
For the sake of explanation, the recorded data from trajectory of Fig. 15a will be described. As illustrated on Fig. 16, the trajectory is divided into three sections, (1) moving with constant speed at V heading(1) , (2) 90° rotation, (3) moving with constant speed at V heading (2) . As shown on Fig. 16a, during (1) and (3), side wall camera detects motions, but during (2) it does not show any motion, instead, IMU data shows rotation during this time.
In Fig. 17 shows results of moving the capsule on different trajectories simulating random 3D motions inside the GI tract. At each experiment, three plot categories are reported, (a) Cumulative displacement data from side wall camera which denoted as traveled distance, (b) Raw data from IMU sensor, and (c) The 3D traveled     www.nature.com/scientificreports/ shape and size of the trajectory matched with the reference track. One point to note is that the side wall cameras works best when the sensor surface touches the GI wall, like small bowel. In some early sections of the GI tract like esophagus and stomach, the side wall cameras may result in some errors. An article by Kalantar-Zadeh et al. 23 , analyzed the gas mixtures of different organs in GI, and showed that gas sensing could be a promising method to consider in the future. So, integrating an Oxygen molecule (O 2 ) gas sensor to the capsule could make the current capsule prototype a full-length tracker of the entire GI tract, something that can be explored in the future.
Validation experiment. The aim of this experiment is to examine the feasibility of tracking inside the intestine. Figure 18 shows the test setup for validation experiments. The capsule was manually inserted into the porcine intestine and pushed through it. Since it is an in-vitro experiment, the ground truth is measured by hand.
As depicted on Fig. 19, the proposed capsule and algorithm is able to perform localization inside in-vitro intestine.
The entire raw data for experiments can be found as Supplementary (see "Supplementary Information").

Power measurement.
Due to the small size of capsule, there is limited space remained for batteries. The battery's capacity is limited, and it must power up the device throughout the process which takes around 8 h. At this section, the power consumption of the capsule is investigated. Table 1 presents the operating voltage and current in active mode for the device at 3.3 V. In order to work properly for 8 h, the capsule needs battery with 94 mAh capacity. However, by integrating all the required components in an integrated chip (IC) or a system-on-a-chip (SoC) will lower the power consumption significantly than individual components. Table 2 provides a list for several intra-body capsule localization methods and compares them based accuracy (how much the estimated measurement is close to the ground truth), extra hardware or weight inside capsule, patient's comfort (i.e., patients' mobility during the period of diagnosis), static reference point (patient's outside movement vs capsule's inside movement), interference of transmitted signal with other sources, involuntary GI motion, and prototype validation (in vitro or in vivo).
The upside of RF and magnetic-based localization is high accuracy, and they provide a global positioning information. However, these methods require a static reference antenna surround the patient body such as a box   www.nature.com/scientificreports/ which interrupt person's daily activities and suffer from external interferences. Radiation and ultrasound-based methods are not feasible as they require hospital facilities and expose patients to unwanted radiation. Imagebased tracking methods alone are not reliable for complete localization, due to the lack of fixed marker inside GI and image blurriness. Furthermore, all the above methods fail to consider involuntary motion of small bowel which leads to errors in localization.
The proposed method has several significant advantages. Hospital facility and technician are not required in this method, and it does not confine patient's movement. Most importantly, both patient's external movement and the small bowel's involuntary movement will not interfere with the capsule's internal movement, and the actual displacement of the capsule can be projected. This is because the involuntary motion is rejected by the side wall cameras which measure the displacement while the capsule is moving inside the intestine and eliminate the GI motion inside the body.
As described on introduction, accuracy alone is not a reliable factor for comparing different methods. Instead, there are several factors contributes for selecting a method, such as considering GI involuntary motion, patient's comfort, need for static external reference point, hospital facility and etc. Finally, the proposed method is experimentally validated in-vitro using multiple motions. However, there are still rooms for improvement and future research is directed towards further improving the WCE localization by improving the sensors and integrating more sensors like, pH and gas sensors for sections like stomach.

Conclusion
In this paper we have developed a new method for WCE localization by incorporating an IMU sensor and four side wall cameras. The IMU sensor has gyroscope, accelerometer and magnetometer. The data from IMU sensors are fused together with the aim of fusion algorithm to estimate the orientation of the capsule. The side wall cameras are similar to an optical mouse sensor which has 18 × 18 pixels CCD and image processing part in a same chipset. By utilizing a four-camera topology ensures that at least one of them will be in constant touch of the GI wall, and measure the true displacement. Finally, the fusion algorithm combines both orientation and displacement to generate the 3D trajectory. The proposed device has no external reference point. Therefore, comparing with other capsule localization methods this system has many robust features including no interference due to patient's external voluntary movement and small bowel's involuntary internal movements.