AI co-pilot bronchoscope robot

The unequal distribution of medical resources and scarcity of experienced practitioners confine access to bronchoscopy primarily to well-equipped hospitals in developed regions, contributing to the unavailability of bronchoscopic services in underdeveloped areas. Here, we present an artificial intelligence (AI) co-pilot bronchoscope robot that empowers novice doctors to conduct lung examinations as safely and adeptly as experienced colleagues. The system features a user-friendly, plug-and-play catheter, devised for robot-assisted steering, facilitating access to bronchi beyond the fifth generation in average adult patients. Drawing upon historical bronchoscopic videos and expert imitation, our AI–human shared control algorithm enables novice doctors to achieve safe steering in the lung, mitigating misoperations. Both in vitro and in vivo results underscore that our system equips novice doctors with the skills to perform lung examinations as expertly as seasoned practitioners. This study offers innovative strategies to address the pressing issue of medical resource disparities through AI assistance.

guide seat, driven by two pairs of antagonistic tendons to achieve omnidirectional bending deformation of the distal section.The tendons are divided into four directions along the grooves of the guide seat and connect with the upper magnet holders.The steering control system consists of four linear motors (LA50-021D, Inspire-Robots, Beijing, China) for pulling the tendons and four force sensors (QLA414, FUTEK, California, America) for measuring the driving force.One side of each force sensor is connected to the linear motor through the motor flange, and another side is connected to the lower magnetic holder.Two sets of magnets are respectively installed inside the upper and lower magnet holders.Based on the magnetic adsorption force, the bronchoscope catheter can be quickly installed on the steering control system.The linear motors are installed on the motor fixture which connects with the electric slide (EZSM3E040AZMK, Oriental Motor, Tokyo, Japan) by the mounting shell to achieve the feed movement of the bronchoscope catheter.The electric slide is actuated by the slide driver (AZD-KD, Oriental Motor, Tokyo, Japan), and installed on the robotic arm (UR5, Universal Robots, Odense, Denmark) to achieve the large range of pose adjustment of the bronchoscope robot.
The bronchoscope catheter consists of the high-stiffness proximal section and the lowstiffness distal section.The proximal section uses the braided mesh structure for increased stiffness, while the distal section uses the snake tube made of stainless steel for steering control.
The catheter is covered with the thin thermoplastic urethanes (TPU) layer for waterproofing.
To improve the application range of the bronchoscope robot, a 3.3mm catheter with a 1.2mm working channel and a 2.1mm catheter are designed, enabling access to the ninth deeper generation bronchi for average adult patients.The two catheters are both installed with the micro camera (OCHTA10, OmniVision Technologies Inc., Carolina, America) with a square cross-section of 0.65*0.65mm,and two Led lights with a cross-section of 0.35*0.65mm.The proximal length of the two catheters is 650mm, while the snake bone length of the 3.3mm and 2.1mm catheter are respectively 35mm and 25mm.The distal section of the two catheters can achieve an omnidirectional bending of about 180 degrees for deep lung examination.Supplementary Fig. 1.The CAD model of the AI co-pilot bronchoscope robot system.The system consists of a robotic arm, an electric slide, a steering control system, and plug-and-play bronchoscope catheters.The steering control system is composed of four linear motors to steer the catheter, and four force sensors to measure the actuation force.The steering control system and bronchoscope catheter are connected by magnetic adsorption force for rapid catheter replacement.The electric slide is used to feed the bronchoscope catheter, while the UR robotic arm is used to achieve the large range of pose adjustment of the bronchoscope robot.

Supplementary Note 2. Characteristic analysis of the bronchoscope robot
In order to comprehensively assess the performance of our bronchoscope catheters, a series of characterization experiments were meticulously conducted.Firstly, record the corresponding bending angle of each bronchoscope catheter alongside the actuation force when pulling the tendon.The results are illustrated in Supplementary Fig. 2a and 2b, where the bending angle and actuation force exhibit a linear increase with the increase of actuation displacement.The four tendons of each catheter have almost the same characteristics, but the two catheters have different rate of increase.This is mainly caused by the different mechanical and material parameters, such as the different elastic modulus and distances of the tendons from the centreline.Supplementary Fig. 2c shows the magnetic adsorption force of four groups of magnets employed in the connection between the steering control system and the bronchoscope catheter.The magnets adopted in our system have a diameter of 15mm and a height of 4mm, yielding the actual magnetic adsorption forces ranging between 6.7-7.5N,which demonstrates the force limit capacity of our system.
To reinforce safety, we implement soft limits on both actuation displacement and actuation force, simultaneously.Supplementary Fig. 2d illustrates the limitation factor heatmaps within a defined range of actuation force and displacement.Notably, the limitation factor indicates a safe distance from the limitation threshold.With the increase of actuation displacement and actuation force, the limitation factor becomes larger, meaning approaching the limitation threshold.This limitation factor serves as a crucial component in our control system, used for the adjustment of impedance stiffness of the teleoperator (Touch, 3D Systems, South Carolina, America).The lower the limitation factor, the smaller the impedance stiffness, which means that the operator can control the bronchoscope robot with ease.Supplementary Fig. 2. Characterization experiments of the bronchoscope robot.a, The relationship between the bending angle of distal section and actuation displacement.As the displacement increases, the bending angle increase linearly, and all four tendons of one catheter have almost the same characteristic.b, The relationship between the actuation force and actuation displacement.As the actuation displacement increases, the actuation force increases linearly, and all four tendons of one catheter have almost the same characteristic.c, The magnetic adsorption force of the magnets used in our system.The adsorption force is between 6.7-7.5N,which demonstrates the capability of force limit.d, The limitation factor heatmaps under a certain range of actuation force and actuation displacement.

Supplementary Note 3. Kinematic model and control strategy
Kinematic model: For simplifying the model, the following hypothesis is proposed.(1) The friction between the tendons and the catheter is ignored.(2) The tendons always run parallel to the centroidal axis of the catheter at a fixed distance.The coordinate system of the bronchoscope catheter is established shown in Supplementary Fig. 3a.The reference frame {        } and �        � are respectively established on the base and tip of the proximal section, and the reference frame {        } and {        } are respectively established on the tip of the distal section and the catheter tip.Their Z axes are perpendicular to the cross section of the catheter, whereas the X axes point to the first tendon, and Y axes are determined by the right-hand rule.
The reference frame {        } of the micro camera is parallel to the frame {        }.When the distal section deforms under the actuation of tendons, its configuration parameters can be represented by the bending angle , bending plane angle φ and the length .The transformation relationship between {        } and {        } can be expressed as where Define the actuation state  as follows: where   represents the actuator displacement of the i-th tendon.The relationship between the actuation state and the configuration parameters could be represented as follows.
where  represents the distance between the tendon and the centre axis of the catheter,   is defined as Teleoperation mapping relationship: In this work, the operator uses the teleoperator to control the bronchoscope robot.The teleoperator has six degrees of freedoms, more than 431*348*165mm workspace and two buttons.Under teleoperation control, the motion on the XY plane of the workspace of the teleoperator is used to control the omnidirectional bending of the catheter, while the two buttons are used to control the forward and backward movement of the bronchoscope robot, as depicted in Supplementary Fig. 3b.Define the motion point on the XY plane of the workspace of the teleoperator as (  ,   ), where  max =  is the maximum bending angle (180 degrees) of the distal section.During AI shared control, the XY plane of the workspace of the teleoperator are divided into five regions, corresponding to the discrete control commands (forward, up, down, left, and right).

Control method:
As depicted in Supplementary Fig. 3c, there are two control modes:

Supplementary Note 4. Establishment of virtual environment
The establishment of the virtual bronchoscopy environment is depicted in Supplementary Fig. 5a.The airway model is segmented from the pre-operative thorax CT scans.To create the hollow airway model, suitable for texture mapping and rendering the inner face, the 3D airway mask is dilated in three directions: coronal, axial, and sagittal view.Subsequently, the original airway mask is subtracted from the dilated 3D mask to produce a hollow bronchus mask.This mask is then used for surface reconstruction, generating the hollow airway model.For clarity, the term 'airway model' will henceforth refer to the hollow airway model within this paper.For Sim-style image rendering, a pink texture is applied with an RGB color setting of (206, 108, 131) and metallic and roughness factors for the surface material are set at 0.1 and 0.7, respectively.For Real-style image rendering, which requires a more realistic texture, we utilize a historic clinical bronchoscopy video from pigs unrelated to our study.Six frames with clear visuals are selected, cropped, and replicated to create the realistic texture.In clinical scenarios, it's essential to keep the bronchoscope's head as centred as possible within the airway to prevent damaging or piercing the bronchial wall.As such, we define the safest trajectories as those that follow the airway centrelines, which are extracted from the airway model using VMTK.The trachea entrance denotes the start of the centreline, with the end located at the bronchus terminal.
Each centreline undergoes a smoothing process and uniform sampling to ensure waypoints are equally spaced.
In the process of establishing virtual bronchoscopy environment, the airway segmentation from pre-operative CT volume is a key step for simulating bronchus and extracting centrelines as reference paths.In this study, we utilize the region-growing algorithm to segment airway tree from CT volume, by manually placing a seed within the trachea.The adjacent regions can be automatically annotated as the same label if the Hounsfield Unit (HU) values are similar.In practice, region-growing algorithm can be easily implemented by the Airway Segmentations Module of 3D Slicer.

Simulated bronchoscope robot:
According to the design of real bronchoscope robot, we employ a virtual perspective camera and a spot light to simulate the robot's head, and they share the same position and orientation.We select Pyrender as the physical rendering engine.During data preparation of the training stage, the simulated bronchoscope robot is placed within the virtual environment, where images and depth can be observed by rendering from the airway model.
The pose of simulated robot's head is identical to that of the camera, represented by the rotation and translation.The rotation matrix can be represented by three Euler angles: pitch, yaw and roll, following an "XYZ" sequence.In the camera coordinate system   of the robot, pitch, yaw and roll rotations correspond to movements around the side-to-side (x axis), vertical axes (y axis) and front-to-back (z axis), respectively.As shown in Supplementary Fig. 5b, the roll rotation of robot is fixed due to mechanical constraints in practice, so we set the rate of roll to 0 and use (∆yaw, ∆pitch, 0), i.e. (∆, ∆, 0), to represent the angle rate as a steering action in current   that rotates the robot's head towards the next location.Then a forward action is needed to move the robot to reach that position.For safety and maneuverability, we assign a fixed forward speed to the robot.Clinicians can adjust or halt this speed at any time in practice.
In clinical scenarios, the skillful control of rotation angle rates to evade contact with the bronchial wall requires considerable expertise.Therefore, we address this challenge by introducing an AI-human shared control algorithm in this paper.This algorithm automatically generates safe ∆ and ∆ angles based on human commands, enabling the bronchoscope robot to follow the optimal safe path, i.e. the airway centreline.serving, which measures the distance from image centre to the target position in image coordinate system without the need of 3D position.In practical terms, a mapping from   to   can be formed in a statistical way, which we have performed new experiments for discussion in the following part.

Supplementary
The image error   is calculated by projecting the next direction vector  � ∈ ℝ 3 , predicted by the policy network, into the current image coordinate system of robot.In this study, we assume the policy network is well trained and  � should point to a waypoint lying on the centreline, satisfying where  �����⃑ is projected from  � in image,  �|  is the z-coordinate value of  �,  = �   0 0 0   0 � is the known intrinsic matrix of camera,   and   are intrinsic parameters and satisfy   =   .
Actually,  � is predicted by the policy network  and derived from the steering action  = (, ) = [∆, ∆] .According to the formulation of ground truth steering action in Supplementary Fig. 18c, if the policy network is well trained,  � will direct to a point   on the centreline, and the projected  �����⃑ will direct to the bronchial lumen in image coordinate system.
For simplicity, the rotation can also be parametrized by an angle  = (∆, ∆) about an axis of rotation, as shown in Supplementary Fig. 18a.Thus, we can rewrite Eq. 17 as following If we assume that robot's head is parallel to the reference path (i.e.centreline) and the curve of centreline is small enough, the 3D position error   can be approximated as where   is a fixed length along the centreline from the nearest waypoint to a far waypoint   pointed by  �, as shown in Supplementary Fig. 18a.Thus, an approximate relation between   and   can be obtained as According to Eq. 20, it can be observed that   has positive relation with   .However, in realistic bronchoscopy procedures, the two hypotheses (i.e., robot is parallel to airway centreline, and the centreline has small curve) are not always satisfied.Thus, we calibrate the pixel-to-millimeter conversion ratio ∆ =   /  in a statistical way, generating a look-up figure as a reference which records ∆ of every position in bronchus.
We randomly sample 13620 positions along four reference paths of two airway models in the virtual environment, as shown in Supplementary Fig. 18c and d.For each sample, the position and direction of robot head is randomly posed around the reference path and the image error   is calculated by policy network prediction and camera projection as the above process.The position error   is measured by the Euclidean distance between robot's head and the reference path, which can be easily accessed in virtual environment.Then the pixel-tomillimeter conversion ratio ∆ is calculated for each sample.The statistical results of ∆ are shown in Supplementary Fig. 18b, and the mean results at every position of reference path are overlaid on Phantom 1 and 2, as displayed in Supplementary Fig. 18e, g and Supplementary Fig. 18f, h, demonstrating a decrease of ∆ with the increase of bronchial generation.It's reasonable for these results because with the bronchoscope gradually going deeper, the bronchus becomes narrower and the   becomes smaller, while   is only determined by robot's direction, so that ∆ is smaller.Thus, according to Supplementary Fig. 18b, ∆ is 0.075mm/pixel in in trachea (0th generation) and 0.018mm/pixel in 9th generation of bronchi, thus, 50-pixel image error means approximate 3.75mm and 0.91mm in 0th an 9th generation of bronchi, respectively.By averaging all samples in Supplementary Fig. 18b, the mean ∆ is 0.064mm/pixel.In our in-vivo animal experiment, AI co-pilot group has a mean image error of 11.38 pixels, which reflects a mean 3D positon error of 0.73mm in whole procedures.

Human intervention ratio:
We define human intervention as the number of switching actions of doctor's hand.Specifically, for teleoperation mode without AI assistance, we record the number of time stamps where the tendon displacements of linear motors change from last time stamp as the number of human interventions.For AI shared control mode, we record the number Phantom 1. h, Distribution of pixel-to-millimeter conversion ratio along reference paths in Phantom 2. It's obvious that with the bronchoscope robot reaching deep bronchus, the pixel-tomillimeter conversion ratio becomes smaller, because the diameter of airway tree becomes thinner.
teleoperation control and AI shared control.During teleoperation control, the operator inputs the teleoperation command (motion trajectory of the teleoperator), then it is mapped into the configuration parameters of the bronchoscope catheter by equation (6-7).Furthermore, the configuration parameters are converted into the actuation displacements by equation (4-5).By the low-level controller, the linear motors can pull the tendons to steer the bronchoscope catheter towards the desired target.During AI shared control, the operator inputs the discrete command (forward, up, down, left, or right).Then, it is inputted into the AI shared control policy along with the bronchoscopic image.The policy outputs the configuration parameters of the bronchoscope catheter for the steering control, as mentioned before.Teleoperation experiment: To test the performance of the teleoperation control, we hold the teleoperator moving along an approximate circle.During the process, the linear motors pull or push the tendons to steer the bronchoscope catheter by the proposed control strategy, as depicted in Supplementary Fig.4a.Snapshots were taken of the catheter movement corresponding to the teleoperator's position at eight directions, as depicted in Supplementary Fig.4b.The motion trajectory of the teleoperator in the motion process is depicted in Supplementary Fig.4c, while the actuation displacements of two catheters converted from the trajectory is depicted in Supplementary Fig.4d, e.The experiment fully verifies the performance of teleoperation control of the robot.Supplementary Fig. 3. Kinematic model and control strategy.a, The kinematic schematic diagram of the bronchoscope catheter.  ,   ,   ,   and   respectively represent the origin of the reference frame {        }, �        �, {        }, {        } and {        }.The configuration of the distal section can be represented by the bending angle , bending plane angle , and the length . 1 ,  2 ,  3 and  4 represent the four tendon displacements. is the arc radius.b, The teleoperation workspace of the teleoperator.During teleoperation control, the input command is continuous trajectory.During AI shared control, the input command is discrete directions.c, Control block diagram of the bronchoscope robot.The operator observes the images and inputs the teleoperation command, and then the command is converted to the configuration parameters of the distal section of the catheter.Next, the parameters are converted into actuation displacement by the inverse kinematics to control the steering of the bronchoscope catheter.The online safe assessment module adjusts the impedance stiffness of the teleoperator according to the limitation factor.

Fig. 5 .
Virtual environment establishment and human command generation.a, Establishment of virtual bronchoscopy environment.A per-operational CT is acquired and the airway is segmented to generate an airway model.Centrelines of the airway model are extracted by VMTK tool as the ground truth safe trajectories for training.Next, an airway shell model is generated by dilation and subtraction operation, with pink and realistic texture mapping respectively.The observation of the simulated bronchoscope robot is rendered from the bronchus shell model using Pyrender.In policy training stage, Sim-style bronchoscopic images are used for training and Real-style for testing.For online monitoring of the training process, all models are loaded in PyBullet environment for visualization and interaction.b, Illustration of AEA.The human commands and ground truth steering actions are automatically generated by AEA rather than doctors in the training stage of policy network, alleviating the burden of human annotation.Supplementary Fig. 6.Simulation experiments in virtual bronchoscopy environments.a, Three airway models containing up to 5th generation of bronchi from three patients.The upper models are mapped with pink texture and the lower models are mapped with realistic texture.In our study, Patient 1 and Patient 2 with pink texture are used for training the policy network and the Patient 3 with realistic texture is used for testing.b, Specific trajectory errors and 3D trajectories of 60 paths of different methods tested in patient 3. Colored lines represent predicted trajectories and the black lines represent the reference paths.It's shown that training in virtual environment by using Sim-style images and adding our domain adaptation and randomization method (Sim+A+R) can achieve the highest accuracy and success rate.Clinical-style images are collected from historical real bronchoscopic videos in pigs.b, Qualitive results of image translation.Original AttentionGAN is implemented as the baseline for image translation from source to target domain.The results show that our method can effectively perform image translation while preserving bronchial structures, but the baseline method mistakenly considers the bronchial structure as a part of image style to translate and cannot preserve bronchial structures after image translation.Supplementary Fig. 9. Training of structure-preserving unpaired image translation network.The network consists of a generator, discriminator and a depth estimator.The Simstyle images rendered from airway models with pink texture are collected as the source domain, and their corresponding depths are rendered to provide depth supervision.Unpaired clinical images from historical bronchoscopy videos serve as the target domain, which are easy to access in hospital.In the training stage, Sim-style images are fed into the generator to translate them into paired realistic-style images.Then the discriminator takes both translated realistic-style images and unpaired clinical images as input, forming the adversarial loss for training.Meanwhile, the realistic-style images are fed into the depth estimator for generating estimated depths, which are supervised by the rendered depths corresponding to the input rendered image, ensuring that the 3D structure information of the generated image remains consistent with the original rendered image.The "hospital icon" is designed by Twitter Emoji and is used under the open access CC BY 4.0 license (Creative Commons Attribution 4.0 International License).Supplementary Fig. 15.Actuation information on multiple paths obtained by the paticipents.a, Actuation displacement curves obtained by the expert with teleoperation on two phantoms.b, Actuation force curves obtained by the expert on two phantoms.c, Actuation displacement curves obtained by the attending doctor with AI co-pilot on two phantoms.d, Actuation force curves obtained by the attending doctor with AI co-pilot on two phantoms.e, Distribution of actuation displacements obtained by the expert and the attending doctor with AI co-pilot.The numbers of recorded points are n = 3023, 2787, 2997, 3088, 3086, 2772, 2806 and 2885 for eight independent experiments.f, Distribution of actuation forces obtained by the expert and the attending doctor with AI co-pilot.The numbers of recorded points are n = 3023,2787, 2997, 3088, 3086, 2772, 2806  and 2885 for eight independent experiments.