The emerging capability to 3D print a diverse palette of functional inks will enable the mass democratization of patient-specific wearable devices and smart biomedical implants for applications such as health monitoring and regenerative biomedicines. These personalized wearables could be fabricated via ex situ printing, which involves first printing a design on a planar substrate and then deploying it to the target surface. However, this can result in a geometrically and dynamically mismatched interface between printed materials and target surfaces. In situ printing provides a potential remedy by directly printing 3D constructs on the target surfaces. This new manufacturing procedure requires the assistance of artificial intelligence (AI) to sense, adapt and predict the state of the printing environment, such as a dynamically morphing organ. In this Review, we discuss electronic and biological inks for in situ 3D printing, AI-empowered 3D-printing approaches with open-loop, closed-loop and predictive control, and recent developments in surgical robotics and AI that could be integrated in future 3D-printing approaches. We anticipate that this convergence of AI, 3D printing, functional materials and personalized biomedical devices will lead to a compelling future for smart manufacturing.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Rapid prototyping of high-resolution large format microfluidic device through maskless image guided in-situ photopolymerization
Nature Communications Open Access 27 July 2023
Nature Communications Open Access 16 January 2023
Systematic review on the application of 3D-bioprinting technology in orthoregeneration: current achievements and open challenges
Journal of Experimental Orthopaedics Open Access 19 September 2022
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
Valentine, A. D. et al. Hybrid 3D printing of soft electronics. Adv. Mater. 29, 1703817 (2017).
Lu, B., Lan, H. & Liu, H. Additive manufacturing frontier: 3D printing electronics. Opto-Electron. Adv. 1, 170004 (2018).
Guo, S.-Z., Qiu, K., Meng, F., Park, S. H. & McAlpine, M. C. 3D printed stretchable tactile sensors. Adv. Mater. 29, 1701218 (2017).
Kong, Y. L. et al. 3D printed quantum dot light-emitting diodes. Nano Lett. 14, 7017–7023 (2014).
Qiu, K. et al. 3D printed organ models with physical properties of tissue and integrated sensors. Adv. Mater. Technol. 3, 1700235 (2018).
Joung, D. et al. 3D printed stem-cell derived neural progenitors generate spinal cord scaffolds. Adv. Funct. Mater. 28, 1801850 (2018).
Kupfer, M. E. et al. In situ expansion, differentiation and electromechanical coupling of human cardiac muscle in a 3D bioprinted, chambered organoid. Circ. Res. 127, 207–224 (2020).
Mannoor, M. S. et al. 3D printed bionic ears. Nano Lett. 13, 2634–2639 (2013).
Grigoryan, B. et al. Multivascular networks and functional intravascular topologies within biocompatible hydrogels. Science 364, 458–464 (2019).
Skylar-Scott, M. A. et al. Biomanufacturing of organ-specific tissues with high cellular density and embedded vascular channels. Sci. Adv. 5, eaaw2459 (2019).
Park, S. H. et al. 3D printed polymer photodetectors. Adv. Mater. 30, 1803980 (2018).
Kong, Y. L., Gupta, M. K., Johnson, B. N. & McAlpine, M. C. 3D printed bionic nanodevices. Nano Today 11, 330–350 (2016).
Zhang, Y. et al. Printing, folding and assembly methods for forming 3D mesostructures in advanced materials. Nat. Rev. Mater. 2, 17019 (2017).
Murphy, S. V. & Atala, A. 3D bioprinting of tissues and organs. Nat. Biotechnol. 32, 773–785 (2014).
Singh, S., Choudhury, D., Yu, F., Mironov, V. & Naing, M. W. In situ bioprinting – bioprinting from benchside to bedside? Acta Biomater. 101, 14–25 (2020).
Zhu, Z. et al. 3D printed functional and biological materials on moving freeform surfaces. Adv. Mater. 30, 1707495 (2018).
O’Neill, J. J., Johnson, R. A., Dockter, R. L. & Kowalewski, T. M. in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. 934–940 (IEEE, 2017).
Razaviarab, N., Sharifi, S. & Banadaki, Y. M. Smart additive manufacturing empowered by a closed-loop machine learning algorithm. Proc. SPIE 10969, 109690H (2019).
DeCost, B. L., Jain, H., Rollett, A. D. & Holm, E. A. Computer vision and machine learning for autonomous characterization of AM powder feedstocks. JOM 69, 456–465 (2017).
Sitthi-Amorn, P. et al. MultiFab: a machine vision assisted platform for multi-material 3D printing. ACM Trans. Graph. 34, 129 (2015).
Stoyanov, S. & Bailey, C. in 40th Int. Spring Semin. Electron. Technol. 1–6 (IEEE, 2017).
Chen, D., Skouras, M., Zhu, B. & Matusik, W. Computational discovery of extremal microstructure families. Sci. Adv. 4, eaao7005 (2018).
MacCurdy, R., Lipton, J., Li, S. & Rus, D. in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. 2628–2635 (IEEE, 2016).
Langlois, T., Shamir, A., Dror, D., Matusik, W. & Levin, D. I. Stochastic structural analysis for context-aware design and fabrication. ACM Trans. Graph. 35, 226 (2016).
Dai, C. et al. Support-free volume printing by multi-axis motion. ACM Trans. Graph. 37, 134 (2018).
Yang, X., Sun, M., Bian, Y. & He, X. A room-temperature high-conductivity metal printing paradigm with visible-light projection lithography. Adv. Funct. Mater. 29, 1807615 (2019).
Huang, T. Q., Qu, X., Liu, J. & Chen, S. 3D printing of biomimetic microstructures for cancer cell migration. Biomed. Microdevices 16, 127–132 (2014).
Wang, Z. et al. A simple and high-resolution stereolithography-based 3D bioprinting system using visible light crosslinkable bioinks. Biofabrication 7, 045009 (2015).
Gauvin, R. et al. Microfabrication of complex porous tissue engineering scaffolds using 3D projection stereolithography. Biomaterials 33, 3824–3834 (2012).
Stevens, A. G. et al. Conformal robotic stereolithography. 3D Print. Addit. Manuf. 3, 226–235 (2016).
Han, D., Yang, C., Fang, N. X. & Lee, H. Rapid multi-material 3D printing with projection micro-stereolithography using dynamic fluidic control. Addit. Manuf. 27, 606–615 (2019).
Yun, H. & Kim, H. Development of DMD-based micro-stereolithography apparatus for biodegradable multi-material micro-needle fabrication. J. Mech. Sci. Technol. 27, 2973–2978 (2013).
Skylar-Scott, M. A., Mueller, J., Visser, C. W. & Lewis, J. A. Voxelated soft matter via multimaterial multinozzle 3D printing. Nature 575, 330–335 (2019).
Su, R., Park, S. H., Li, Z. & McAlpine,. M. C. in Robotic Systems and Autonomous Platforms (eds Walsh, S. M. & Strano, M. S.) 309–334 (Woodhead Publishing, 2018).
Haghiashtiani, G., Habtour, E., Park, S.-H., Gardea, F. & McAlpine, M. C. 3D printed electrically-driven soft actuators. Extreme Mech. Lett. 21, 1–8 (2018).
Hwang, S. W. et al. High-performance biodegradable/transient electronics on biodegradable polymers. Adv. Mater. 26, 3905–3911 (2014).
Kim, D.-H. et al. Epidermal electronics. Science 333, 838–843 (2011).
Zhou, N. et al. Perovskite nanowire–block copolymer composites with digitally programmable polarization anisotropy. Sci. Adv. 5, eaav8141 (2019).
Kim, D.-H. et al. Dissolvable films of silk fibroin for ultrathin conformal bio-integrated electronics. Nat. Mater. 9, 511–517 (2010).
Zare Bidoky, F., Hyun, W. J., Song, D. & Frisbie, C. D. Printed, 1 V electrolyte-gated transistors based on poly(3-hexylthiophene) operating at >10 kHz on plastic. Appl. Phys. Lett. 113, 053301 (2018).
Kim, H., Fernando, T., Li, M., Lin, Y. & Tseng, T.-L. B. Fabrication and characterization of 3D printed BaTiO3/PVDF nanocomposites. J. Compos. Mater. 52, 197–206 (2018).
Kim, K. et al. 3D optical printing of piezoelectric nanoparticle–polymer composite materials. ACS Nano 8, 9799–9806 (2014).
Ahn, B. Y. et al. Omnidirectional printing of flexible, stretchable, and spanning silver microelectrodes. Science 323, 1590–1593 (2009).
Nge, T. T., Nogi, M. & Suganuma, K. Electrical functionality of inkjet-printed silver nanoparticle conductive tracks on nanostructured paper compared with those on plastic substrates. J. Mater. Chem. C 1, 5235–5243 (2013).
Kamyshny, A. & Magdassi, S. Conductive nanomaterials for 2D and 3D printed flexible electronics. Chem. Soc. Rev. 48, 1712–1740 (2019).
Russo, A. et al. Pen-on-paper flexible electronics. Adv. Mater. 23, 3426–3430 (2011).
Benn, T., Cavanagh, B., Hristovski, K., Posner, J. D. & Westerhoff, P. The release of nanosilver from consumer products used in the home. J. Environ. Qual. 39, 1875–1882 (2010).
Liu, P. et al. Toxicity of nano- and micro-sized silver particles in human hepatocyte cell line L02. J. Phys. Conf. Ser. 304, 012036 (2011).
Kim, M. J. & Shin, S. Toxic effects of silver nanoparticles and nanowires on erythrocyte rheology. Food Chem. Toxicol. 67, 80–86 (2014).
Yu, Z., Qin, W., Lin, J., Fang, S. & Qiu, J. Antibacterial mechanisms of polymyxin and bacterial resistance. Biomed. Res. Int. 2015, 679109 (2015).
Williams, N. X. et al. Silver nanowire inks for direct-write electronic tattoo applications. Nanoscale 11, 14294–14302 (2019).
Le Bideau, J., Viau, L. & Vioux, A. Ionogels, ionic liquid based hybrid materials. Chem. Soc. Rev. 40, 907–925 (2011).
Truby, R. L. et al. Soft somatosensitive actuators via embedded 3D printing. Adv. Mater. 30, 1706383 (2018).
Wong, J. et al. 3D printing ionogel auxetic frameworks for stretchable sensors. Adv. Mater. Technol. 4, 1900452 (2019).
Sun, J.-Y., Keplinger, C., Whitesides, G. M. & Suo, Z. Ionic skin. Adv. Mater. 26, 7608–7614 (2014).
Lodge, T. P. A unique platform for materials design. Science 321, 50–51 (2008).
Zhu, Z., Park, H. S. & McAlpine, M. C. 3D printed deformable sensors. Sci. Adv. 6, eaba5575 (2020).
Lee, K. Y. & Mooney, D. J. Hydrogels for tissue engineering. Chem. Rev. 101, 1869–1880 (2001).
Seliktar, D. Designing cell-compatible hydrogels for biomedical applications. Science 336, 1124–1128 (2012).
Yuk, H., Lu, B. & Zhao, X. Hydrogel bioelectronics. Chem. Soc. Rev. 48, 1642–1667 (2019).
Van Vlierberghe, S., Dubruel, P. & Schacht, E. Biopolymer-based hydrogels as scaffolds for tissue engineering applications: a review. Biomacromolecules 12, 1387–1408 (2011).
Albanna, M. et al. In situ bioprinting of autologous skin cells accelerates wound healing of extensive excisional full-thickness wounds. Sci. Rep. 9, 1856 (2019).
Coradin, T., Allouche, J., Boissiere, M. & Livage, J. Sol-gel biopolymer/silica nanocomposites in biotechnology. Curr. Nanosci. 2, 219–230 (2006).
Broguiere, N., Cavalli, E., Salzmann, G. M., Applegate, L. A. & Zenobi-Wong, M. Factor XIII cross-linked hyaluronan hydrogels for cartilage tissue engineering. ACS Biomater. Sci. Eng. 2, 2176–2184 (2016).
Murphy, S. V., Skardal, A. & Atala, A. Evaluation of hydrogels for bio-printing applications. J. Biomed. Mater. Res. A 101A, 272–284 (2013).
Vijayavenkataraman, S., Lu, W. F. & Fuh, J. Y. H. 3D bioprinting of skin: a state-of-the-art review on modelling, materials, and processes. Biofabrication 8, 032001 (2016).
Cacopardo, L., Guazzelli, N., Nossa, R., Mattei, G. & Ahluwalia, A. Engineering hydrogel viscoelasticity. J. Mech. Behav. Biomed. Mater. 89, 162–167 (2019).
Yuk, H. et al. Dry double-sided tape for adhesion of wet tissues and devices. Nature 575, 169–174 (2019).
Li, J. et al. Tough adhesives for diverse wet surfaces. Science 357, 378–381 (2017).
Yuk, H., Zhang, T., Lin, S., Parada, G. A. & Zhao, X. Tough bonding of hydrogels to diverse non-porous surfaces. Nat. Mater. 15, 190–196 (2016).
Freyer, J. P., Fillak, D. & Jett, J. H. Use of xantham gum to suspend large particles during flow cytometric analysis and sorting. Cytometry 10, 803–806 (1989).
Schuurman, W. et al. Bioprinting of hybrid tissue constructs with tailorable mechanical properties. Biofabrication 3, 021001 (2011).
Hinton, T. J. et al. Three-dimensional printing of complex biological structures by freeform reversible embedding of suspended hydrogels. Sci. Adv. 1, e1500758 (2015).
Smith, S., Maclean, M., MacGregor, S. J., Anderson, J. G. & Grant, M. H. Exposure of 3T3 mouse fibroblasts and collagen to high intensity blue light. Int. Conf. Biomed. Eng. 23, 1352–1355 (2009).
Lewis, J. B. et al. Blue light differentially alters cellular redox properties. J. Biomed. Mater. Res. B 72B, 223–229 (2005).
Ligon, S. C., Liska, R., Stampfl, J., Gurr, M. & Mülhaupt, R. Polymers for 3D printing and customized additive manufacturing. Chem. Rev. 117, 10212–10290 (2017).
Singh, M. et al. 3D printed conformal microfluidics for isolation and profiling of biomarkers from whole organs. Lab Chip 17, 2561–2571 (2017).
Johnson, B. N. et al. 3D printed anatomical nerve regeneration pathways. Adv. Funct. Mater. 25, 6205–6217 (2015).
Lin, S. et al. Stretchable hydrogel electronics and devices. Adv. Mater. 28, 4497–4505 (2016).
Yu, Y. et al. Multifunctional “hydrogel skins” on diverse polymers with arbitrary shapes. Adv. Mater. 31, 1807101 (2019).
Yuk, H., Zhang, T., Parada, G. A., Liu, X. & Zhao, X. Skin-inspired hydrogel–elastomer hybrids with robust interfaces and functional microstructures. Nat. Commun. 7, 12028 (2016).
Yang, H. et al. Printing hydrogels and elastomers in arbitrary sequence with strong adhesion. Adv. Funct. Mater. 29, 1901721 (2019).
Reece, T. B., Maxey, T. S. & Kron, I. L. A prospectus on tissue adhesives. Am. J. Surg. 182, S40–S44 (2001).
Vakalopoulos, K. A. et al. Mechanical strength and rheological properties of tissue adhesives with regard to colorectal anastomosis: an ex vivo study. Ann. Surg. 261, 323–331 (2015).
Rose, S. et al. Nanoparticle solutions as adhesives for gels and biological tissues. Nature 505, 382–385 (2014).
Annabi, N., Yue, K., Tamayol, A. & Khademhosseini, A. Elastic sealants for surgical applications. Eur. J. Pharm. Biopharm. 95, 27–39 (2015).
Aboali, M., Manap, N. A., Darsono, A. M. & Yusof, Z. M. Review on three-dimensional (3-D) acquisition and range imaging techniques. Int. J. Appl. Eng. 12, 2409–2421 (2017).
Farahani, N. et al. Three-dimensional imaging and scanning: current and future applications for pathology. J. Pathol. Inform. 8, 36–36 (2017).
Cohen, D. L., Lipton, J. I., Bonassar, L. J. & Lipson, H. Additive manufacturing for in situ repair of osteochondral defects. Biofabrication 2, 035004 (2010).
Zhao, X., Pan, Y., Zhou, C., Chen, Y. & Wang, C. C. An integrated CNC accumulation system for automatic building-around-inserts. J. Manuf. Process. 15, 432–443 (2013).
Binder, K. W. In situ Bioprinting of the Skin. Thesis, Wake Forest Univ. (2011).
Li, L. et al. In situ repair of bone and cartilage defects using 3D scanning and 3D printing. Sci. Rep. 7, 9416 (2017).
Adams, J. J. et al. Conformal printing of electrically small antennas on three-dimensional surfaces. Adv. Mater. 23, 1335–1340 (2011).
Jafari, B. H. et al. in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. 1789–1794 (IEEE, 2018).
Song, X., Pan, Y. & Chen, Y. Development of a low-cost parallel kinematic machine for multidirectional additive manufacturing. J. Manuf. Sci. Eng. 137, 021005 (2015).
Wu, C., Dai, C., Fang, G., Liu, Y.-J. & Wang, C. C. in Proc. IEEE Int. Conf. Robot. Autom. 1175–1180 (IEEE, 2017).
Keriquel, V. et al. In situ printing of mesenchymal stromal cells, by laser-assisted bioprinting, for in vivo bone regeneration applications. Sci. Rep. 7, 1778 (2017).
Keriquel, V. et al. In vivo bioprinting for computer- and robotic-assisted medical intervention: preliminary study in mice. Biofabrication 2, 014101 (2010).
Xu, R. et al. Flexible and wearable 3D graphene sensor with 141 kHz frequency signal response capability. Appl. Phys. Lett. 111, 103501 (2017).
Chen, L. & Qi, S. 50 3D printing of dermal ECM hyfrogel enhances the therapeutic effects of split thickness skin grafting in full-thickness skin wound repair. J. Burn Care Res. 40, S35–S36 (2019).
Zolfagharian, A., Kaynak, A. & Kouzani, A. Closed-loop 4D-printed soft robots. Mater. Des. 188, 108411 (2020).
Agostiniani, V., DeSimone, A. & Koumatos, K. Shape programming for narrow ribbons of nematic elastomers. J. Elast. 127, 1–24 (2017).
Ferguson, A. L. Machine learning and data science in soft materials engineering. J. Phys. Condens. Matter 30, 043002 (2017).
Fernandes, P., Guedes, J. M. & Rodrigues, H. Topology optimization of three-dimensional linear elastic structures with a constraint on “perimeter”. Comput. Struct. 73, 583–594 (1999).
Maute, K. et al. Level set topology optimization of printed active composites. J. Mech. Des. 137, 111402 (2015).
Sawhney, R. & Crane, K. Boundary first flattening. ACM Trans. Graph. 37, 5 (2017).
Konaković, M. et al. Beyond developable: computational design and fabrication with auxetic materials. ACM Trans. Graph. 35, 89 (2016).
Kent, J. R., Carlson, W. E. & Parent, R. E. Shape transformation for polyhedral objects. ACM SIGGRAPH Comput. Graph. 26, 47–54 (1992).
Hamel, C. M. et al. Machine-learning based design of active composite structures for 4D printing. Smart Mater. Struct. 28, 065005 (2019).
Xiong, J., Yin, Z. & Zhang, W. Closed-loop control of variable layer width for thin-walled parts in wire and arc additive manufacturing. J. Mater. Process. Technol. 233, 100–106 (2016).
Go, J. & Hart, A. J. Fast desktop-scale extrusion additive manufacturing. Addit. Manuf. 18, 276–284 (2017).
Wang, T., Kwok, T.-H. & Zhou, C. In-situ droplet inspection and control system for liquid metal jet 3D printing process. Procedia Manuf. 10, 968–981 (2017).
Tlegenov, Y., Hong, G. S. & Lu, W. F. Nozzle condition monitoring in 3D printing. Robot. Comput. Integr. Manuf. 54, 45–55 (2018).
Zhang, X., Lies, B., Lyu, H. & Qin, H. In-situ monitoring of electrohydrodynamic inkjet printing via scalar diffraction for printed droplets. J. Manuf. Syst. 53, 1–10 (2019).
Lies, B. T., Cai, Y., Spahr, E., Lin, K. & Qin, H. Machine vision assisted micro-filament detection for real-time monitoring of electrohydrodynamic inkjet printing. Procedia Manuf. 26, 29–39 (2018).
Greeff, G. P. & Schilling, M. Closed loop control of slippage during filament transport in molten material extrusion. Addit. Manuf. 14, 31–38 (2017).
Jin, Z., Zhang, Z. & Gu, G. X. Autonomous in-situ correction of fused deposition modeling printers using computer vision and deep learning. Manuf. Lett. 22, 11–15 (2019).
Jin, Z., Zhang, Z. & Gu, G. X. Automated real-time detection and prediction of interlayer imperfections in additive manufacturing processes using artificial intelligence. Adv. Intell. Syst. 2, 1900130 (2020).
Faes, M. et al. Process monitoring of extrusion based 3D printing via laser scanning. Conf. Proc. PMI 6, 363–367 (2014).
Holzmond, O. & Li, X. In situ real time defect detection of 3D printed parts. Addit. Manuf. 17, 135–142 (2017).
Delli, U. & Chang, S. Automated process monitoring in 3D printing using supervised machine learning. Procedia Manuf. 26, 865–870 (2018).
French, A., Neill, J. O., Madson, R. & Kowalewski, T. M. in Int. Symp. Med. Robot. 1–6 (IEEE, 2018).
O’Neill, J. J. & Kowalewski, T. M. Online free anatomy registration via noncontact skeletal tracking for collaborative human/robot interaction in surgical robotics. J. Med. Devices 8, 030952 (2014).
Johnson, R. A., O’Neill, J. J., Dockter, R. L. & Kowalewski, T. M. in Des. Med. Devices Conf. V001T011A016 (ASME, 2017).
Torresani, L. & Hertzmann, A. Automatic non-rigid 3D modeling from video. Comput. Vis. ECCV 3022, 299–312 (2004).
Bregler, C., Hertzmann, A. & Biermann, H. Recovering non-rigid 3D shape from image streams. Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2, 690–696 (2000).
Lanitis, A., Taylor, C. J., Cootes, T. & Ahmed, T. in Proc. IEEE Int. Workshop Autom. Face Gesture Recognit. (IEEE, 1995).
Torresani, L., Yang, D. B., Alexander, E. J. & Bregler, C. Tracking and modeling non-rigid objects with rank constraints. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1, 493–500 (2001).
Guenter, B., Grimm, C., Wood, D., Malvar, H. & Pighin, F. in Proc. Annu. Conf. Comput. Graph. Interact. Tech. 55–56 (ACM, 1998).
Lin, B. et al. Video-based 3D reconstruction, laparoscope localization and deformation recovery for abdominal minimally invasive surgery: a survey. Int. J. Med. Robot. Comp. Assist. Surg. 12, 158–178 (2016).
Stephens, T. K. et al. Blended shared control utilizing online identification. Int. J. Comput. Assist. Radiol. Surg. 13, 769–776 (2018).
Sie, A., Winek, M. & Kowalewski, T. M. in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. 2036–2042 (IEEE, 2014).
Tholey, G., Desai, J. P. & Castellanos, A. E. Force feedback plays a significant role in minimally invasive surgery: results and analysis. Ann. Surg. 241, 102–109 (2005).
Young-Eun, S., Chi-Yen, K. & Lee, M. in Proc. IEEE Int. Symp. Industr. Electron. 2153–2158 (IEEE, 2009).
Kim, C. Y., Yoon, S. M., Lee, M. C. & Kang, B. H. in 8th Asian Control Conf. 553–557 (IEEE, 2011).
McVeigh, E. R. et al. Real-time interactive MRI-guided cardiac surgery: aortic valve replacement using a direct apical approach. Magn. Reson. Med. 56, 958–964 (2006).
Kenngott, H. G. et al. Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging. Surg. Endosc. 28, 933–940 (2014).
Hartley, R. & Zisserman, A. Multiple View Geometry in Computer Vision 2nd edn (Cambridge Univ. Press, 2003).
Geiger, A., Roser, M. & Urtasun, R. Efficient large-scale stereo matching. Asian Conf. Comput. Vis. 6492, 25–38 (2011).
Geiger, A., Ziegler, J. & Stiller, C. in IEEE Intell. Veh. Symp. 963–968 (IEEE, 2011).
Schmalz, C., Forster, F., Schick, A. & Angelopoulou, E. An endoscopic 3D scanner based on structured light. Med. Image Anal. 16, 1063–1072 (2012).
Lin, J., Clancy, N. T. & Elson, D. S. An endoscopic structured light system using multispectral detection. Int. J. Comput. Assist. Radiol. Surg. 10, 1941–1950 (2015).
Clancy, N. T. et al. Spectrally encoded fiber-based structured lighting probe for intraoperative 3D imaging. Biomed. Opt. Express 2, 3119–3128 (2011).
Reiter, A., Sigaras, A., Fowler, D. & Allen, P. K. in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. 1282–1287 (IEEE, 2014).
Zhang, L., Ye, M., Giataganas, P., Hughes, M. & Yang, G.-Z. in Proc. IEEE Int. Conf. Robot. Autom. 3587–3593 (IEEE, 2017).
Hughes, M. & Yang, G.-Z. Line-scanning fiber bundle endomicroscopy with a virtual detector slit. Biomed. Opt. Express 7, 2257–2268 (2016).
Zhang, L. et al. From macro to micro: autonomous multiscale image fusion for robotic surgery. IEEE Robot. Autom. Mag. 24, 63–72 (2017).
Stoyanov, D., Scarzanella, M. V., Pratt, P. & Yang, G.-Z. Real-time stereo reconstruction in robotically assisted minimally invasive surgery. Med. Image Comput. Comput. Assist. Interv. 6361, 275–282 (2010).
Lin, B., Sun, Y., Sanchez, J. E. & Qian, X. Efficient vessel feature detection for endoscopic image analysis. IEEE Trans. Biomed. Eng. 62, 1141–1150 (2015).
Rieke, N. et al. Real-time localization of articulated surgical instruments in retinal microsurgery. Med. Image Anal. 34, 82–100 (2016).
Zhao, Z., Voros, S., Weng, Y., Chang, F. & Li, R. Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method. Comput. Assist. Surg. 22, 26–35 (2017).
Chen, Z., Zhao, Z. & Cheng, X. in Chin. Autom. Congr. 2711–2714 (IEEE, 2017).
Pakhomov, D., Premachandran, V., Allan, M., Azizian, M. & Navab, N. Deep residual learning for instrument segmentation in robotic surgery. Mach. Learn. Med. Imaging 11861, 566–573 (2019).
Shvets, A. A., Rakhlin, A., Kalinin, A. A. & Iglovikov, V. I. in 17th IEEE Int. Conf. Mach. Learn. Appl. 624–628 (IEEE, 2018).
Laina, I. et al. Concurrent segmentation and localization for tracking of surgical instruments. Med. Image Comput. Comput. Assist. Interv. 10434, 664–672 (2017).
García-Peraza-Herrera, L. C. et al. Real-time segmentation of non-rigid surgical tools based on deep learning and tracking. Comput. Assist. Robot. Endosc. 10170, 84–95 (2017).
Al Hajj, H., Lamard, M., Conze, P.-H., Cochener, B. & Quellec, G. Monitoring tool usage in surgery videos using boosted convolutional and recurrent neural networks. Med. Image Anal. 47, 203–218 (2018).
Kumar, S., Narayanan, M. S., Singhal, P., Corso, J. J. & Krovi, V. in Proc. IEEE Int. Conf. Autom. Sci. Eng. 480–485 (IEEE, 2013).
Islam, M. & Ren, H. Multi-modal PixelNet for brain tumor segmentation. Brainles. 10670, 298–308 (2018).
Winzeck, S. et al. ISLES 2016 and 2017-benchmarking ischemic stroke lesion outcome prediction based on multispectral MRI. Front. Neurol. 9, 679 (2018).
Kamnitsas, K. et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med. Image Anal. 36, 61–78 (2017).
Wu, A., Xu, Z., Gao, M., Buty, M. & Mollura, D. J. in Proc. IEEE 13th Int. Symp. Biomed. Imaging 1363–1367 (IEEE, 2016).
Terunuma, T., Tokui, A. & Sakae, T. Novel real-time tumor-contouring method using deep learning to prevent mistracking in X-ray fluoroscopy. Radiol. Phys. Technol. 11, 43–53 (2018).
Islam, M., Atputharuban, D. A., Ramesh, R. & Ren, H. Real-time instrument segmentation in robotic surgery using auxiliary supervised deep adversarial learning. IEEE Robot. Autom. Lett. 4, 2188–2195 (2019).
Zhao, H., Qi, X., Shen, X., Shi, J. & Jia, J. ICNet for real-time semantic segmentation on high-resolution images. Eur. Conf. Comput. Vis. 11207, 418–434 (2018).
Zhao, H., Shi, J., Qi, X., Wang, X. & Jia, J. in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2881–2890 (IEEE, 2017).
Ronneberger, O., Fischer, P. & Brox, T. U-Net: convolutional networks for biomedical Image segmentation. Med. Image Comput. Comput. Assist. Interv. 9351, 234–241 (2015).
Voros, S., Long, J.-A. & Cinquin, P. Automatic detection of instruments in laparoscopic images: a first step towards high-level command of robotic endoscopic holders. Int. J. Robot. Res. 26, 1173–1190 (2007).
Krupa, A. et al. Autonomous 3-D positioning of surgical instruments in robotized laparoscopic surgery using visual servoing. IEEE Trans. Robot. Autom. 19, 842–853 (2003).
Lladó, X., Del Bue, A., Oliver, A., Salvi, J. & Agapito, L. Reconstruction of non-rigid 3D shapes from stereo-motion. Pattern Recognit. Lett. 32, 1020–1028 (2011).
Bue, A. D., Llad, X. & Agapito, L. Non-rigid metric shape and motion recovery from uncalibrated images using priors. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. 1, 1191–1198 (IEEE, 2006).
Sorkine, O. & Alexa, M. in Proc. 5th Eurograph. Symp. Geom. Process. 109–116 (Eurographics Association, 2007).
Zollhöfer, M. et al. Real-time non-rigid reconstruction using an RGB-D camera. ACM Trans. Graph. 33, 156 (2014).
Agudo, A., Calvo, B. & Montiel, J. M. M. 3D reconstruction of non-rigid surfaces in real-time using wedge elements. Eur. Conf. Comput. Vis. 7583, 113–122 (2012).
Agudo, A., Calvo, B. & Montiel, J. M. M. in Proc. IEEE Int. Conf. Comput. Vis. 1586–1593 (IEEE, 2011).
Sumner, R. W., Schmid, J. & Pauly, M. Embedded deformation for shape manipulation. ACM Trans. Graph. 26, 80 (2007).
Dou, M. et al. Fusion4D: real-time performance capture of challenging scenes. ACM Trans. Graph. 35, 114 (2016).
Song, J., Wang, J., Zhao, L., Huang, S. & Dissanayake, G. Dynamic reconstruction of deformable soft-tissue with stereo scope in minimal invasive surgery. IEEE Robot. Autom. Lett. 3, 155–162 (2018).
Guo, K. et al. Real-time geometry, albedo, and motion reconstruction using a single RGB-D camera. ACM Trans. Graph. 36, 44a (2017).
Torresani, L., Hertzmann, A. & Bregler, C. Nonrigid structure-from-motion: estimating shape and motion with hierarchical priors. IEEE Trans. Pattern Anal. Mach. Intell. 30, 878–892 (2008).
Blanz, V. & Vetter, T. in Proc. 26th Annu. Conf. Comput. Graph. Interact. Tech. 187–194 (ACM, 1999).
Cootes, T. F. & Taylor, C. J. Statistical models of appearance for medical image analysis and computer vision. Proc. SPIE 4322, 236–248 (2001).
Sirovich, L. & Kirby, M. Low-dimensional procedure for the characterization of human faces. J. Opt. Soc. Am. A 4, 519–524 (1987).
Turk, M. & Pentland, A. Eigenfaces for recognition. J. Cogn. Neurosci. 3, 71–86 (1991).
Song, J., Wang, J., Zhao, L., Huang, S. & Dissanayake, G. MIS-SLAM: Real-time large-scale dense deformable slam system in minimal invasive surgery based on heterogeneous computing. IEEE Robot. Autom. Lett. 3, 4068–4075 (2018).
Newcombe, R. A., Fox, D. & Seitz, S. M. in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 343–352 (IEEE, 2015).
Innmann, M., Zollhöfer, M., Nießner, M., Theobalt, C. & Stamminger, M. VolumeDeform: real-time volumetric non-rigid reconstruction. Eur. Conf. Comput. Vis. 9912, 362–379 (2016).
Turan, M., Almalioglu, Y., Araujo, H., Konukoglu, E. & Sitti, M. A non-rigid map fusion-based direct SLAM method for endoscopic capsule robots. Int. J. Intell. Robot. Appl. 1, 399–409 (2017).
Lamarca, J., Parashar, S., Bartoli, A. & Montiel, J. DefSLAM: tracking and mapping of deforming scenes from monocular sequences. Preprint at arXiv https://arxiv.org/abs/1908.08918 (2019).
Song, J., Zhao, L., Huang, S. & Dissanayake, G. An observable time series based SLAM algorithm for deforming environment. Preprint at arXiv https://arxiv.org/abs/1906.08563 (2019).
Yang, B., Liu, C., Zheng, W. & Liu, S. Motion prediction via online instantaneous frequency estimation for vision-based beating heart tracking. Inf. Fusion 35, 58–67 (2017).
Mountney, P. & Yang, G.-Z. Motion compensated SLAM for image guided surgery. Med. Image Comput. Comput. Assist. Interv. 6362, 496–504 (2010).
Kehoe, B. et al. in Proc. IEEE Int. Conf. Robot. Autom. 1432–1439 (IEEE, 2014).
Losi, P. et al. Cyanoacrylate surgical glue as an alternative to suture threads for mesh fixation in hernia repair. J. Surg. Res. 163, e53–e58 (2010).
Hanson, T. L., Diaz-Botia, C. A., Kharazia, V., Maharbiz, M. M. & Sabes, P. N. The “sewing machine” for minimally invasive neural recording. Preprint at bioRxiv https://www.biorxiv.org/content/10.1101/578542v1 (2019).
Hakimi, N. et al. Handheld skin printer: in situ formation of planar biomaterials and tissues. Lab Chip 18, 1440–1451 (2018).
Di Bella, C. et al. In situ handheld three-dimensional bioprinting for cartilage regeneration. J. Tissue Eng. Regen. Med. 12, 611–621 (2018).
O’Connell, C. D. et al. Development of the biopen: a handheld device for surgical printing of adipose stem cells at a chondral wound site. Biofabrication 8, 015019 (2016).
Lane, T. A short history of robotic surgery. Ann. R. Coll. Surg. Engl. 100, 5–7 (2018).
Shojania, K. G. & Dixon-Woods, M. Estimating deaths due to medical error: the ongoing controversy and why it matters. BMJ Qual. Saf. 26, 423–428 (2017).
Nio, D., Diks, J., Bemelman, W. A., Wisselink, W. & Legemate, D. A. Laparoscopic vascular surgery: a systematic review. Eur. J. Vasc. Endovasc. Surg. 33, 263–271 (2007).
Kim, Y., Parada, G. A., Liu, S. & Zhao, X. Ferromagnetic soft continuum robots. Sci. Robot. 4, eaax7329 (2019).
Lehman, A. C., Wood, N. A., Farritor, S., Goede, M. R. & Oleynikov, D. Dexterous miniature robot for advanced minimally invasive surgery. Surg. Endosc. 25, 119–123 (2011).
Niccolini, M., Petroni, G., Menciassi, A. & Dario, P. in Proc. IEEE Int. Conf. Roboti. Autom. 3395–3400 (IEEE, 2012).
Orekhov, A. L., Abah, C. & Simaan, N. in The Encyclopedia of Medical Robotics Vol. 1 (ed. Patel, R.) 203–243 (World Scientific, 2018).
Kaouk, J. H. et al. A novel robotic system for single-port urologic surgery: first clinical investigation. Eur. Urol. 66, 1033–1043 (2014).
Agarwal, D. K. et al. Initial experience with da Vinci single-port robot-assisted radical prostatectomies. Eur. Urol. 77, 373–379 (2019).
Morelli, L. et al. Da Vinci single site surgical platform in clinical practice: a systematic review. Int. J. Med. Robot. Comp. Assist. Surg. 12, 724–734 (2016).
de Moura, D. T. H. et al. Robot-assisted endoscopic submucosal dissection versus conventional ESD for colorectal lesions: outcomes of a randomized pilot study in endoscopists without prior ESD experience (with video). Gastrointest. Endosc. 90, 290–298 (2019).
Sethi, N. et al. Transoral robotic surgery using the Medrobotic Flex system: the Adelaide experience. J. Robot. Surg. 14, 109–113 (2019).
Persky, M. J. et al. Transoral surgery using the Flex robotic system: initial experience in the United States. Head Neck 40, 2482–2486 (2018).
Peters, B. S., Armijo, P. R., Krause, C., Choudhury, S. A. & Oleynikov, D. Review of emerging surgical robotic technology. Surg. Endosc. 32, 1636–1655 (2018).
Musk, E. An integrated brain-machine interface platform with thousands of channels. J. Med. Internet Res. 21, e16194 (2019).
McKinley, S. et al. in Proc. IEEE Int. Conf. Robot. Autom. (IEEE, 2016).
Xu, L. et al. 3D multifunctional integumentary membranes for spatiotemporal cardiac measurements and stimulation across the entire epicardium. Nat. Commun. 5, 3329 (2014).
McClintock, H., Temel, F. Z., Doshi, N., Koh, J.-S. & Wood, R. J. The milliDelta: A high-bandwidth, high-precision, millimeter-scale Delta robot. Sci. Robot. 3, eaar3018 (2018).
Roach, D. J. et al. The m4 3D printer: a multi-material multi-method additive manufacturing platform for future 3D printed structures. Addit. Manuf. 29, 100819 (2019).
Padoy, N. & Hager, G. D. in Proc. IEEE Int. Conf. Robot. Autom. 5285–5292 (IEEE, 2011).
Moustris, G. P., Hiridis, S. C., Deliparaschos, K. M. & Konstantinidis, K. M. Evolution of autonomous and semi-autonomous robotic surgical systems: a review of the literature. Int. J. Med. Robot. Comp. Assist. Surg. 7, 375–392 (2011).
Peng, H. et al. in Proc. Conf. Hum. Factors Comput. Syst. 579 (ACM, 2018).
Zhang, J., Zhong, Y. & Gu, C. Deformable models for surgical simulation: a survey. IEEE Rev. Biomed. Eng. 11, 143–164 (2017).
Pfeiffer, M., Riediger, C., Weitz, J. & Speidel, S. Learning soft tissue behavior of organs for surgical navigation with convolutional neural networks. Int. J. Comput. Assist. Radiol. Surg. 14, 1147–1155 (2019).
M.C.M. acknowledges support by the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award number DP2EB020537. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. H.S.P. acknowledges support from the Division of Information and Intelligent Systems of the National Science Foundation (1846031). Z.Z. acknowledges support from the graduate school of the University of Minnesota (2019–20 Doctoral Dissertation Fellowship).
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Zhu, Z., Ng, D.W.H., Park, H.S. et al. 3D-printed multifunctional materials enabled by artificial-intelligence-assisted fabrication technologies. Nat Rev Mater 6, 27–47 (2021). https://doi.org/10.1038/s41578-020-00235-2
This article is cited by
Nature Communications (2023)
Rapid prototyping of high-resolution large format microfluidic device through maskless image guided in-situ photopolymerization
Nature Communications (2023)
Nature Reviews Materials (2023)
Frontiers of Physics (2023)
Metals and Materials International (2023)