Abstract
Biological visual systems have inspired the development of various artificial visual systems including those based on human eyes (terrestrial environment), insect eyes (terrestrial environment) and fish eyes (aquatic environment). However, attempts to develop systems for both terrestrial and aquatic environments remain limited, and bioinspired electronic eyes are restricted in their maximum field of view to a hemispherical field of view (around 180°). Here we report the development of an amphibious artificial vision system with a panoramic visual field inspired by the functional and anatomical structure of the compound eyes of a fiddler crab. We integrate a microlens array with a graded refractive index and a flexible comb-shaped silicon photodiode array on a spherical structure. The microlenses have a flat surface and maintain their focal length regardless of changes in the external refractive index between air and water. The comb-shaped image sensor arrays on the spherical substrate exhibit an extremely wide field of view covering almost the entire spherical geometry. We illustrate the capabilities of our system via optical simulations and imaging demonstrations in both air and water.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 12 digital issues and online access to articles
$119.00 per year
only $9.92 per issue
Buy this article
- Purchase on SpringerLink
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
The data that support the plots within this paper and other findings of this study are available from the corresponding authors upon reasonable request.
Code availability
The source codes for MATLAB are available from the corresponding authors upon request.
References
Ko, H. C. et al. A hemispherical electronic eye camera based on compressible silicon optoelectronics. Nature 454, 748–753 (2008).
Choi, C. et al. Human eye-inspired soft optoelectronic device using high-density MoS2-graphene curved image sensor array. Nat. Commun. 8, 1664 (2017).
Gu, L. et al. A biomimetic eye with a hemispherical perovskite nanowire array retina. Nature 581, 278–282 (2020).
Rao, Z. et al. Curvy, shape-adaptive imagers based on printed optoelectronic pixels with a kirigami design. Nat. Electron. 4, 513–521 (2021).
Song, Y. M. et al. Digital cameras with designs inspired by the arthropod eye. Nature 497, 95–99 (2013).
Zhang, K. et al. Origami silicon optoelectronics for hemispherical electronic eye systems. Nat. Commun. 8, 1782 (2017).
Kim, M. S. et al. An aquatic-vision-inspired camera based on a monocentric lens and a silicon nanorod photodiode array. Nat. Electron. 3, 546–553 (2020).
How, M. J. et al. Target detection is enhanced by polarization vision in a fiddler crab. Cur. Biol. 25, 3069–3073 (2015).
Smolka, J. & Hemmi, J. M. Topography of vision and behavior. J. Exp. Biol. 212, 3522–3532 (2009).
Alkaladi, A. & Zeil, J. Functional anatomy of the fiddler crab compound eye (Uca vomeris: Ocypodidae, Brachyura, Decapoda). J. Comp. Neurol. 522, 1264–1283 (2014).
Bryceson, K. P. Focusing of light by corneal lenses in a reflecting superposition eye. J. Exp. Biol. 90, 347–350 (1981).
Nilsson, D.-E. Three unexpected cases of refracting superposition eyes in crustaceans. J. Comp. Physiol. A 167, 71–78 (1990).
Horváth, G. Geometric optical optimization of the corneal lens of Notonecta glauca. J. Theor. Biol. 139, 389–404 (1989).
Jang, H. J. et al. Double-sided anti-reflection nanostructures on optical convex lenses for imaging applications. Coatings 9, 404 (2019).
Zeil, J., Hemmi, J. M. & Backwell, P. R. Y. Fiddler crabs. Curr. Biol. 16, R40–R41 (2006).
Castro, P., Davie, P., Guinot, D., Schram, F. R. & von Vaupel Klein, J. C. The Crustacea. in Treatise on Zoology—Anatomy, Taxonomy, Biology Vol. 9 (Brill, 2015).
Ng, P. K. L., Schubart, C. D. & Lukhaup, C. New species of ‘vampire crabs’ (Geosesarma De Man, 1892) from central Java, Indonesia, and the identity of Sesarma (Geosesarma) nodulifera De Man, 1892 (Crustacea, Brachyura, Thoracotremata, Sesarmidae). Raffles Bull. Zool. 63, 3–13 (2015).
Green, P. T. Field observations of moulting and moult increment in the red land crab, Gecarcoidea natalis (Brachyura, Gecarcinidae), on Christmas Island (Indian Ocean). Crustaceana 77, 125–128 (2004).
Nilsson, D. E. A new type of imaging optics in compound eyes. Nature 332, 76–78 (1988).
Arikawa, K., Kawamata, K., Suzuki, T. & Eguchi, E. Daily changes of structure, function and rhodopsin content in the compound eye of the crab Hemigrapsus sanguineus. J. Comp. Physiol. A 161, 161–174 (1987).
McLay, C. L. Brachyura and crab-like Anomura of New Zealand. Leigh Lab. Bull. 22, 463 (1988).
Eguchi, E., Dezawa, M. & Meyer-Rochow, V. B. Compound eye fine structure in Paralomis multispina Benedict, an anomuran half-crab from 1200 m depth (Crustacea; Decapoda; Anomura). Biol. Bull. 192, 300–308 (1997).
Luque, J. et al. Evolution of crab eye structures and the utility of ommatidia morphology in resolving phylogeny. Preprint at bioRxiv https://doi.org/10.1101/786087 (2019).
Cai, J., Townsend, J. P., Dodson, T. C., Heiney, P. A. & Sweeney, A. M. Eye patches: protein assembly of index-gradient squid lenses. Science 357, 564–569 (2017).
Alagboso, F. et al. Ultrastructure and mineral composition of the cornea cuticle in the compound eyes of a supralittoral and a marine isopod. J. Struct. Biol. 187, 158–173 (2014).
Kim, K., Jang, K.-W., Ryu, J.-K. & Jeong, K.-H. Biologically inspired ultrathin arrayed camera for high-contrast and high-resolution imaging. Light. Sci. Appl. 9, 28 (2020).
Lee, W. et al. Two-dimensional materials in functional three-dimensional architectures with applications in photodetection and imaging. Nat. Commun. 9, 1417 (2018).
Sim, K. et al. Three-dimensional curvy electronics created using conformal additive stamp printing. Nat. Electron. 2, 471–479 (2019).
Lee, W. et al. High-resolution spin-on-patterning of perovskite thin films for a multiplexed image sensor array. Adv. Mater. 29, 1702902 (2017).
Sheng, X. et al. Silicon-based visible-blind ultraviolet detection and imaging using down-shifting luminophores. Adv. Opt. Mater. 2, 314–319 (2014).
Kim, J. et al. Stretchable silicon nanoribbon electronics for skin prosthesis. Nat. Commun. 5, 5747 (2014).
Song, E. et al. Flexible electronic/optoelectronic microsystems with scalable designs for chronic biointegration. Proc. Natl Acad. Sci. USA 116, 15398–15406 (2019).
Hughes, C., Glavin, M., Jones, E. & Denny, P. Review of geometric distortion compensation in fish-eye cameras. In IET Signals and Systems Conference (ISSC 2008) 162–167 (IEEE, 2008).
Liang, W.-L., Pan, J.-G. & Su, G.-D. J. One-lens camera using a biologically based artificial compound eye with multiple focal lengths. Optica 6, 326–334 (2019).
Brodrick, E. A., Roberts, N. W., Sumner-Ronney, L., Schleputz, C. M. & How, M. J. Light adaptation mechanisms in the eye of the fiddler crab Afruca tangeri. J. Comp. Neurol. 529, 616–634 (2021).
Kim, H. M., Kim, M. S., Lee, G. J., Yoo, Y. J. & Song, Y. M. Large area fabrication of engineered microlens array with low sag height for light-field imaging. Opt. Express 27, 4435–4444 (2019).
Acknowledgements
This research was supported by IBS-R006-A1. This research was also supported by the National Research Foundation (NRF) of Korea (2017M3D1A1039288 and 2021R1A6A3A01087961) and the GIST-MIT Research Collaboration grant funded by the GIST in 2022.
Author information
Authors and Affiliations
Contributions
M.L., G.J.L., H.J.J., D.-H.K. and Y.M.S. designed the experiments, analysed the data and wrote the paper. M.L. and H.C. prepared the biological specimens. J.H.L. performed the electron microscopy experiments. M.L., H.C., E.J. and M.K. fabricated the photodiode array and performed the characterization of individual devices. M.L. and H.C. performed the assembly of devices. G.J.L., H.J.J., M.S.K., H.M.K. and J.-E.Y. performed the theoretical and experimental analysis of the optics. G.J.L., H.J.J., K.M.K. and F.D. conducted the imaging simulation and demonstration. H.J. and N.L. performed the theoretical analysis of the mechanics. All the authors discussed the results and commented on the manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Electronics thanks Guo-Dung Su and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data
Extended Data Fig. 1 Hemispherical (convex type) cornea facet lenses in the eyes of terrestrial arthropods.
(a) Optical microscope images of a dragonfly eye (Epophthalmia elegans, left) and its SEM image (right). Its cornea facet lens has a convex-type hemispherical surface. (b) SEM images of a fly eye (Calliphora vomitori, left) and its magnified view (right), which show the convex-type hemispherical surface of the cornea facet lens.
Extended Data Fig. 2 Characterization of the fabricated g-ML arrays with various sizes.
(a–c), Each layer of each g-ML array is scanned by confocal microscopy to reconstruct the 3D surface profile of each layer. The diameters (D) of the g-MLs are 17 μm (a), 80 μm (b), and 400 μm (c). (d–f) Colorized cross-sectional SEM images of the g-MLs (top) and deviations of the RoCs from the average RoCs in the g-ML layers (bottom) for three different micro-lens diameters, that is, 17 µm (d), 80 µm (e), and 400 µm (f). (g) 2D cross-sectional profiles of g-MLs with three different diameters (17 µm, 80 µm, and 400 µm in the top, middle, and bottom frames, respectively).
Extended Data Fig. 3 Custom-made experimental setup for lens characterization in air and water, and measurement results.
(a) Optical camera image of the custom-made experimental setup. The setup consists of a white light source or laser sources with wavelengths of 450, 532, and 635 nm; an objective lens (10×); and a CMOS image sensor. (b–d) Optical camera images showing the components, including the water container, for amphibious imaging. (e) Measured light intensity distribution of the h-ML (micro-lens size of 400 μm) for three wavelengths (left: 450 nm, centre: 532 nm, and right: 635 nm). (f) Cross-sectional light intensity profile of the g-ML with a micro-lens size of 400 μm at the focal length in air (solid line) and water (dotted line). These results show that the g-ML obtains identical profiles in both media at the same focal length. (g) Imaging results of a letter ‘E’ obtained by the g-ML (top) and h-ML (bottom) for various micro-lens sizes (that is 17, 80, and 400 μm) under white light. (h) Measured focal lengths of g-MLs and h-MLs with three sizes in air and water. For all sizes, the g-MLs show consistent focal lengths in air and water, whereas the h-MLs show large changes of their focal lengths between air and water.
Extended Data Fig. 4 Schematic illustrations of multiple soft moulding processes for the fabrication of the g-ML array.
(a–c) Fabrication steps for a spacer layer and a base micro-lens array. (a) An anti-adhesive (Teflon AFTM) is coated on a quartz mould. (b) The prepolymer (clear flexTM 50) for the spacer and base micro-lenses is poured onto the quartz mould and thermally cured for 20 h. (c) The spacer and base micro-lens layer is detached from the quartz mould. (d–g) Fabrication steps for intermediate lens layers of the g-ML. For intermediate g-ML layers, silicone (EcoflexTM) moulds of plano-concave shapes with different curvatures are prepared. These steps are repeated for two intermediate NOA layers. (d) NOA is poured onto the silicone mould, and air bubbles are removed. (e) The silicone mould and base micro-lenses are aligned using a microscope. (f) The silicone mould and base micro-lenses are brought into contact. NOA is cured by UV illumination. (g) The silicone mould is removed, and unnecessary NOA is removed using tweezers. (h) The multi-layered NOA structure is formed on the base micro-lens. (i–l) Fabrication steps for a flat g-ML surface. (i) The silicone mould for the flat surface lens is prepared. NOA for the flat surface lens is poured onto the silicone mould. The multi-layered lens and silicone mould are aligned using a microscope. (j) The silicone mould and the multi-layered lens are brought into contact. NOA is cured using UV radiation. (k) Unnecessary NOA is removed using tweezers. (l) A g-ML with a flat surface is fabricated. (m, n) The screening pigment (SP) is spray-printed onto the g-ML and dried. (o, p) For light transmission, the SP on the g-ML surface is removed by bringing it into contact with solvent-coated (propylene glycol methyl ether acetate) glass.
Extended Data Fig. 5 Design of the panoramic artificial vision using the spherical 3D structure and finite element analysis (FEA) of the induced strain in the image sensor array.
(a) Schematic illustration of the panoramic artificial vision. The comb-shaped image sensor array and the g-ML array are integrated on the spherical 3D structure. (b) Schematic illustration defining the acceptance angle (∆θ) and inter-ommatidial angles in the vertical and horizontal directions (∆φv and ∆φh). Considering the acceptance angle, the inter-ommatidial angles are determined to avoid overlapping between the visual fields of neighbouring pixels. (c) Geometry of the g-ML used for FEA. (d) Elastic modulus (e) of each layer in the g-ML. (e) Strain distributions induced in the image sensor and g-ML arrays. Because the outermost lens has the highest elastic modulus, the entire g-ML experiences only a nominal strain under the bending deformation.
Extended Data Fig. 6 Imaging simulations for the optical parameters of the artificial vision.
(a) Imaging simulation results of a cubic pattern with eight different acceptance angles (∆θ) from 5.7° to 18°. The inset shows an original image of the cubic pattern. (b) Imaging simulation results of a cubic pattern with eight different inter-ommatidial angles (∆φ) from 1° to 15°. The dashed boxes indicate the used conditions in the measurement.
Extended Data Fig. 7 Optical simulations for inter-ommatidial and acceptance angles.
(a) Schematic for the distance-relative visual field of the ommatidia with an acceptance angle of ∆θ and an inter-ommatidial angle of ∆φ. (b) Illustration for the visual field variation by changing the acceptance and inter-ommatidial angles. An undetectable region, the area which cannot be captured by ommatidia, can be compensated by changing the angles. (c) Schematic illustration for describing the number of ommatidia capturing the object. (d) Calculated number of related ommatidia in (c) as a function of the acceptance angle and inter-ommatidial angle at an object distance of 5 mm, 40 mm, and 80 mm. (e) Original crab image for the image simulation. (f) Results of the imaging simulation with an object distance of 5 mm and an inter-ommatidial angle of 1°. (g) Results of the imaging simulation at an object distance of 40 mm with various acceptance angles and inter-ommatidial angles for the enhanced imaging resolution.
Extended Data Fig. 8 Simulation and experimental results of the panoramic imaging by the artificial vision.
(a) Schematic illustration of the artificial vision from the top view. (b) Simulation results of the panoramic imaging. (c) Experimental results of the panoramic imaging by the artificial vision.
Supplementary information
Supplementary Information
Supplementary Figs. 1–22, Tables 1–5 and Notes 1–7.
Rights and permissions
About this article
Cite this article
Lee, M., Lee, G.J., Jang, H.J. et al. An amphibious artificial vision system with a panoramic visual field. Nat Electron 5, 452–459 (2022). https://doi.org/10.1038/s41928-022-00789-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41928-022-00789-9
This article is cited by
-
A bionic self-driven retinomorphic eye with ionogel photosynaptic retina
Nature Communications (2024)
-
Toward grouped-reservoir computing: organic neuromorphic vertical transistor with distributed reservoir states for efficient recognition and prediction
Nature Communications (2024)
-
Damage-free dry transfer method using stress engineering for high-performance flexible two- and three-dimensional electronics
Nature Materials (2024)
-
Anti-distortion bioinspired camera with an inhomogeneous photo-pixel array
Nature Communications (2024)
-
Synaptic a-Si:H/a-Ga2O3 phototransistor inspired by the phototaxis behavior of organisms with all-optical and all-electrical stimulation modes
Nano Research (2024)
