Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

An amphibious artificial vision system with a panoramic visual field

Abstract

Biological visual systems have inspired the development of various artificial visual systems including those based on human eyes (terrestrial environment), insect eyes (terrestrial environment) and fish eyes (aquatic environment). However, attempts to develop systems for both terrestrial and aquatic environments remain limited, and bioinspired electronic eyes are restricted in their maximum field of view to a hemispherical field of view (around 180°). Here we report the development of an amphibious artificial vision system with a panoramic visual field inspired by the functional and anatomical structure of the compound eyes of a fiddler crab. We integrate a microlens array with a graded refractive index and a flexible comb-shaped silicon photodiode array on a spherical structure. The microlenses have a flat surface and maintain their focal length regardless of changes in the external refractive index between air and water. The comb-shaped image sensor arrays on the spherical substrate exhibit an extremely wide field of view covering almost the entire spherical geometry. We illustrate the capabilities of our system via optical simulations and imaging demonstrations in both air and water.

This is a preview of subscription content, access via your institution

Access options

Buy article

Get time limited or full article access on ReadCube.

$32.00

All prices are NET prices.

Fig. 1: Structural characteristics of the fiddler crab eye.
Fig. 2: Optical simulation and experimental characterization of microlenses for amphibious imaging.
Fig. 3: Integrated device on a 3D structure for panoramic imaging.
Fig. 4: Amphibious and panoramic imaging demonstration.

Data availability

The data that support the plots within this paper and other findings of this study are available from the corresponding authors upon reasonable request.

Code availability

The source codes for MATLAB are available from the corresponding authors upon request.

References

  1. Ko, H. C. et al. A hemispherical electronic eye camera based on compressible silicon optoelectronics. Nature 454, 748–753 (2008).

    Article  Google Scholar 

  2. Choi, C. et al. Human eye-inspired soft optoelectronic device using high-density MoS2-graphene curved image sensor array. Nat. Commun. 8, 1664 (2017).

    Article  Google Scholar 

  3. Gu, L. et al. A biomimetic eye with a hemispherical perovskite nanowire array retina. Nature 581, 278–282 (2020).

    Article  Google Scholar 

  4. Rao, Z. et al. Curvy, shape-adaptive imagers based on printed optoelectronic pixels with a kirigami design. Nat. Electron. 4, 513–521 (2021).

    Article  Google Scholar 

  5. Song, Y. M. et al. Digital cameras with designs inspired by the arthropod eye. Nature 497, 95–99 (2013).

    Article  Google Scholar 

  6. Zhang, K. et al. Origami silicon optoelectronics for hemispherical electronic eye systems. Nat. Commun. 8, 1782 (2017).

    Article  Google Scholar 

  7. Kim, M. S. et al. An aquatic-vision-inspired camera based on a monocentric lens and a silicon nanorod photodiode array. Nat. Electron. 3, 546–553 (2020).

    Article  Google Scholar 

  8. How, M. J. et al. Target detection is enhanced by polarization vision in a fiddler crab. Cur. Biol. 25, 3069–3073 (2015).

    Article  Google Scholar 

  9. Smolka, J. & Hemmi, J. M. Topography of vision and behavior. J. Exp. Biol. 212, 3522–3532 (2009).

    Article  Google Scholar 

  10. Alkaladi, A. & Zeil, J. Functional anatomy of the fiddler crab compound eye (Uca vomeris: Ocypodidae, Brachyura, Decapoda). J. Comp. Neurol. 522, 1264–1283 (2014).

    Article  Google Scholar 

  11. Bryceson, K. P. Focusing of light by corneal lenses in a reflecting superposition eye. J. Exp. Biol. 90, 347–350 (1981).

    Article  Google Scholar 

  12. Nilsson, D.-E. Three unexpected cases of refracting superposition eyes in crustaceans. J. Comp. Physiol. A 167, 71–78 (1990).

    Article  Google Scholar 

  13. Horváth, G. Geometric optical optimization of the corneal lens of Notonecta glauca. J. Theor. Biol. 139, 389–404 (1989).

    Article  Google Scholar 

  14. Jang, H. J. et al. Double-sided anti-reflection nanostructures on optical convex lenses for imaging applications. Coatings 9, 404 (2019).

    Article  Google Scholar 

  15. Zeil, J., Hemmi, J. M. & Backwell, P. R. Y. Fiddler crabs. Curr. Biol. 16, R40–R41 (2006).

    Article  Google Scholar 

  16. Castro, P., Davie, P., Guinot, D., Schram, F. R. & von Vaupel Klein, J. C. The Crustacea. in Treatise on Zoology—Anatomy, Taxonomy, Biology Vol. 9 (Brill, 2015).

  17. Ng, P. K. L., Schubart, C. D. & Lukhaup, C. New species of ‘vampire crabs’ (Geosesarma De Man, 1892) from central Java, Indonesia, and the identity of Sesarma (Geosesarma) nodulifera De Man, 1892 (Crustacea, Brachyura, Thoracotremata, Sesarmidae). Raffles Bull. Zool. 63, 3–13 (2015).

    Google Scholar 

  18. Green, P. T. Field observations of moulting and moult increment in the red land crab, Gecarcoidea natalis (Brachyura, Gecarcinidae), on Christmas Island (Indian Ocean). Crustaceana 77, 125–128 (2004).

    Article  Google Scholar 

  19. Nilsson, D. E. A new type of imaging optics in compound eyes. Nature 332, 76–78 (1988).

    Article  Google Scholar 

  20. Arikawa, K., Kawamata, K., Suzuki, T. & Eguchi, E. Daily changes of structure, function and rhodopsin content in the compound eye of the crab Hemigrapsus sanguineus. J. Comp. Physiol. A 161, 161–174 (1987).

    Article  Google Scholar 

  21. McLay, C. L. Brachyura and crab-like Anomura of New Zealand. Leigh Lab. Bull. 22, 463 (1988).

  22. Eguchi, E., Dezawa, M. & Meyer-Rochow, V. B. Compound eye fine structure in Paralomis multispina Benedict, an anomuran half-crab from 1200 m depth (Crustacea; Decapoda; Anomura). Biol. Bull. 192, 300–308 (1997).

    Article  Google Scholar 

  23. Luque, J. et al. Evolution of crab eye structures and the utility of ommatidia morphology in resolving phylogeny. Preprint at bioRxiv https://doi.org/10.1101/786087 (2019).

  24. Cai, J., Townsend, J. P., Dodson, T. C., Heiney, P. A. & Sweeney, A. M. Eye patches: protein assembly of index-gradient squid lenses. Science 357, 564–569 (2017).

    Article  Google Scholar 

  25. Alagboso, F. et al. Ultrastructure and mineral composition of the cornea cuticle in the compound eyes of a supralittoral and a marine isopod. J. Struct. Biol. 187, 158–173 (2014).

    Article  Google Scholar 

  26. Kim, K., Jang, K.-W., Ryu, J.-K. & Jeong, K.-H. Biologically inspired ultrathin arrayed camera for high-contrast and high-resolution imaging. Light. Sci. Appl. 9, 28 (2020).

    Article  Google Scholar 

  27. Lee, W. et al. Two-dimensional materials in functional three-dimensional architectures with applications in photodetection and imaging. Nat. Commun. 9, 1417 (2018).

    Article  Google Scholar 

  28. Sim, K. et al. Three-dimensional curvy electronics created using conformal additive stamp printing. Nat. Electron. 2, 471–479 (2019).

    Article  Google Scholar 

  29. Lee, W. et al. High-resolution spin-on-patterning of perovskite thin films for a multiplexed image sensor array. Adv. Mater. 29, 1702902 (2017).

    Article  Google Scholar 

  30. Sheng, X. et al. Silicon-based visible-blind ultraviolet detection and imaging using down-shifting luminophores. Adv. Opt. Mater. 2, 314–319 (2014).

    Article  Google Scholar 

  31. Kim, J. et al. Stretchable silicon nanoribbon electronics for skin prosthesis. Nat. Commun. 5, 5747 (2014).

    Article  Google Scholar 

  32. Song, E. et al. Flexible electronic/optoelectronic microsystems with scalable designs for chronic biointegration. Proc. Natl Acad. Sci. USA 116, 15398–15406 (2019).

    Article  Google Scholar 

  33. Hughes, C., Glavin, M., Jones, E. & Denny, P. Review of geometric distortion compensation in fish-eye cameras. In IET Signals and Systems Conference (ISSC 2008) 162–167 (IEEE, 2008).

  34. Liang, W.-L., Pan, J.-G. & Su, G.-D. J. One-lens camera using a biologically based artificial compound eye with multiple focal lengths. Optica 6, 326–334 (2019).

    Google Scholar 

  35. Brodrick, E. A., Roberts, N. W., Sumner-Ronney, L., Schleputz, C. M. & How, M. J. Light adaptation mechanisms in the eye of the fiddler crab Afruca tangeri. J. Comp. Neurol. 529, 616–634 (2021).

    Article  Google Scholar 

  36. Kim, H. M., Kim, M. S., Lee, G. J., Yoo, Y. J. & Song, Y. M. Large area fabrication of engineered microlens array with low sag height for light-field imaging. Opt. Express 27, 4435–4444 (2019).

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by IBS-R006-A1. This research was also supported by the National Research Foundation (NRF) of Korea (2017M3D1A1039288 and 2021R1A6A3A01087961) and the GIST-MIT Research Collaboration grant funded by the GIST in 2022.

Author information

Authors and Affiliations

Authors

Contributions

M.L., G.J.L., H.J.J., D.-H.K. and Y.M.S. designed the experiments, analysed the data and wrote the paper. M.L. and H.C. prepared the biological specimens. J.H.L. performed the electron microscopy experiments. M.L., H.C., E.J. and M.K. fabricated the photodiode array and performed the characterization of individual devices. M.L. and H.C. performed the assembly of devices. G.J.L., H.J.J., M.S.K., H.M.K. and J.-E.Y. performed the theoretical and experimental analysis of the optics. G.J.L., H.J.J., K.M.K. and F.D. conducted the imaging simulation and demonstration. H.J. and N.L. performed the theoretical analysis of the mechanics. All the authors discussed the results and commented on the manuscript.

Corresponding authors

Correspondence to Dae-Hyeong Kim or Young Min Song.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Electronics thanks Guo-Dung Su and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Hemispherical (convex type) cornea facet lenses in the eyes of terrestrial arthropods.

(a) Optical microscope images of a dragonfly eye (Epophthalmia elegans, left) and its SEM image (right). Its cornea facet lens has a convex-type hemispherical surface. (b) SEM images of a fly eye (Calliphora vomitori, left) and its magnified view (right), which show the convex-type hemispherical surface of the cornea facet lens.

Extended Data Fig. 2 Characterization of the fabricated g-ML arrays with various sizes.

(a–c), Each layer of each g-ML array is scanned by confocal microscopy to reconstruct the 3D surface profile of each layer. The diameters (D) of the g-MLs are 17 μm (a), 80 μm (b), and 400 μm (c). (d–f) Colorized cross-sectional SEM images of the g-MLs (top) and deviations of the RoCs from the average RoCs in the g-ML layers (bottom) for three different micro-lens diameters, that is, 17 µm (d), 80 µm (e), and 400 µm (f). (g) 2D cross-sectional profiles of g-MLs with three different diameters (17 µm, 80 µm, and 400 µm in the top, middle, and bottom frames, respectively).

Extended Data Fig. 3 Custom-made experimental setup for lens characterization in air and water, and measurement results.

(a) Optical camera image of the custom-made experimental setup. The setup consists of a white light source or laser sources with wavelengths of 450, 532, and 635 nm; an objective lens (10×); and a CMOS image sensor. (b–d) Optical camera images showing the components, including the water container, for amphibious imaging. (e) Measured light intensity distribution of the h-ML (micro-lens size of 400 μm) for three wavelengths (left: 450 nm, centre: 532 nm, and right: 635 nm). (f) Cross-sectional light intensity profile of the g-ML with a micro-lens size of 400 μm at the focal length in air (solid line) and water (dotted line). These results show that the g-ML obtains identical profiles in both media at the same focal length. (g) Imaging results of a letter ‘E’ obtained by the g-ML (top) and h-ML (bottom) for various micro-lens sizes (that is 17, 80, and 400 μm) under white light. (h) Measured focal lengths of g-MLs and h-MLs with three sizes in air and water. For all sizes, the g-MLs show consistent focal lengths in air and water, whereas the h-MLs show large changes of their focal lengths between air and water.

Extended Data Fig. 4 Schematic illustrations of multiple soft moulding processes for the fabrication of the g-ML array.

(a–c) Fabrication steps for a spacer layer and a base micro-lens array. (a) An anti-adhesive (Teflon AFTM) is coated on a quartz mould. (b) The prepolymer (clear flexTM 50) for the spacer and base micro-lenses is poured onto the quartz mould and thermally cured for 20 h. (c) The spacer and base micro-lens layer is detached from the quartz mould. (d–g) Fabrication steps for intermediate lens layers of the g-ML. For intermediate g-ML layers, silicone (EcoflexTM) moulds of plano-concave shapes with different curvatures are prepared. These steps are repeated for two intermediate NOA layers. (d) NOA is poured onto the silicone mould, and air bubbles are removed. (e) The silicone mould and base micro-lenses are aligned using a microscope. (f) The silicone mould and base micro-lenses are brought into contact. NOA is cured by UV illumination. (g) The silicone mould is removed, and unnecessary NOA is removed using tweezers. (h) The multi-layered NOA structure is formed on the base micro-lens. (i–l) Fabrication steps for a flat g-ML surface. (i) The silicone mould for the flat surface lens is prepared. NOA for the flat surface lens is poured onto the silicone mould. The multi-layered lens and silicone mould are aligned using a microscope. (j) The silicone mould and the multi-layered lens are brought into contact. NOA is cured using UV radiation. (k) Unnecessary NOA is removed using tweezers. (l) A g-ML with a flat surface is fabricated. (m, n) The screening pigment (SP) is spray-printed onto the g-ML and dried. (o, p) For light transmission, the SP on the g-ML surface is removed by bringing it into contact with solvent-coated (propylene glycol methyl ether acetate) glass.

Extended Data Fig. 5 Design of the panoramic artificial vision using the spherical 3D structure and finite element analysis (FEA) of the induced strain in the image sensor array.

(a) Schematic illustration of the panoramic artificial vision. The comb-shaped image sensor array and the g-ML array are integrated on the spherical 3D structure. (b) Schematic illustration defining the acceptance angle (∆θ) and inter-ommatidial angles in the vertical and horizontal directions (∆φv and ∆φh). Considering the acceptance angle, the inter-ommatidial angles are determined to avoid overlapping between the visual fields of neighbouring pixels. (c) Geometry of the g-ML used for FEA. (d) Elastic modulus (e) of each layer in the g-ML. (e) Strain distributions induced in the image sensor and g-ML arrays. Because the outermost lens has the highest elastic modulus, the entire g-ML experiences only a nominal strain under the bending deformation.

Extended Data Fig. 6 Imaging simulations for the optical parameters of the artificial vision.

(a) Imaging simulation results of a cubic pattern with eight different acceptance angles (∆θ) from 5.7° to 18°. The inset shows an original image of the cubic pattern. (b) Imaging simulation results of a cubic pattern with eight different inter-ommatidial angles (∆φ) from 1° to 15°. The dashed boxes indicate the used conditions in the measurement.

Extended Data Fig. 7 Optical simulations for inter-ommatidial and acceptance angles.

(a) Schematic for the distance-relative visual field of the ommatidia with an acceptance angle of ∆θ and an inter-ommatidial angle of ∆φ. (b) Illustration for the visual field variation by changing the acceptance and inter-ommatidial angles. An undetectable region, the area which cannot be captured by ommatidia, can be compensated by changing the angles. (c) Schematic illustration for describing the number of ommatidia capturing the object. (d) Calculated number of related ommatidia in (c) as a function of the acceptance angle and inter-ommatidial angle at an object distance of 5 mm, 40 mm, and 80 mm. (e) Original crab image for the image simulation. (f) Results of the imaging simulation with an object distance of 5 mm and an inter-ommatidial angle of 1°. (g) Results of the imaging simulation at an object distance of 40 mm with various acceptance angles and inter-ommatidial angles for the enhanced imaging resolution.

Extended Data Fig. 8 Simulation and experimental results of the panoramic imaging by the artificial vision.

(a) Schematic illustration of the artificial vision from the top view. (b) Simulation results of the panoramic imaging. (c) Experimental results of the panoramic imaging by the artificial vision.

Supplementary information

Supplementary Information

Supplementary Figs. 1–22, Tables 1–5 and Notes 1–7.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lee, M., Lee, G.J., Jang, H.J. et al. An amphibious artificial vision system with a panoramic visual field. Nat Electron 5, 452–459 (2022). https://doi.org/10.1038/s41928-022-00789-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41928-022-00789-9

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing