Actor neural networks for the robust control of partially measured nonlinear systems showcased for image propagation through diffuse media

Abstract

The output of physical systems, such as the scrambled pattern formed by shining the spot of a laser pointer through fog, is often easily accessible by direct measurements. However, selection of the input of such a system to obtain a desired output is difficult, because it is an ill-posed problem; that is, there are multiple inputs yielding the same output. Information transmission through scattering media is an example of this problem. Machine learning approaches for imaging have been implemented very successfully in photonics to recover the original input phase and amplitude objects of the scattering system from the distorted intensity diffraction pattern outputs. However, controlling the output of such a system, without having examples of inputs that can produce outputs in the class of the output objects the user wants to produce, is a challenging problem. Here, we propose an online learning approach for the projection of arbitrary shapes through a multimode fibre when a sample of intensity-only measurements is taken at the output. This projection system is nonlinear, because the intensity, not the complex amplitude, is detected. We show an image projection fidelity as high as ~90%, which is on par with the gold-standard methods that characterize the system fully by phase and amplitude measurements. The generality and simplicity of the proposed approach could potentially provide a new way of target-oriented control in real-world applications when only partial measurements are available.

A preprint version of the article is available at ArXiv.

Access options

Rent or Buy article

Get time limited or full article access on ReadCube.

from$8.99

All prices are NET prices.

Fig. 1: Fibre projector.
Fig. 2: Neural network architecture and training procedure.
Fig. 3: MMF system arbitrary output control with partial measurements.
Fig. 4: Neural network generalization ability for controlling the MMF output.
Fig. 5: Continuous grey-scale image projection.

Data availability

The directory to the dataset required to reproduce results appearing in Table 1 is provided at https://github.com/Babak70/Projector_network.

Code availability

Our neural network framework is available at https://github.com/Babak70/Projector_network and https://zenodo.org/record/3727136. Any correspondence and reasonable requests for extra materials should be directed to the corresponding author.

References

  1. 1.

    Spitz, E. & Werts, A. Transmission des images à travers une fibre optique. C. R. Hebd. Des. Seances De. L. Acad. Des. Sci. Ser. B 264, 1015 (1967).

    Google Scholar 

  2. 2.

    Di Leonardo, R. & Bianchi, S. Hologram transmission through multi-mode optical fibers. Opt. Express 19, 247–254 (2011).

    Article  Google Scholar 

  3. 3.

    Čižmár, T. & Dholakia, K. Shaping the light transmission through a multimode optical fibre: complex transformation analysis and applications in biophotonics. Opt. Express 19, 18871–18884 (2011).

    Article  Google Scholar 

  4. 4.

    Čižmár, T. & Dholakia, K. Exploiting multimode waveguides for pure fibre-based imaging. Nat. Commun. 3, 1027 (2012).

    Article  Google Scholar 

  5. 5.

    Bianchi, S. & Di Leonardo, R. A multi-mode fiber probe for holographic micromanipulation and microscopy. Lab Chip 12, 635–639 (2012).

    Article  Google Scholar 

  6. 6.

    Andresen, E. R., Bouwmans, G., Monneret, S. & Rigneault, H. Toward endoscopes with no distal optics: video-rate scanning microscopy through a fiber bundle. Opt. Lett. 38, 609–611 (2013).

    Article  Google Scholar 

  7. 7.

    Gover, A., Lee, C. P. & Yariv, A. Direct transmission of pictorial information in multimode optical fibers. J. Opt. Soc. Am. 66, 306–311 (1976).

    Article  Google Scholar 

  8. 8.

    Friesem, A. A., Levy, U. & Silberberg, Y. Parallel transmission of images through single optical fibers. Proc. IEEE 71, 208–221 (1983).

    Article  Google Scholar 

  9. 9.

    Yariv, A., AuYeung, J., Fekete, D. & Pepper, D. M. Image phase compensation and real‐time holography by four‐wave mixing in optical fibers. Appl. Phys. Lett. 32, 635–637 (1978).

    Article  Google Scholar 

  10. 10.

    Yamaguchi, I. & Zhang, T. Phase-shifting digital holography. Opt. Lett. 22, 1268–1270 (1997).

    Article  Google Scholar 

  11. 11.

    Cuche, E., Bevilacqua, F. & Depeursinge, C. Digital holography for quantitative phase-contrast imaging. Opt. Lett. 24, 291–293 (1999).

    Article  Google Scholar 

  12. 12.

    Papadopoulos, I. N., Farahi, S., Moser, C. & Psaltis, D. Focusing and scanning light through a multimode optical fiber using digital phase conjugation. Opt. Express 20, 10583–10590 (2012).

    Article  Google Scholar 

  13. 13.

    Papadopoulos, I. N., Farahi, S., Moser, C. & Psaltis, D. High-resolution, lensless endoscope based on digital scanning through a multimode optical fiber. Biomed. Opt. Express 4, 260–270 (2013).

    Article  Google Scholar 

  14. 14.

    Choi, Y. et al. Scanner-free and wide-field endoscopic imaging by using a single multimode optical fiber. Phys. Rev. Lett. 109, 203901 (2012).

    Article  Google Scholar 

  15. 15.

    Caravaca-Aguirre, A. M., Niv, E., Conkey, D. B. & Piestun, R. Real-time resilient focusing through a bending multimode fiber. Opt. Express 21, 12881–12887 (2013).

    Article  Google Scholar 

  16. 16.

    Gu, R. Y., Mahalati, R. N. & Kahn, J. M. Design of flexible multi-mode fiber endoscope. Opt. Express 23, 26905–26918 (2015).

    Article  Google Scholar 

  17. 17.

    Loterie, D. et al. Digital confocal microscopy through a multimode fiber. Opt. Express 23, 23845–23858 (2015).

    Article  Google Scholar 

  18. 18.

    Popoff, S., Lerosey, G., Fink, M., Boccara, A. C. & Gigan, S. Image transmission through an opaque material. Nat. Commun. 1, 81 (2010).

    Article  Google Scholar 

  19. 19.

    LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).

    Article  Google Scholar 

  20. 20.

    McCann, M. T., Jin, K. H. & Unser, M. Convolutional neural networks for inverse problems in imaging: a review. IEEE Signal Process Mag. 34, 85–95 (2017).

    Article  Google Scholar 

  21. 21.

    Rivenson, Y. et al. Deep learning microscopy. Optica 4, 1437–1443 (2017).

    Article  Google Scholar 

  22. 22.

    Molesky, S. et al. Inverse design in nanophotonics. Nat. Photon. 12, 659–670 (2018).

    Article  Google Scholar 

  23. 23.

    Paruzzo, F. M. et al. Chemical shifts in molecular solids by machine learning. Nat. Commun. 9, 4501 (2018).

    Article  Google Scholar 

  24. 24.

    Rahmani, B., Loterie, D., Konstantinou, G., Psaltis, D. & Moser, C. Multimode optical fiber transmission with a deep learning network. Light Sci. Appl. 7, 69 (2018).

    Article  Google Scholar 

  25. 25.

    Borhani, N., Kakkava, E., Moser, C. & Psaltis, D. Learning to see through multimode fibers. Optica 5, 960–966 (2018).

    Article  Google Scholar 

  26. 26.

    Caramazza, P., Moran, O., Murray-Smith, R. & Faccio, D. Transmission of natural scene images through a multimode fibre. Nat. Commun. 10, 2029 (2019).

    Article  Google Scholar 

  27. 27.

    Li, Y., Xue, Y. & Tian, L. Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media. Optica 5, 1181–1190 (2018).

    Article  Google Scholar 

  28. 28.

    Li, S., Deng, M., Lee, J., Sinha, A. & Barbastathis, G. Imaging through glass diffusers using densely connected convolutional networks. Optica 5, 803–813 (2018).

    Article  Google Scholar 

  29. 29.

    Turpin, A., Vishniakou, I. & d Seelig, J. Light scattering control in transmission and reflection with neural networks. Opt. Express 26, 30911–30929 (2018).

    Article  Google Scholar 

  30. 30.

    Psaltis, D., Sideris, A. & Yamamura, A. A. A multilayered neural network controller. IEEE Control Syst. Mag. 8, 17–21 (1988).

    Article  Google Scholar 

  31. 31.

    Xu, Z., Yang, P., Hu, K., Xu, B. & Li, H. Deep learning control model for adaptive optics systems. Appl. Opt. 58, 1998–2009 (2019).

    Article  Google Scholar 

  32. 32.

    Sutton, R. S. & Barto, A. G. Reinforcement Learning: An Introduction (MIT Press, 2011).

  33. 33.

    Cohen, G., Afshar, S., Tapson, J. & van Schaik, A. EMNIST: an extension of MNIST to handwritten letters. Preprint at https://arxiv.org/pdf/1702.05373.pdf (2017).

  34. 34.

    Närhi, M. et al. Machine learning analysis of extreme events in optical fibre modulation instability. Nat. Commun. 9, 4923 (2018).

    Article  Google Scholar 

  35. 35.

    Xiong, W. et al. Deep learning of ultrafast pulses with a multimode fiber. Preprint at https://arxiv.org/pdf/1911.00649.pdf (2019).

  36. 36.

    Salmela, L., Lapre, C., Dudley, J. M. & Genty, G. Machine learning analysis of rogue solitons in supercontinuum generation. Sci. Rep. 10, 9596 (2020).

    Article  Google Scholar 

  37. 37.

    Teğin, U. et al. Controlling spatiotemporal nonlinearities in multimode fibers with deep neural networks. APL Photonics 5, 030804 (2020).

    Article  Google Scholar 

  38. 38.

    Davis, J. A., Cottrell, D. M., Campos, J., Yzuel, M. J. & Moreno, I. Encoding amplitude information onto phase-only filters. Appl. Opt. 38, 5004–5013 (1999).

    Article  Google Scholar 

Download references

Acknowledgements

C.M. and D.L. acknowledge financial support from Gebert Rüf Stiftung via the grant ‘Flexprint’ (GRS-057/18, Pilot Projects track).

Author information

Affiliations

Authors

Contributions

B.R. performed the experiment, analysed the data and wrote the manuscript. D.L. wrote the code for the transmission matrix measurements. E.K., N.B. and U.T. participated in the analysis and in the SLM modulation format. D.P. and C.M. proposed and supervised the project and contributed to writing the manuscript.

Corresponding author

Correspondence to Babak Rahmani.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 System arbitrary output control with partial measurements.

Additional examples as in Fig. 3.

Extended Data Fig. 2 Fidelity trajectory of the system’s output.

a, the fidelity trajectory of experimentally projected images versus the training iteration number is plotted for all three colors. b, while training, the instability of the system (estimated as the correlation between instances of the system’s response to a constant input signal being sent through the system over and over) is monitored over time (If the system is time-invariant, then the decorrelation plot holds a value of one continually). c, degradation in the fidelity of projected images due to the non-perfect modulation scheme as well as the variation of the system with time is shown by using the experimentally measured transmission matrix (TM) to forward the neural network’s predicted SLM images for all three colors. The fidelities in part (a) are redrawn in part (c) for comparison. As observed, the experimentally projected images (solid circles) closely follow the track of time variant TM-based relayed projections (dashed lines) and both eventually fall below the track of time-invariant TM-based relayed projections (solid lines). In the former, what is taken out from the learning algorithm is only the effect of modulation scheme, whereas in the latter, it is the lumped effect of time variation as well as the modulation scheme. The ripples in the trajectory of the graphs in (c) (dashed lines) show that the network is continuously trying to correct for the drifts.

Extended Data Fig. 3 Projection via amplitude-only control patterns.

a, examples of images projected onto a camera at the output of a MMF (wavelength 780 nm) are shown. The network is forced to generate amplitude-only control patterns. These patterns are then sent to the system and the outputs on the camera are captured. The network is trained with target images of Latin characters but it is also used to predict control patterns for target images from different categories. The visible background of the projected images accounts for the lower signal to noise ratio of the images (also lower fidelities) as compared with that of the complex value control patterns. This is attributed to missing out on controlling the phase of control signals. b, plot of the convergence speed for amplitude-only input controls.

Extended Data Fig. 4 Optical setup.

Detailed diagram of the optical setup. Control patterns are generated via the SLM, guided through the fiber and captured by the camera. L1: Aspheric lens, L2: f = 100 mm lens; L3: f = 250 mm lens; L4: f = 250 mm lens; OBJ1, OBJ2: 60x microscope objective; SLM: spatial light modulator; M1: mirror; FM: flip mirror; SMF: single mode fiber; MMF: multimode fiber, BS: beam splitter.

Extended Data Fig. 5 Neural network’s architecture.

The Actor and Model networks are comprised of two sub-networks, (Areal, Aimag) and (Mreal, Mimag) respectively, to cope with the real and imaginary parts of input-output fields. All sub-networks are fully-connected. Thereby, the input images to the Actor network is first flattened out (from size 200×200 pixels to 40000×1 vectors) and then fed to the sub-networks Areal and Aimag (input nodes 40000, output nodes 2601). In the training step, the output vectors (size 2601×1) of Areal and Aimag are passed on to Mreal and Mimag (input nodes 2601, output nodes 40000), respectively. The virtual neural network output image of the target image (size 200×200) is then produced at the output of the Model. Once trained, the output vectors of the Actor network can be directly reshaped to produce the real and imaginary parts of the SLM images (reshaping from size 2601×1 to 51×51 pixels). If these SLM images are uploaded to the SLM and sent through the fiber, they produce projected images on the camera that are similar to the target images.

Extended Data Fig. 6 Semi-supervised learning.

The Actor network is additionally trained with images used for training of the Model network. The loss function for training the Actor is comprised of two terms: the loss term in Equation. 3 and an extra mean square term with weight λ The fidelity is obtained for multiple values of λ. The original learning algorithm corresponds to the case in which λ=0.

Extended Data Fig. 7 Robustness analysis.

Two scenarios of minor (a, b) and major (c, d) perturbations due to misalignment of the MMF system is studied. a- The proximal side of the fiber has been slightly blocked with a spatial filter depicted in (a). b- the fidelity plot of the projected images before and after the perturbation event (denoted by a star symbol) is shown. c- In the second scenario, the proximal side of the fiber has been slightly tilted (changed in angle). This results in a shift in the reception bandwidth of the fiber facet from spatial frequency configuration 1 to 2. d- The fidelity plot of the projected images for the perturbed system is shown.

Extended Data Fig. 8 Supervised learning architecture for amplitude-only modulation.

a- Schematic of a neural network (NN) that is made of one single network as an alternative architecture for the Actor–Model network proposed in the main text. The single NN is trained with images obtained from the camera and the corresponding input control patterns (here amplitude-only control patterns). Once trained, the target image is directly passed on to the network and is asked for its corresponding control pattern. The predicted pattern is then sent through the fiber. b- The average fidelity trajectory of the projected images of 20000 Latin alphabet characters versus the training iteration number.

Supplementary information

Supplementary Video

The video accompanying the manuscript containing animations of several projected images captured on the camera.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rahmani, B., Loterie, D., Kakkava, E. et al. Actor neural networks for the robust control of partially measured nonlinear systems showcased for image propagation through diffuse media. Nat Mach Intell 2, 403–410 (2020). https://doi.org/10.1038/s42256-020-0199-9

Download citation

Search

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing