A central goal of artificial intelligence in high-stakes decision-making applications is to design a single algorithm that simultaneously expresses generalizability by learning coherent representations of their world and interpretable explanations of its dynamics. Here, we combine brain-inspired neural computation principles and scalable deep learning architectures to design compact neural controllers for task-specific compartments of a full-stack autonomous vehicle control system. We discover that a single algorithm with 19 control neurons, connecting 32 encapsulated input features to outputs by 253 synapses, learns to map high-dimensional inputs into steering commands. This system shows superior generalizability, interpretability and robustness compared with orders-of-magnitude larger black-box learning systems. The obtained neural agents enable high-fidelity autonomy for task-specific parts of a complex autonomous system.
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Rent or Buy article
Get time limited or full article access on ReadCube.
All prices are NET prices.
A description of how to obtain the data and code used for this manuscript is available at the manuscript’s GitHub repository: https://github.com/mlech26l/keras-ncp/ (https://doi.org/10.5281/zenodo.3999484). The data generated by the active test runs is available for download from the repository, while the full dataset of 193 GB is available on request from M.L.
Lecun, Y., Cosatto, E., Ben, J., Muller, U. & Flepp, B. Dave: Autonomous Off-road Vehicle Control Using End-to-end Learning Technical Report DARPA-IPTO Final Report (Courant Institute/CBLL, 2004); https://cs.nyu.edu/~yann/research/dave/
Bojarski, M. et al. End to end learning for self-driving cars. Preprint at http://arXiv.org/abs/1604.07316 (2016).
Kato, S. et al. Global brain dynamics embed the motor command sequence of Caenorhabditis elegans. Cell 163, 656–669 (2015).
Stephens, G. J., Johnson-Kerner, B., Bialek, W. & Ryu, W. S. Dimensionality and dynamics in the behavior of C. elegans. PLoS Comput. Biol. 4, e1000028 (2008).
Gray, J. M., Hill, J. J. & Bargmann, C. I. A circuit for navigation in caenorhabditis elegans. Proc. Natl Acad. Sci. USA 102, 3184–3191 (2005).
Yan, G. et al. Network control principles predict neuron function in the Caenorhabditis elegans connectome. Nature 550, 519–523 (2017).
Cook, S. J. et al. Whole-animal connectomes of both Caenorhabditis elegans sexes. Nature 571, 63–71 (2019).
Kaplan, H. S., Thula, O. S., Khoss, N. & Zimmer, M. Nested neuronal dynamics orchestrate a behavioral hierarchy across timescales. Neuron 105(3), 562–576 (2019).
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning. Nature 521, 436–444 (2015).
Hassabis, D., Kumaran, D., Summerfield, C. & Botvinick, M. Neuroscience-inspired artificial intelligence. Neuron 95, 245–258 (2017).
Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019).
Mnih, V. et al. Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015).
Silver, D. et al. Mastering the game of go with deep neural networks and tree search. Nature 529, 484–489 (2016).
Silver, D. et al. Mastering the game of go without human knowledge. Nature 550, 354–359 (2017).
Schrittwieser, J. et al. Mastering atari, go, chess and shogi by planning with a learned model. Preprint at http://arXiv.org/abs/1911.08265 (2019).
Vinyals, O. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019).
Bengio, Y., Courville, A. & Vincent, P. Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1798–1828 (2013).
Lipton, Z. C. The mythos of model interpretability. Queue 16, 31–57 (2018).
Lechner, M., Hasani, R., Rus, D. & Grosu, R. Gershgorin loss stabilizes the recurrent neural network compartment of an end-to-end robot learning scheme. In Proc. 2020 International Conference on Robotics and Automation (ICRA) 5446–5452 (2020).
Knight, J. C. Safety critical systems: challenges and directions. In Proc. 24th International Conference on Software Engineering 547–550 (2002).
Pearl, J. Causality (Cambridge Univ. Press, 2009).
Peters, J., Janzing, D. & Schölkopf, B. Elements of Causal Inference: Foundations and Learning Algorithms (MIT Press, 2017).
Joseph, M., Kearns, M., Morgenstern, J. H. & Roth, A. Fairness in learning: classic and contextual bandits. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 325–333 (2016).
Fish, B., Kun, J. & Lelkes, Á. D. A confidence-based approach for balancing fairness and accuracy. In Proc. SIAM International Conference on Data Mining 144–152 (2016).
Vaswani, A. et al. Attention is all you need. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 5998–6008 (2017).
Xu, H., Gao, Y., Yu, F. & Darrell, T. End-to-end learning of driving models from large-scale video datasets. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 2174–2182 (2017).
Amini, A., Paull, L., Balch, T., Karaman, S. & Rus, D. Learning steering bounds for parallel autonomous systems. In IEEE International Conference on Robotics and Automation (ICRA) 1–8 (2018).
Fridman, L. et al. MIT advanced vehicle technology study: large-scale naturalistic driving study of driver behavior and interaction with automation. IEEE Access 7, 102021–102038 (2019).
LeCun, Y. et al. Handwritten digit recognition with a back-propagation network. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 396–404 (1990).
Amini, A., Rosman, G., Karaman, S. & Rus, D. Variational end-to-end navigation and localization. In Proc. 2019 International Conference on Robotics and Automation (ICRA) 8958–8964 (2019).
Hochreiter, S. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Universität München 91 (1991).
Bengio, Y., Simard, P. & Frasconi, P. et al. Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5, 157–166 (1994).
Rumelhart, D. E., Hinton, G. E. & Williams, R. J. Learning representations by back-propagating errors. Nature 323, 533–536 (1986).
Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735–1780 (1997).
Reimer, B., Mehler, B., Wang, Y. & Coughlin, J. F. A field study on the impact of variations in short-term memory demands on drivers’ visual attention and driving performance across three age groups. Hum. Factors 54, 454–468 (2012).
Funahashi, K.-i & Nakamura, Y. Approximation of dynamical systems by continuous time recurrent neural networks. Neural Netw. 6, 801–806 (1993).
Chen, T. Q., Rubanova, Y., Bettencourt, J. & Duvenaud, D. K. Neural ordinary differential equations. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 6571–6583 (2018).
Lechner, M. & Hasani, R. Learning long-term dependencies in irregularly-sampled time series. Preprint at http://arXiv.org/abs/2006.04418 (2020).
Sarma, G. P. et al. Openworm: overview and recent advances in integrative biological simulation of Caenorhabditis elegans. Phil. Trans. R. Soc. B 373, 20170382 (2018).
Gleeson, P., Lung, D., Grosu, R., Hasani, R. & Larson, S. D. c302: a multiscale framework for modelling the nervous system of Caenorhabditis elegans. Phil. Trans. R. Soc. B. 373, 20170379 (2018).
Hasani, R., Lechner, M., Amini, A., Rus, D. & Grosu, R. Liquid time-constant networks. Preprint at http://arXiv.org/abs/2006.04439 (2020).
LeCun, Y. et al. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1, 541–551 (1989).
Wicks, S. R., Roehrig, C. J. & Rankin, C. H. A dynamic network simulation of the nematode tap withdrawal circuit: predictions concerning synaptic function using behavioral criteria. J. Neurosci. 16, 4017–4031 (1996).
Lechner, M., Hasani, R., Zimmer, M., Henzinger, T. A. & Grosu, R. Designing worm-inspired neural networks for interpretable robotic control. In International Conference on Robotics and Automation (ICRA) 87–94 (2019).
Hasani, R., Lechner, M., Amini, A., Rus, D. & Grosu, R. The natural lottery ticket winner: reinforcement learning with ordinary neural circuits. In Proc. International Conference on Machine Learning (2020).
Bengio, Y. & Grandvalet, Y. No unbiased estimator of the variance of k-fold cross-validation. J. Mach. Learn. Res. 5, 1089–1105 (2004).
Molnar, C. Interpretable Machine Learning (Lulu.com, 2019).
Hasani, R. Interpretable Recurrent Neural Networks in Continuous-time Control Environments. PhD dissertation, Technische Universität Wien (2020).
Erhan, D., Bengio, Y., Courville, A. & Vincent, P. Visualizing Higher-layer Features of a Deep Network Technical Report 1341 (Univ. Montreal, 2009).
Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In European Conference on Computer Vision 818–833 (2014).
Yosinski, J., Clune, J., Nguyen, A., Fuchs, T. & Lipson, H. Understanding neural networks through deep visualization. Preprint at http://arXiv.org/abs/1506.06579 (2015).
Karpathy, A., Johnson, J. & Fei-Fei, L. Visualizing and understanding recurrent networks. Preprint at http://arXiv.org/abs/1506.02078 (2015).
Strobelt, H., Gehrmann, S., Pfister, H. & Rush, A. M. LSTMVis: a tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE Trans. Vis. Comput Graph. 24, 667–676 (2018).
Bilal, A., Jourabloo, A., Ye, M., Liu, X. & Ren, L. Do convolutional neural networks learn class hierarchy? IEEE Trans. Vis. Comput. Graph. 24, 152–162 (2018).
Olah, C. et al. The building blocks of interpretability. Distill 3, e10 (2018).
Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. Preprint at http://arXiv.org/abs/1312.6034 (2013).
Fong, R. C. & Vedaldi, A. Interpretable explanations of black boxes by meaningful perturbation. Proc. IEEE International Conference on Computer Vision 3449–3457 (IEEE, 2017).
Kindermans, P.-J., Schütt, K. T., Alber, M., Müller, K.-R. & Dähne, S. Learning how to explain neural networks: PatternNet and PatternAttribution. Proc. International Conference on Learning Representations (ICLR) (2018).
Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. Proc. 34th International Conference on Machine Learning (ICML) (2017).
Doshi-Velez, F. & Kim, B. Towards a rigorous science of interpretable machine learning. Preprint at http://arXiv.org/abs/1702.08608 (2017).
Trask, A. et al. Neural arithmetic logic units. In Proc. Advances in Neural Information Processing Systems (NeurIPS) 8035–8044 (2018).
Bojarski, M. et al. Visualbackprop: efficient visualization of cnns for autonomous driving. In IEEE International Conference on Robotics and Automation (ICRA) 1–8 (2018).
Maaten, Lvd & Hinton, G. Visualizing data using t-sne. J. Mach. Learn. Res. 9, 2579–2605 (2008).
Tesla Autopilot (Tesla, 2020); https://www.tesla.com/autopilot
Karpathy, A. PyTorch at Tesla. In PyTorch Devcon Conference 19 https://youtu.be/oBklltKXtDE (2019).
Press, W. H., Teukolsky, S. A., Vetterling, W. T. & Flannery, B. P. Numerical Recipes: The Art of Scientific Computing 3rd edn (Cambridge Univ. Press, 2007).
Naser, F. et al. A parallel autonomy research platform. In 2017 IEEE Intelligent Vehicles Symposium (IV) 933–940 (IEEE, 2017).
Amini, A. et al. Learning robust control policies for end-to-end autonomous driving from data-driven simulation. IEEE Robot. Autom. Lett. 5, 1143–1150 (2020).
Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. In Proc. 3rd International Conference for Learning Representations (ICLR) (2015).
Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. et al. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).
Girosi, F., Jones, M. & Poggio, T. Regularization theory and neural networks architectures. Neural Comput. 7, 219–269 (1995).
Smale, S. & Zhou, D.-X. Learning theory estimates via integral operators and their approximations. Constr. Approx. 26, 153–172 (2007).
We thank M. Zimmer and the Zimmer Group for constructive discussions. R.H. and R.G. are partially supported by Horizon-2020 ECSEL Project grant no. 783163 (iDev40), and the Austrian Research Promotion Agency (FFG), project no. 860424. M.L. and T.A.H. were supported in part by the Austrian Science Fund (FWF) under grant Z211-N23 (Wittgenstein Award). A.A. is supported by the National Science Foundation (NSF) Graduate Research Fellowship Program. A.A. and D.R. were partially sponsored by the United States Air Force Research Laboratory and was accomplished under Cooperative Agreement no. FA8750-19-2-1000. R.H. and D.R. are partially supported by The Boeing Company. This research work is drawn from the PhD dissertation of R.H.
The authors declare no competing interests.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
a, Hidden state dynamics of 64 LSTM cells as a function of network output. b, Hidden state dynamics of 13 NCP cells as a function of the network output. c, PCA on LSTM cells + output, d, PCA on LSTM cells only. e, PCA on NCP cells + output. f, PCA on NCP cells only. x-axis represents the activity of the output, y-axis stands for the dynamics of an individual neuron. The colour represents the steering angle (The more yellow regions depict sharper turns to the left-hand-side, and the more blue regions stand for sharper turns to the right-hand-side).
The colour-bar represents the neuron output of each individual neuron in the NCP architecture.
The colour-bar represents the time constant of each individual neuron in the NCP architecture.
Size of the convolutional kernels.
Conv2D refers to a convolutional layer, F to the number of filters, K to the kernel size, S to the strides, U to the number of units in a fully-connected layer. The values of the dropout-rates δ1,δ2, and δ3 were optimised on the passive benchmark and reported in Extended Data Figure 3.
The values of all hyperparameters were selected through empirical evaluation over the passive training dataset. We did not search through the hyperparameters space exhaustively, due to the computational costs. However, the use of a systematic meta-learning algorithm over these parameter-spaces can presumably result in achieving better performances.
Training and validation metrics of the models tested in the active driving experiment. As discussed thoroughly (Fig. 4), LSTM model achieves the best performance in the passive test but fails to express proper driving behaviour under environmental disturbances.
NCP—driving performance with no perturbation.
CNN—driving performance with no perturbation.
LSTM—driving performance with no perturbation.
CT-RNN—driving performance with no perturbation.
NCP—driving performance with perturbation variance = 0.1.
CNN—driving performance with perturbation variance = 0.1.
LSTM—driving performance with perturbation variance = 0.1.
CT-RNN—driving performance with perturbation variance = 0.1.
NCP—driving performance with perturbation variance = 0.2.
CNN—driving performance with perturbation variance = 0.2.
LSTM—driving performance with perturbation variance = 0.2.
CT-RNN—driving performance with perturbation variance = 0.2.
NCP—driving performance with perturbation variance = 0.3.
CNN—driving performance with perturbation variance = 0.3.
LSTM—driving performance with perturbations variance = 0.3.
CT-RNN—driving performance with perturbation variance = 0.3.
About this article
Cite this article
Lechner, M., Hasani, R., Amini, A. et al. Neural circuit policies enabling auditable autonomy. Nat Mach Intell 2, 642–652 (2020). https://doi.org/10.1038/s42256-020-00237-3
npj Computational Materials (2021)
Nature Machine Intelligence (2020)