Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Evolutionary autonomous agents: A neuroscience perspective

Nature Reviews Neuroscience volume 3, pages 132–141 (2002)Cite this article

Key Points

  • The development of evolutionary autonomous agents (EAA) harnesses genetic algorithms and genetic programming to evolve neurally controlled autonomous agents that perform biologically oriented tasks, such as navigation and foraging. It has significant potential as a new research tool for computational neuroscience.

  • The promise of EAA studies stems from a few important factors. The neuro-controller 'brains' that govern the agents' behaviour are emergent phenomena in a biologically relevant setting. As such, their architecture and organization are not a biased reflection of our current views and models of neural information processing. Their small size and the relative simplicity of the environments in which they evolve, coupled with the availability of full information on both the agents and their environment, make these networks amenable to functional and structural analysis.

  • The nascent field of EAA studies in neuroscience modelling still suffers from a few serious limitations, among them the inadequacy of the existing EAA methodology to generate controller network solutions to many complex behavioural tasks, and the very restricted sensors and motor models that are used in current EAA applications.

  • EAA studies have already made interesting contributions to computational-neuroscience research. These include studies that have evolved biological circuits such as central pattern generators and memory-dependent command neurons that modulate behaviour. EAAs are essential for theoretically studying the complex interactions between evolution and learning, and the principles of genetic encoding and development of neural circuits. Last, but not least, EAA models are an important conceptual vehicle for finding new ways of characterizing neural information processing and representation, and for quantitatively estimating these properties.

  • The combination of the results reviewed here, the clear challenges that await in the near future, and the continuing fast growth of computing resources, support the idea that EAAs represent a promising way of making neuroscience modelling as simple as possible, without simplifying it so much that it ceases to be useful. “As simple as possible” might turn out to be very complex, but EAA studies are still one of our best bets.

Abstract

In this article, I discuss the use of neurally driven evolutionary autonomous agents (EAAs) in neuroscientific investigations. Two fundamental questions are addressed. Can EAA studies shed new light on the structure and function of biological nervous systems? And can these studies lead to the development of new tools for neuroscientific analysis? The value and significant potential of EAA modelling in both respects is demonstrated and discussed. Although the study of EAAs for neuroscience research still faces difficult conceptual and technical challenges, it is a promising and timely endeavour.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1
Figure 2: An outline of the food zone (southwest corner of the grid arena) and the agent's controlling network.
Figure 3: A typical evolutionary run.
Figure 4: Evolution of cortical circuits.
Figure 5: Contributions of neurons (units) to tasks.

Similar content being viewed by others

References

  1. Mitchell, M. An Introduction to Genetic Algorithms (MIT Press, Cambridge, Massachusetts, 1996).

    Google Scholar 

  2. Langton, C. Artificial Life: an Overview (MIT Press, Boston, Massachusetts, 1997).

    Google Scholar 

  3. Fogel, D. B. Evolutionary Computation — Toward a New Philosophy of Machine Intelligence (IEEE Press, Piscataway, New Jersey, 1999).

    Google Scholar 

  4. Adami, C. Introduction to Artificial Life (Springer, New York, 1998).

    Book  Google Scholar 

  5. Aharonov-Barki, R., Beker, T. & Ruppin, E. Emergence of memory-driven command neurons in evolved artificial agents. Neural Comput. 13, 691–716 (2001).This study was one of the first to evolve general, recurrent controlling networks, manifesting the spontaneous emergence of biological-like command neurons that switch between distinct behavioural modes.

    Article  CAS  Google Scholar 

  6. Meyer, J. A. & Guillot, A. From SAB90 to SAB94: four years of animat research. Proc. 3rd Int. Conf. Simul. Adaptive Behav. (eds Cliff, D., Husbands, P., Meyer, J. A. & Wilson, S. W.) 2–11 (MIT Press, Cambridge, Massachusetts, 1994).

    Google Scholar 

  7. Kodjabachian, J. & Meyer, J. A. Evolution and development of neural controllers for locomotion, gradient-following and obstacle-avoidance in artificial insects. IEEE Trans. Neural Netw. 9, 796–812 (1998).

    Article  CAS  Google Scholar 

  8. Yao, X. Evolving artificial neural networks. Proc. IEEE 87, 1423–1447 (1999).

    Article  Google Scholar 

  9. Guillot, A. & Meyer, J. A. From SAB94 to SAB2000: what's new, animat? Proc. 6th Int. Conf. Simul. Adaptive Behav. (eds Meyer, J. A., Berthoz, A., Floreano, D., Roitblat, H. L. & Wilson, S. W.) 2–11 (MIT Press, Cambridge, Massachusetts, 2000).

    Google Scholar 

  10. Guillot, A. & Meyer, J. A. The animat contribution to cognitive systems research. J. Cogn. Syst. Res. 2, 157–165 (2001).

    Article  Google Scholar 

  11. Cangelosi, A. & Parisi, D. A neural network model of Caenorhabditis elegans : the circuit of touch sensitivity. Neural Process. Lett. 6, 91–98 (1997).

    Article  Google Scholar 

  12. Ijspeert, A. J., Hallam, J. & Willshaw, D. Evolving swimming controllers for a simulated lamprey with inspiration from neurobiology. Adaptive Behav. 7, 151–172 (1999).This paper describes an elaborate study that evolved swimming controllers, showing the power of EAAs in generating biologically relevant network models that outperform existing handcrafted ones.

    Article  Google Scholar 

  13. Ekeberg, O. A combined neuronal and mechanical model of fish swimming. Biol. Cybern. 69, 363–374 (1993).

    Article  Google Scholar 

  14. Jung, R., Kimmel, T. & Cohen, A. H. Dynamical behavior of a neural network model of locomotor control in the lamprey. J. Neurophysiol. 75, 1074–1086 (1996).

    Article  CAS  Google Scholar 

  15. Edwards, D. H., Heitler, W. J. & Krasne, F. B. Fifty years of command neurons: the neurobiology of escape behavior in the crayfish. Trends Neurosci. 22, 153–161 (1999).

    Article  CAS  Google Scholar 

  16. Xin, Y., Weiss, K. R. & Kupfermann, I. A pair of identified interneurons in Aplysia that are involved in multiple behaviors are necessary and sufficient for the arterial-shortening component of a local withdrawal reflex. J. Neurosci. 16, 4518–4528 (1996).

    Article  CAS  Google Scholar 

  17. Nagahama, T., Weiss, K. & Kupfermann, I. Body postural muscles active during food arousal in Aplysia are modulated by diverse neurons that receive monosynaptic excitation from the neuron CPR. J. Neurophysiol. 72, 314–325 (1994).

    Article  CAS  Google Scholar 

  18. Teyke, T., Weiss, K. & Kupfermann, I. An identified neuron (CPR) evokes neuronal responses reflecting food arousal in Aplysia. Science 247, 85–87 (1990).

    Article  CAS  Google Scholar 

  19. Panchin, Y. V. et al. Control of locomotion in the marine mollusc Clione limacina. XI. Effects of serotonin. Exp. Brain Res. 109, 361–365 (1996).

    Article  CAS  Google Scholar 

  20. Norris, B. J., Coleman, M. J. & Nusbaum, M. P. Recruitment of a projection neuron determines gastric mill motor pattern selection in the stomatogastric nervous system of the crab, Cancer borealis. J. Neurophysiol. 72, 1451–1463 (1994).

    Article  CAS  Google Scholar 

  21. DiCaprio, R. A. An interneurone mediating motor programme switching in the ventilatory system of the crab. J. Exp. Biol. 154, 517–535 (1990).

    CAS  Google Scholar 

  22. Combes, D., Meyrand, P. & Simmers, J. Motor pattern specification by dual descending pathways to a lobster rhythm-generating network. J. Neurosci. 19, 3610–3619 (1999).

    Article  CAS  Google Scholar 

  23. Brisson, M. T. & Simmers, J. Neuromodulatory inputs maintain expression of a lobster motor pattern generating network in a modulation-dependent state: evidence from long-term decentralization in vitro. J. Neurosci. 18, 2212–2225 (1998).

    Article  Google Scholar 

  24. Husbands, P., Smith, T., Jacobi, N. & Oshea, M. Better living through chemistry: evolving GasNets for robot control. Connection Sci. 10, 185–210 (1998).

    Article  Google Scholar 

  25. Ishiguro, A. Evolving an adaptive controller for a legged robot with dynamically-rearranging neural networks. Proc. 6th Int. Conf. Simul. Adaptive Behav. (eds Meyer, J. A., Berthoz, A., Floreano, D., Roitblat, H. L. & Wilson, S. W.) (MIT Press, Cambridge, Massachusetts, 2000).

    Google Scholar 

  26. Beer, R. D., Chiel, H. J. & Gallagher, J. C. Evolution and analysis of model CPGs for walking. II. General principles and individual variability. J. Comput. Neurosci. 7, 119–147 (1999).This paper and reference 48 show the use of EAA models as a tool for generating and analysing simple and tractable dynamic network systems, introducing the concept of dynamical modules as a computational analogue of Getting's classical dynamically varying modules.

    Article  CAS  Google Scholar 

  27. Scheier, C., Pfiefer, R. & Kunyioshi, Y. Embedded neural networks: exploiting constraints. Neural Netw. 7–8, 1551–1569 (1998).This paper describes a set of experiments that show the importance and significance of studying neural-network controllers of agents embodied in their environment.

    Article  Google Scholar 

  28. Beer, R. D. Dynamical approaches to cognitive science. Trends Cogn. Sci. 4, 91–99 (2000).

    Article  CAS  Google Scholar 

  29. Nolfi, S. & Floreano, D. Learning and evolution. Auton. Robots 7, 89–113 (1999).

    Article  Google Scholar 

  30. Hinton, G. E. & Nowlan, S. How learning can guide evolution. Complex Syst. 1, 495–502 (1987).

    Google Scholar 

  31. Miller, G. F. & Todd, P. Exploring adaptive agency. I. Theory and methods for simulating the evolution of learning. Proc. 1990 Connectionist Models Summer School (eds Touretzky, D. S., Elman, J. L., Sejnowski, T. J. & Hinton, G. E.) 65–80 (Morgan Kaufmann, San Mateo, California, 1990).

    Google Scholar 

  32. Nolfi, S. & Parisi, D. Learning to adapt to environments in evolving neural networks. Adaptive Behav. 5, 75–98 (1997).

    Article  Google Scholar 

  33. Churchland, P. S. & Sejnowski, T. J. The Computational Brain (MIT Press, Boston, Massachusetts, 1989).

    Google Scholar 

  34. Niv, Y., Joel, D., Meilijson, I. & Ruppin, E. Evolution of reinforcement learning in foraging bees in neural terms. 10th Annu. Comput. Neurosci. Meet. CNS2001 (Monterey, California, 2001). Neurocomputing (in the press).

  35. Montague, P. R., Dayan, P., Person, C. & Sejnowski, T. J. Bee foraging in uncertain environments using predictive Hebbian learning. Nature 377, 725–728 (1995).

    Article  CAS  Google Scholar 

  36. Schacher, S., Wu, F. & Sun, Z.-Y. Pathway-specific synaptic plasticity: activity-dependent enhancement and suppression of long-term heterosynaptic facilitation at converging inputs on a single target. J. Neurosci. 17, 597–606 (1997).

    Article  CAS  Google Scholar 

  37. Vogt, K. E. & Nicoll, R. E. Glutamate and γ-aminobutyric acid mediate a heterosynaptic depression at mossy fiber synapses in the hippocampus. Proc. Natl Acad. Sci. USA 96, 1118–1122 (1999).

    Article  CAS  Google Scholar 

  38. Fisken, R., Garey, L. & Powell, T. Patterns of degeneration after intrinsic lesions of the visual cortex of the monkey. Brain Res. 53, 208–213 (1973).

    Article  CAS  Google Scholar 

  39. Hess, R., Negishi, K. & Creutzfeldt, O. The horizontal spread of intracortical inhibition in visual cortex. Exp. Brain Res. 22, 415–419 (1975).

    Article  Google Scholar 

  40. Douglas, R. & Martin, K. in The Synaptic Organization of the Brain (ed. Shepherd, G. M.) 389–438 (Oxford Univ. Press, Oxford, 1997).

  41. Reggia, J., Autrechy, L., Sutton, G. & Weinrich, M. A competitive distribution theory of neocortical dynamics. Neural Comput. 4, 287–317 (1992).

    Article  Google Scholar 

  42. Mountcastle, V. Perceptual Neuroscience: Cerebral Cortex (Harvard Univ Press, Boston, Massachusetts, 1998).

    Google Scholar 

  43. Ayers, D. & Reggia, J. A. Evolving columnar circuitry for lateral cortical inhibition. Proc. INNS–IEEE Int. Joint Conf. Neural Netw. 278–283 (IEEE Press, Washington DC, 2001).Going beyond EAAs, this study shows the power of evolutionary computation to search for candidate cortical columnar architectures that might best subserve observed patterns of cortical activity.

    Google Scholar 

  44. Rolls, E. T. & Stringer, S. M. On the design of neural networks in the brain by genetic algorithms. Prog. Neurobiol. 61, 557–579 (2000).

    Article  CAS  Google Scholar 

  45. Dellaert, F. & Beer, R. D. Toward an evolvable model of development for autonomous agent synthesis. Proc. 4th Conf. Artif. Life (eds Brooks. R. & Maes, P.) 246–257 (MIT Press, Cambridge, Massachusetts, 1994).

    Google Scholar 

  46. Floreano, D. & Urzelai, J. Neural morphogenesis, synaptic plasticity and evolution. Theory Biosci. 120, 223–238 (2001).

    Article  Google Scholar 

  47. Cliff, D. & Miller, G. F. Co-evolution of pursuit and evasion. II: Simulation methods and results. Proc. 4th Int. Conf. Simul. Adaptive Behav. (eds Maes, P. et al.) 506–515 (MIT Press, Cambridge, Massachusetts, 1996).

    Google Scholar 

  48. Chiel, H. J., Beer, R. D. & Gallagher, J. C. Evolution and analysis of model CPGs for walking. I. Dynamical modules. J. Comput. Neurosci. 7, 99–118 (1999).

    Article  CAS  Google Scholar 

  49. Getting, P. Emerging principles governing the operations of neural networks. Annu. Rev. Neurosci. 12, 185–204 (1989).

    Article  CAS  Google Scholar 

  50. Marder, E. & Calabrese, R. L. Principles of rhythmic motor pattern generation. Physiol. Rev. 76, 687–717 (1996).

    Article  CAS  Google Scholar 

  51. Marder, E. & Abbott, L. F. Theory in motion. Curr. Opin. Neurobiol. 5, 832–840 (1995).

    Article  CAS  Google Scholar 

  52. Dumont, J. P. C. & Robertson, R. M. Neuronal circuits: an evolutionary perspective. Science 233, 849–853 (1986).

    Article  CAS  Google Scholar 

  53. Floreano, D. & Mondada, F. Evolution of homing navigation in a real mobile robot. IEEE Trans. Syst. Man Cybern. B 26, 396–407 (1996).

    Article  CAS  Google Scholar 

  54. Harvey, I., Husbands, P. & Cliff, D. Seeing the light: artificial evolution, real vision. Proc. 3rd Int. Conf. Simul. Adaptive Behav. (eds Cliff, D., Husbands, P., Meyer, J. A. & Wilson, S. W.) 392–401 (MIT Press, Cambridge, Massachusetts, 1994).

    Google Scholar 

  55. Stanley, K. O. & Miikkulainen, R. Evolving neural networks through augmenting topologies. Technical Report No. AI01-290 〈http://citeseer.nj.nec.com/stanley01evolving.html〉 (University of Texas at Austin, 2001).

  56. Aharonov, R., Meilijson, I. & Ruppin, E. in Advances in Neural Information Processing Systems Vol. 13 (eds Leen, T. K., Dietterich, T. G. & Volker, T.) 3–9 (MIT Press, Boston, Massachusetts, 2001).This paper uses the EAA paradigm to develop a new functional-contribution analysis method, which assesses how different behavioural tasks are localized and represented across a controller network.

    Google Scholar 

  57. Wu, J., Cohen, L. B. & Falk, C. X. Neuronal activity during different behaviors in Aplysia: a distributed organization? Science 263, 820–822 (1994).

    Article  CAS  Google Scholar 

  58. Thorpe, S. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) 549–553 (MIT Press, Boston, Massachusetts, 1995).

    Google Scholar 

  59. Lashley, K. S. Brain Mechanisms in Intelligence (Univ. Chicago Press, Chicago, 1929).

    Google Scholar 

  60. Lomber, S. G. The advantages and limitations of permanent or reversible deactivation techniques in the assessment of neural function. J. Neurosci. Methods 86, 109–117 (1999).

    Article  CAS  Google Scholar 

  61. Walsh, V. & Cowey, A. Transcranial magnetic stimulation and cognitive neuroscience. Nature Rev. Neurosci. 1, 73–79 (2000).

    Article  CAS  Google Scholar 

  62. Friston, K. J., Frith, C. D. & Frackowiak, R. S. J. Time-dependent changes in effective connectivity measured with PET. Hum. Brain Imaging 1, 69–79 (1993).

    Article  Google Scholar 

  63. Braitenberg, V. Vehicles, Experiments in Synthetic Psychology (MIT Press, Cambridge, Massachusetts, 1984).

    Google Scholar 

  64. Marr, D. Vision (W. H. Freeman, New York, 1982).

    Google Scholar 

  65. Brooks, R. A. Intelligence without representations. Artif. Intell. 47, 139–159 (1991).

    Article  Google Scholar 

  66. Cliff, D. & Noble, S. Knowledge-based vision and simple visual machines. Phil. Trans. R. Soc. Lond. B 352, 1165–1175 (1997).This paper reviews a series of EAA studies of visual processing to conclude that this task is performed in a distributed, complex manner that is fundamentally different from the top–down, hierarchical processing used in classical computational/engineering approaches.

    Article  CAS  Google Scholar 

  67. Cliff, D. in The Handbook of Brain Theory and Neural Networks (ed. Arbib, M. A.) 626–631 (MIT Press, Boston, Massachusetts, 1995). | PubMed |

    Google Scholar 

  68. Kitano, H. Designing neural networks using genetic algorithms with graph generation system. Complex Syst. 4, 461–476 (1990).

    Google Scholar 

  69. Gruau, F. Automatic definition of modular neural networks. Adaptive Behav. 3, 151–183 (1994).This paper presents graph grammar phenotype-to-genotype encodings that achieve highly compact encodings, leading to modular network architectures.

    Article  Google Scholar 

  70. Kodjabachian, J. & Meyer, J. A. Evolution and development of modular control architectures for 1-D locomotion in six-legged animats. Connect. Sci. 10, 211–254 (1998).

    Article  Google Scholar 

  71. Zhang, B.-T. & Muhlenbein, H. Evolving optimal neural networks using genetic algorithms with Occam's razor. Complex Syst. 7, 199–220 (1993).

    Google Scholar 

  72. Belew, R. K. in Advances in Neural Information Processing (NIPS5) (ed. Cowan, J.) 99–106 (Morgan Kaufmann, San Mateo, California, 1993).

    Google Scholar 

  73. Cangelosi, A., Parisi, D. & Nolfi, S. Cell division and migration in a 'genotype' for neural networks. Network 5, 497–515 (1994).This paper was one of the first to study biologically inspired 'developmental' encodings that are based on a genetic program composed of steps such as cell division and axonal growth. This lays the foundations for more detailed 'regulatory' encodings, in which the identity of the expressed gene subset is determined by interacting transcription factors.

    Article  Google Scholar 

  74. Nolfi, S. & Parisi, D. Evolving artificial neural networks that develop in time. Eur. Conf. Artif. Life 353–367 (Springer, Berlin, 1995).This is one of a series of papers that exploit the potential of EAA models to study the interesting and complex interactions between learning and evolution (originally cast in the Baldwin effect).

    Chapter  Google Scholar 

  75. Cangelosi, A. & Elman, J. L. Gene regulation and biological development in neural networks: an exploratory model. Technical Report No. CRL–UCSD 〈http://www.citeseer.nj.nec.com/context/15377/132530〉 (University of California at San Diego, 1995).

  76. Eggenberger, P. Cell interactions as a control tool of developmental processes for evolutionary robotics. Proc. 4th Int. Conf. Simul. Adaptive Behav. (eds Maes, P. et al.) 440–448 (MIT Press, Cambridge, Massachusetts, 1996).

    Google Scholar 

  77. Floreano, D. & Urzelai, J. Evolutionary robots with online self-organization and behavioral fitness. Neural Netw. 13, 431–443 (2000).This paper explores the evolution of EAA neurocontrollers through learning in a self-organizing manner without any specification of a set of inherited initial synaptic weights.

    Article  CAS  Google Scholar 

  78. Rust, A. G., Adams, R., George, S. & Bolouri, H. Activity-based pruning in developmental artificial neural networks. Proc. 4th Eur. Conf. Artif. Life (ECAL 97) (eds Husbands, P. & Harvey, I.) 224–233 (MIT Press, Cambridge, Massachusetts, 1997).

    Google Scholar 

Download references

Acknowledgements

Supported by the FIRST grant of the Israeli Academy of Sciences, and the Adams Brain Centre, Tel-Aviv University. I thank R. Aharonov, T. Beker, N. Gal-Ruppin, D. Horn, I. Meilijson, Y. Niv, Y. Oron, J. A. Reggia and Z. Solan for careful reading of this manuscript and many helpful comments.

Author information

Authors and Affiliations

  1. School of Computer Science and School of Medicine, Tel-Aviv University, Tel-Aviv, 69978, Israel

    Eytan Ruppin

Authors
  1. Eytan Ruppin
    View author publications

    You can also search for this author in PubMed Google Scholar

Related links

Related links

FURTHER INFORMATION

central pattern generators

neurons and neural networks: computational models

neural networks and behaviour 

Eytan Ruppin's lab 

MIT Encyclopedia of Cognitive Sciences

evolutionary computation

neural networks

Glossary

EVOLUTIONARY COMPUTATION

A computational paradigm that is based on searching the space of possible solutions to a problem by selectively creating new solutions from the best obtained so far.

EMBODIED AGENTS

Agents whose fitness is determined by an interaction with the environment in which they live, given a set of sensors and motors, and a controlling network that mediates between the two.

ARTIFICIAL NEURAL NETWORK

A computational model, the architecture of which is modelled after the brain. They contain idealized neurons called nodes, which are connected together in a network.

GENETIC ALGORITHM

One of the main ways of performing evolutionary computing, based on generating new solutions from existing ones by applying to them genetically inspired operations, such as mutations and crossover. In most cases, the genotype (in which variation takes place) is different from the phenotype (in which fitness-dependent selection take place).

ANIMATS

Artificial embodied agents.

GENOTYPE-TO-PHENOTYPE ENCODING

A mapping (transformation) that specifies the phenotype(s) created from a given genotype.

ANIMATES

Biological embodied agents.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ruppin, E. Evolutionary autonomous agents: A neuroscience perspective. Nat Rev Neurosci 3, 132–141 (2002). https://doi.org/10.1038/nrn729

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrn729

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing