Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

The evolution of citation graphs in artificial intelligence research

Nature Machine Intelligence volume 1, pages 79–85 (2019)Cite this article

Subjects

Abstract

As artificial intelligence (AI) applications see wider deployment, it becomes increasingly important to study the social and societal implications of AI adoption. Therefore, we ask: are AI research and the fields that study social and societal trends keeping pace with each other? Here, we use the Microsoft Academic Graph to study the bibliometric evolution of AI research and its related fields from 1950 to today. Although early AI researchers exhibited strong referencing behaviour towards philosophy, geography and art, modern AI research references mathematics and computer science most strongly. Conversely, other fields, including the social sciences, do not reference AI research in proportion to its growing paper production. Our evidence suggests that the growing preference of AI researchers to publish in topic-specific conferences over academic journals and the increasing presence of industry research pose a challenge to external researchers, as such research is particularly absent from references made by social scientists.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Access options

Rent or buy this article

Get just this article for as long as you need it

$39.95

Learn more

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Citation patterns among CS subfields identify areas of AI-related research.
Fig. 2: The referencing strength between AI and other sciences is declining.
Fig. 3: AI research is increasingly dominated by only a few research institutions and AI-specific conferences.
Fig. 4: Industry is increasingly central to AI research, but industry-authored AI papers are referenced less often by other academic fields.

References

  1. Brynjolfsson, E. & Mitchell, T. What can machine learning do? Workforce implications. Science 358, 1530–1534 (2017).

    Article  Google Scholar 

  2. Kirilenko, A., Kyle, A. S., Samadi, M. & Tuzun, T. The flash crash: high-frequency trading in an electronic market. J. Finance 72, 967–998 (2017).

    Article  Google Scholar 

  3. Brogaard, J. et al. High Frequency Trading and its Impact on Market Quality Working Paper No. 66 (Northwestern University Kellogg School of Management, 2010).

  4. Verghese, A., Shah, N. H. & Harrington, R. A. What this computer needs is a physician: humanism and artificial intelligence. J. Am. Med. Assoc. 319, 19–20 (2018).

    Article  Google Scholar 

  5. Hosny, A., Parmar, C., Quackenbush, J., Schwartz, L. H. & Aerts, H. J. Artificial intelligence in radiology. Nat. Rev. Cancer 18, 500–510 (2018).

  6. Bonnefon, J.-F., Shariff, A. & Rahwan, I. The social dilemma of autonomous vehicles. Science 352, 1573–1576 (2016).

    Article  Google Scholar 

  7. The Road to Zero: A Vision of Achieving Zero Roadway Deaths by 2050 (National Safety Council and the RAND Corporation, 2018).

  8. Russell, S., Hauert, S., Altman, R. & Veloso, M. Ethics of artificial intelligence. Nature 521, 415–416 (2015).

    Article  Google Scholar 

  9. Rahwan, I. Society-in-the-loop: programming the algorithmic social contract. Ethics Inf. Technol. 20, 5–14 (2018).

    Article  Google Scholar 

  10. Crandall, J. W. et al. Cooperating with machines. Nat. Commun. 9, 233 (2018).

    Article  Google Scholar 

  11. Taddeo, M. & Floridi, L. How AI can be a force for good. Science 361, 751–752 (2018).

    Article  MathSciNet  Google Scholar 

  12. Miller, A. P. Want less-biased decisions? Use algorithms. Harvard Business Review https://hbr.org/2018/07/want-less-biased-decisions-use-algorithms (2018).

  13. Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J. & Mullainathan, S. Human decisions and machine predictions. Q. J. Econ. 133, 237–293 (2017).

    MATH  Google Scholar 

  14. Naik, N., Kominers, S. D., Raskar, R., Glaeser, E. L. & Hidalgo, C. A. Computer vision uncovers predictors of physical urban change. Proc. Natl Acad. Sci. USA 114, 7571–7576 (2017).

    Article  Google Scholar 

  15. Erel, S. L. H. T. C., Isil & Weisbach, M. S. Could machine learning help companies select better board directors? Harvard Business Review https://hbr.org/2018/04/research-could-machine-learning-help-companies-select-better-board-directors (2018).

  16. Buolamwini, J. & Gebru, T. Gender shades: intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency 77–91 (MLR, 2018).

  17. Buolamwini, J. How I’m fighting bias in algorithms. TED Talks https://www.ted.com/talks/joy_buolamwini_how_i_m_fighting_bias_in_algorithms (2016).

  18. Frank, M. R., Sun, L., Cebrian, M., Youn, H. & Rahwan, I. Small cities face greater impact from automation. J. R. Soc. Interface 15, 20170946 (2018).

    Article  Google Scholar 

  19. Frey, C. B. & Osborne, M. A. The future of employment: how susceptible are jobs to computerisation? Technol. Forecast. Soc. Change 114, 254–280 (2017).

    Article  Google Scholar 

  20. Acemoglu, D. & Restrepo, P. Robots and jobs: evidence from US labor markets (National Bureau of Economic Research, 2017).

  21. Klinger, J., Mateos-Garcia, J. C. & Stathoulopoulos, K. Deep learning, deep change? Mapping the development of the artificial intelligence general purpose technology. Preprint at https://arxiv.org/abs/1808.06355 (2018).

  22. Sinha, A. et al. An overview of Microsoft Academic Service (MAS) and applications. In Proc. 24th International Conference on World Wide Web 243–246 (ACM, 2015).

  23. Effendy, S. & Yap, R. H. Analysing trends in computer science research: a preliminary study using the microsoft academic graph. In Proceedings of the 26th International Conference on World Wide Web Companion, 1245–1250 (International World Wide Web Conferences Steering Committee, 2017).

  24. Hug, S. E. & Brändle, M. P. The coverage of Microsoft academic: analyzing the publication output of a university. Scientometrics 113, 1551–1571 (2017).

    Article  Google Scholar 

  25. Burd, R. et al. GRAM: global research activity map. In Proc. 2018 International Conference on Advanced Visual Interfaces 31 (ACM, 2018).

  26. Fiala, D. & Tutoky, G. Computer science papers in web of science: a bibliometric analysis. Publications 5, 23 (2017).

    Article  Google Scholar 

  27. Russell, S. J. & Norvig, P. Artificial Intelligence: A Modern Approach (Pearson Education Limited, London, 2016).

  28. McCorduck, P. Machines Who Think: A Personal Inquiry into the History and Prospects of Artificial Intelligence (CRC, Natik, 2009).

  29. Kieval, H. J. Pursuing the Golem of Prague: Jewish culture and the invention of a tradition. Mod. Jud. 17, 1–20 (1997).

    Article  Google Scholar 

  30. Pollin, B. R. Philosophical and literary sources of Frankenstein. Comp. Lit. 17, 97–108 (1965).

    Article  Google Scholar 

  31. Floridi, L. Distributed morality in an information society. Sci. Eng. Ethics 19, 727–743 (2013).

    Article  Google Scholar 

  32. Plant, S. Zeros and ones (Doubleday Books, 1997).

  33. David, A. H. Why are there still so many jobs? The history and future of workplace automation. J. Econ. Perspect. 29, 3–30 (2015).

    Google Scholar 

  34. Sinatra, R., Deville, P., Szell, M., Wang, D. & Barabási, A.-L. A century of physics. Nat. Phys. 11, 791 (2015).

    Article  Google Scholar 

  35. Morgan, A. C., Economou, D., Way, S. F. & Clauset, A. Prestige drives epistemic inequality in the diffusion of scientific ideas. EPJ Data Sci. 7, 40 (2018).

    Article  Google Scholar 

  36. Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484 (2016).

    Article  Google Scholar 

  37. Bergholz, R., Timm, K. & Weisser, H. Autonomous vehicle arrangement and method for controlling an autonomous vehicle. US patent 6,151,539 (2000).

  38. Pilutti, T. E., Rupp, M. Y., Trombley, R. A., Waldis, A. & Yopp, W. T. Autonomous vehicle identification. US patent 9,552,735 (2017).

  39. Herbach, J. S. & Fairfield, N. Detecting that an autonomous vehicle is in a stuck condition. US patent 8,996,224 (2015).

  40. Pham, M. C., Klamma, R. & Jarke, M. Development of computer science disciplines: a social network analysis approach. Soc. Netw. Anal. Min. 1, 321–340 (2011).

    Article  Google Scholar 

  41. Barabási, A.-L. & Albert, R. Emergence of scaling in random networks. Science 286, 509–512 (1999).

    Article  MathSciNet  Google Scholar 

  42. Page, L., Brin, S., Motwani, R. & Winograd, T. The PageRank Citation Ranking: Bringing Order to the Web (Stanford InfoLab, 1999).

  43. Larivière, V., Macaluso, B., Mongeon, P., Siler, K. & Sugimoto, C. R. Vanishing industries and the rising monopoly of universities in published research. PLoS ONE 13, 1–10 (2018).

    Article  Google Scholar 

  44. Freyne, J., Coyle, L., Smyth, B. & Cunningham, P. Relative status of journal and conference publications in computer science. Commun. ACM 53, 124–132 (2010).

    Article  Google Scholar 

  45. Newman, M. E. Finding community structure in networks using the eigenvectors of matrices. Phys. Rev. E 74, 036104 (2006).

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

The authors would like to thank E. Moro and Z. Epstein for their comments.

Author information

Authors and Affiliations

  1. Media Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA

    Morgan R. Frank, Manuel Cebrian & Iyad Rahwan

  2. Kellogg School of Management, Northwestern University, Evanston, IL, USA

    Dashun Wang

  3. Northwestern Institute on Complex Systems, Northwestern University, Evanston, IL, USA

    Dashun Wang

  4. Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA, USA

    Iyad Rahwan

  5. Center for Humans and Machines, Max Planck Institute for Human Development, Berlin, Germany

    Iyad Rahwan

Authors
  1. Morgan R. Frank
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dashun Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manuel Cebrian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Iyad Rahwan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

M.R.F. and D.W. processed data and produced figures. All authors wrote the manuscript.

Corresponding author

Correspondence to Iyad Rahwan.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary Information

Supplementary notes and figures

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Frank, M.R., Wang, D., Cebrian, M. et al. The evolution of citation graphs in artificial intelligence research. Nat Mach Intell 1, 79–85 (2019). https://doi.org/10.1038/s42256-019-0024-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-019-0024-5

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing