Turning principles into practice is one of the most pressing challenges of artificial intelligence (AI) governance. In this Perspective, we reflect on a governance initiative by one of the world’s largest AI conferences. In 2020, the Conference on Neural Information Processing Systems (NeurIPS) introduced a requirement for submitting authors to include a statement on the broader societal impacts of their research. Drawing insights from similar governance initiatives, including institutional review boards (IRBs) and impact requirements for funding applications, we investigate the risks, challenges and potential benefits of such an initiative. Among the challenges, we list a lack of recognized best practice and procedural transparency, researcher opportunity costs, institutional and social pressures, cognitive biases and the inherently difficult nature of the task. The potential benefits, on the other hand, include improved anticipation and identification of impacts, better communication with policy and governance experts, and a general strengthening of the norms around responsible research. To maximize the chance of success, we recommend measures to increase transparency, improve guidance, create incentives to engage earnestly with the process, and facilitate public deliberation on the requirement’s merits and future. Perhaps the most important contribution from this analysis are the insights we can gain regarding effective community-based governance and the role and responsibility of the AI research community more broadly.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
AI and Ethics Open Access 31 May 2022
Subscribe to Nature+
Get immediate online access to Nature and 55 other Nature journal
Subscribe to Journal
Get full journal access for 1 year
only $8.25 per issue
All prices are NET prices.
VAT will be added later in the checkout.
Tax calculation will be finalised during checkout.
Get time limited or full article access on ReadCube.
All prices are NET prices.
Winfield, A. F. T. & Jirotka, M. Ethical governance is essential to building trust in robotics and artificial intelligence systems. Phil. Trans. R. Soc. A 376, 20180085 (2018).
Fisher, E., Mahajan, R. L. & Mitcham, C. Midstream modulation of technology: governance from within. Bull. Sci. Technol. Soc. 26, 485–496 (2006).
NeurIPS Call For Papers (2020); https://nips.cc/Conferences/2020/CallForPapers
Johnson, K. NeurIPS requires AI researchers to account for societal impact and financial conflicts of interest. Venturebeat https://venturebeat.com/2020/02/24/neurips-requires-ai-researchers-to-account-for-societal-impact-and-financial-conflicts-of-interest/ (24 February 2020).
Brundage, M. Artificial intelligence and responsible innovation. In Fundamental Issues of Artificial Intelligence 543−554 (Synthese Library, 2016).
Hecht, B. et al. It’s Time To Do Something: Mitigating The Negative Impacts Of Computing Through A Change To The Peer Review Process (ACM Future of Computing Academy, 29 March 2018); https://acm-fca.org/2018/03/29/negativeimpacts/
NeurIPS Getting Started with NeurIPS 2020 (2020); https://medium.com/@NeurIPSConf/getting-started-with-neurips-2020-e350f9b39c28
NeurIPS NeurIPS 2020 FAQ for Authors (2020); https://nips.cc/Conferences/2020/PaperInformation/NeurIPS-FAQ
Lin, H.-T., Balcan, M. F., Hadsell, R. & Ranzato, M. A. What we learned from NeurIPS 2020 reviewing process. Medium https://medium.com/@NeurIPSConf/what-we-learned-from-neurips-2020-reviewing-process-e24549eea38f (2020).
Hamburger, P. The new censorship: institutional review boards. Supreme Court Rev. 2004, 271–354 (2004).
Buchanan, E., Aycock, J., Dexter, S., Dittrich, D. & Hvizdak, E. Computer science security research and human subjects: emerging considerations for research ethics boards. J. Emp. Res. Human Res. Ethics 6, 71–83 (2011).
Amorim, P. F., Sacramento, C., Capra, E. P., Tavares, P. Z. & Ferreira, S. B. L. Submit or not my HCI research project to the ethics committee, that is the question. In Proc. 18th Brazilian Symp. on Human Factors in Computing Systems (IHC ’19) 1−11 (Association for Computing Machinery, 2019).
Abbott, L. & Grady, C. A systematic review of the empirical literature evaluating IRBs: what we know and what we still need to learn. J. Emp. Res. Human Res. Ethics 6, 3–19 (2011).
Hyman, D. A. Institutional review boards: is this the least worst we can do? Northwestern Univ. Law Rev. 101, 749–774 (2007).
Zywicki, T. J. Institutional review boards as academic bureaucracies: an economic and experiential analysis. Northwestern Univ. Law Rev. 101, 861–896 (2007).
Whitney, S. N. et al. Principal investigator views of the IRB system. Int. J. Med. Sci. 5, 68–72 (2008).
Chadwick, G. L. & Dunn, C. Institutional review boards: changing with the times? J. Public Health Manage. Practice 6, 19–27 (2000).
Fost, N. & Levine, R. J. The dysregulation of human subjects research. JAMA 298, 2196 (2007).
Dziak, K. et al. Variations among institutional review board reviews in a multisite health services research study. Health Serv. Res. 40, 279–290 (2005).
Larson, E., Bratts, T., Zwanziger, J. & Stone, P. A survey of IRB process in 68 U.S. hospitals. J. Nurs. Scholarship 36, 260–264 (2004).
Shah, S., Whittle, A., Wilfond, B., Gensler, G. & Wendler, D. How do institutional review boards apply the federal risk and benefit standards for pediatric research? JAMA 291, 476 (2004).
McWilliams, R. Problematic variation in local institutional review of a multicenter genetic epidemiology study. JAMA 290, 360 (2003).
Goldman, J. Inconsistency and institutional review boards. JAMA 248, 197 (1982).
Reeser, J. C., Austin, D. M., Jaros, L. M., Mukesh, B. N. & McCarty, C. A. Investigating perceived institutional review board quality and function using the IRB researcher assessment tool. J. Emp. Res. Human Res. Ethics 3, 25–34 (2008).
Stryjewski, T. P., Kalish, B. T., Silverman, B. & Lehmann, L. S. The impact of institutional review boards (IRBs) on clinical innovation: a survey of investigators and IRB members. J. Emp. Res. Human Res. Ethics 10, 481–487 (2015).
Keith-Spiegel, P., Koocher, G. P. & Tabachnick, B. What scientists want from their research ethics committee. J. Emp. Res. Human Res. Ethics 1, 67–81 (2006).
Saleem, T. & Khalid, U. Institutional review boards—a mixed blessing. Int. Arch. Med. 4, 19 (2011).
ACM SIGMETRICS 2021. Call for Papers (2020). https://www.sigmetrics.org/sigmetrics2021/call_for_papers.html
Narayanan, A. & Zevenbergen, B. No Encore for Encore? Ethical Questions for Web-Based Censorship Measurement SSRN Scholarly Paper ID 2665148 (Social Science Research Network, 2015).
Kenneally, E. & Bailey, M. Cyber-security research ethics dialogue and strategy workshop. ACM SIGCOMM Comput. Commun. Rev. 44, 76–79 (2014).
Burnett, S. & Feamster, N. Encore: lightweight measurement of web censorship with cross-origin requests. In Proc. 2015 ACM Conf. on Special Interest Group on Data Communication (SIGCOMM ’15) 653-667 (Association for Computing Machinery, 2015).
Kramer, A. D. I., Guillory, J. E. & Hancock, J. T. Experimental evidence of massive-scale emotional contagion through social networks. Proc. Natl Acad. Sci. 111, 8788–8790 (2014).
Editorial Expression of Concern: Experimental evidence of massive-scale emotional contagion through social networks. Proc. Natl Acad. Sci. 111, 10779−10779 (2014).
EPSRC Framework for Responsible Innovation https://epsrc.ukri.org/research/framework/ (2020).
NSF Ch. II—Proposal Preparation Instructions. Proposal & Award Policies & Procedures Guide (29 January 2018); https://www.nsf.gov/pubs/policydocs/pappg18_1/pappg_2.jsp#IIC2di
Tretkoff, E. NSF’s ‘broader impacts’ criterion gets mixed reviews. Am. Phys. Soc. News 16, https://www.aps.org/publications/apsnews/200706/nsf.cfm (2007).
Frodeman, R. & Holbrook, J. B. Science’s social effects. Iss. Sci. Technol. 23, 28–30 (2007).
Bozeman, B. & Boardman, C. Broad impacts and narrow perspectives: passing the buck on science and social impacts. Soc. Epist. 23, 183–198 (2009).
Holbrook, J. B. & Frodeman, R. Peer review and the ex ante assessment of societal impacts. Res. Eval. 20, 239–246 (2011).
Bozeman, B. & Youtie, J. Socio-economic impacts and public value of government-funded research: lessons from four US National Science Foundation initiatives. Res. Pol. 46, 1387–1398 (2017).
Owen, R. & Goldberg, N. Responsible innovation: a pilot study with the U.K. Engineering and Physical Sciences Research Council. Risk Anal. 30, 1699–1707 (2010).
EPSRC Anticipate, Reflect, Engage And Act (AREA) https://epsrc.ukri.org/research/framework/area/ (2020).
Owen, R., Macnaghten, P. & Stilgoe, J. Responsible research and innovation: from science in society to science for society, with society. Sci. Public Pol. 39, 751–760 (2012).
Stilgoe, J., Owen, R. & Macnaghten, P. Developing a framework for responsible innovation. Res. Pol. 42, 1568–1580 (2013).
Marchant, G. E., Allenby, B. R. & Herkert, J. R. (eds.) The Growing Gap Between Emerging Technologies and Legal-Ethical Oversight: The Pacing Problem (The International Library of Ethics, Law and Technology, Springer, 2011).
Gray, I. M. & Edwards-Jones, G. A review of the quality of environmental impact assessments in the Scottish forest sector. Forestry Int. J. Forest Res. 72, 1–10 (1999).
Assessing the Social and Environmental Impacts of European Research Tech. Rep. EUR 21702 (European Commission, 2005).
Spaapen, J. & van Drooge, L. Introducing ’productive interactions’ in social impact assessment. Res. Eval. 20, 211–218 (2011).
Pathways to Impact: Impact core to the UK Research and Innovation Application Process (UK Research and Innovation, 2020); https://webarchive.nationalarchives.gov.uk/20200923120040/https://ahrc.ukri.org/newsevents/news/pathways-to-impact-impact-core-to-the-uk-research-and-innovation-application-process/
Bietti, E. From ethics washing to ethics bashing: a view on tech ethics from within moral philosophy. In Proc. 2020 Conf. on Fairness, Accountability, and Transparency 210−219 (Association for Computing Machinery, 2020).
Hagendorff, T. & Meding, K. The big picture: ethical considerations and statistical analysis of industry involvement in machine learning research. Preprint at http://arxiv.org/abs/2006.04541 (2020).
Stanovich, K. E., West, R. F. & Toplak, M. E. Myside bias, rational thinking, and intelligence. Curr. Dir. Psychol. Sci. 22, 259–264 (2013).
Plous, S. The Psychology Of Judgment And Decision Making (McGraw-Hill, 1993).
Curley, S. P., Yates, J. F. & Abrams, R. A. Psychological sources of ambiguity avoidance. Org. Behav. Human Decision Process. 38, 230–256 (1986).
Nickerson, R. S. Confirmation bias: a ubiquitous phenomenon in many guises. Rev. Gen. Psychol. 2, 175–220 (1998).
Ashurst, C. et al. A Guide to Writing the NeurIPS Impact Statement (Centre for the Governance of AI, 13 May 2020); https://medium.com/@GovAI/a-guide-to-writing-the-neurips-impact-statement-4293b723f832
Hecht, B. Suggestions for Writing NeurIPS 2020 Broader Impacts Statements (22 February, 2020); https://medium.com/@BrentH/suggestions-for-writing-neurips-2020-broader-impacts-statements-121da1b765bf
Porter, A. L., Garner, J. & Crowl, T. Research coordination networks: evidence of the relationship between funded interdisciplinary networking and scholarly impact. BioScience 62, 282–288 (2012).
We thank J. Tenenbaum, Y. Gal, T. Shevlane and colleagues at the Centre for the Governance of AI for helpful feedback and comments.
The authors declare no competing interests.
Peer review information Nature Machine Intelligence thanks Gillian Hadfield, Sean Legassick and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Prunkl, C.E.A., Ashurst, C., Anderljung, M. et al. Institutionalizing ethics in AI through broader impact requirements. Nat Mach Intell 3, 104–110 (2021). https://doi.org/10.1038/s42256-021-00298-y
This article is cited by
Ethics methods are required as part of reporting guidelines for artificial intelligence in healthcare
Nature Machine Intelligence (2022)
Nature Machine Intelligence (2022)
AI and Ethics (2022)
Nature Machine Intelligence (2021)