It has been the historic responsibility of the social sciences to investigate human societies. Fulfilling this responsibility requires social theories, measurement models and social data. Most existing theories and measurement models in the social sciences were not developed with the deep societal reach of algorithms in mind. The emergence of ‘algorithmically infused societies’—societies whose very fabric is co-shaped by algorithmic and human behaviour—raises three key challenges: the insufficient quality of measurements, the complex consequences of (mis)measurements, and the limits of existing social theories. Here we argue that tackling these challenges requires new social theories that account for the impact of algorithmic systems on social realities. To develop such theories, we need new methodologies for integrating data and measurements into theory construction. Given the scale at which measurements can be applied, we believe measurement models should be trustworthy, auditable and just. To achieve this, the development of measurements should be transparent and participatory, and include mechanisms to ensure measurement quality and identify possible harms. We argue that computational social scientists should rethink what aspects of algorithmically infused societies should be measured, how they should be measured, and the consequences of doing so.
This is a preview of subscription content, access via your institution
Open Access articles citing this article.
Scientific Reports Open Access 19 August 2022
Cognitive Research: Principles and Implications Open Access 19 August 2021
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Rent or buy this article
Prices vary by article type
Prices may be subject to local taxes which are calculated during checkout
boyd, d. The future of privacy: how privacy norms can inform regulation. In 32nd Intl Conf. Data Protection and Privacy Commissioners (2010); https://www.danah.org/papers/talks/2010/PrivacyGenerations.html.
Gill, K. S. The internet of things! Then what? AI Soc. 28, 367–371 (2013).
O’Reilly, T. Open data and algorithmic regulation. In Beyond Transparency: Open Data and the Future of Civic Innovation (eds Goldstein, B. & Dyson, L.) 289–300 (Code for America Press, 2013).
Castells, M. The Information Age: Economy, Society and Culture. Vol. 1: The Rise of the Network Society (Wiley–Blackwell, 1996).
Fleder, D. & Hosanagar, K. Blockbuster culture’s next rise or fall: the impact of recommender systems on sales diversity. Manage. Sci. 55, 697–712 (2009).
Tufekci, Z. Engineering the public: big data, surveillance and computational politics. First Monday 19, https://doi.org/10.5210/fm.v19i7.4901 (2014).
Bakshy, E., Messing, S. & Adamic, L. A. Exposure to ideologically diverse news and opinion on Facebook. Science 348, 1130–1132 (2015).
Ferrara, E., Varol, O., Davis, C., Menczer, F. & Flammini, A. The rise of social bots. Commun. ACM 59, 96–104 (2016).
Le Chen, A. M. & Wilson, C. An empirical analysis of algorithmic pricing on Amazon Marketplace. In Proc. 25th Intl Conf. World Wide Web (WWW’16) (eds Bourdeau, J. et al.) 1339–1349 (International World Wide Web Conferences Steering Committee, 2016); https://doi.org/10.1145/2872427.2883089.
Salganik, M. J., Sheridan Dodds, P. & Watts, D. J. Experimental study of inequality and unpredictability in an artificial cultural market. Science 311, 854–856 (2006).
Baym, N. K. Playing to the Crowd: Musicians, Audiences, and the Intimate Work of Connection (NYU Press, 2018).
Burgess, J. & Green, J. YouTube: Online Video and Participatory Culture (Wiley, 2018).
Hitsch, G. J., Hortaçsu, A. & Ariely, D. Matching and sorting in online dating. Am. Econ. Rev. 100, 130–163 (2010).
Zignani, M. et al. Link and triadic closure delay: temporal metrics for social network dynamics. In Proc. 8th Intl AAAI Conf. Web and Social Media (eds Adar, E. & Resnick, P.) 564–573 (2014).
Malik, M. & Pfeffer, J. Identifying platform effects in social media data. In Proc. 10th Intl AAAI Conf. Web and Social Media (eds Krishna, G. & Strohmaier, M.) 241–249 (2016).
Su, J., Sharma, A. & Goel, S. The effect of recommendations on network structure. In Proc. 25th Intl Conf. World Wide Web (WWW’16) (eds Bourdeau, J. et al.) 1157–1167 (International World Wide Web Conferences Steering Committee, 2016); https://doi.org/10.1145/2872427.2883040.
Loscalzo, J. & Barabasi, A.-L. Systems biology and the future of medicine. Wiley Interdiscip. Rev. Syst. Biol. Med. 3, 619–627 (2011).
Frizzell, J. D., et al. Prediction of 30-day all-cause readmissions in patients hospitalized for heart failure: comparison of machine learning and other statistical approaches. JAMA Cardiol. 2, 204–209 (2017).
Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 366, 447–453 (2019).
Huang, C.-L., Chen, M.-C. & Wang, C.-J. Credit scoring with a data mining approach based on support vector machines. Expert Syst. Appl. 33, 847–856 (2007).
Perry, W. L., McInnis, B., Price, C. C., Smith, S. C. & Hollywood, J. S. Predictive Policing: The Role of Crime Forecasting in Law Enforcement Operations (RAND Corporation, 2013).
Dressel, J. & Farid, H. The accuracy, fairness, and limits of predicting recidivism. Sci. Adv. 4, eaao5580 (2018).
Raghavan, M., Barocas, S., Kleinberg, J. & Levy, K. Mitigating bias in algorithmic hiring: evaluating claims and practices. In Proc. 2020 Conf. Fairness, Accountability, and Transparency (eds Hildebrandt, M. & Castillo, C.) 469–481 (ACM, 2020); https://doi.org/10.1145/3351095.3372828.
Hannák, A. et al. Bias in online freelance marketplaces: evidence from TaskRabbit and Fiverr. In Proc. 2017 ACM Conf. Computer Supported Cooperative Work and Social Computing (CSCW’17) (eds Lee, C. P. & Poltrock, S.) 1914–1933 (ACM, 2017); https://doi.org/10.1145/2998181.2998327. This study reports on sociodemographic inequalities in online marketplaces.
Gray, M. L. & Suri, S. Ghost Work: How to Stop Silicon Valley From Building a New Global Underclass (Eamon Dolan Books, 2019).
Beer, D. The social power of algorithms. Inf. Commun. Soc. 20, 1–13 (2017).
Kleinberg, J., Ludwig, J., Mullainathan, S. & Sunstein, C. R. Algorithms as discrimination detectors. Proc. Natl Acad. Sci. USA 117, 30096–30100 (2020).
Buolamwini, J. & Gebru, T. Gender shades: intersectional accuracy disparities in commercial gender classification. In Proc. 1st Conf. Fairness, Accountability and Transparency (eds Friedler, S. A. & Wilson, C.) 77–91 (2018).
Moy, L. How police technology aggravates racial inequity: a taxonomy of problems and a path forward. Univ. Illinois Law Rev. 2021, 139–193 (2021).
Hutchinson, B. & Mitchell, M. 50 years of test (un)fairness: lessons for machine learning. In Proc. Conf. Fairness, Accountability, and Transparency (eds Morgenstern, J. & boyd, d.) 49–58 (2019).
Barocas, S. & Selbst, A. D. Big data’s disparate impact. Calif. Law Rev. 104, 671–732 (2016).
Milli, S., Miller, J., Dragan, A. D. & Hardt M. The social cost of strategic classification. In Proc. Conf. Fairness, Accountability, and Transparency 230–239 (2019).
Bender, E. M., Gebru, T., McMillan-Major, A. & Shmitchell. S. On the dangers of stochastic parrots: can language models be too big? In Proc. Conf. Fairness, Accountability, and Transparency (eds Elish, M. C. et al.) 610–623 (2021).
Schnabel, T., Swaminathan, A., Singh, A., Chandak, N. & Joachims, T. Recommendations as treatments: debiasing learning and evaluation. In Intl Conf. Machine Learning (eds Balcan, M. F. & Weinberger, K. Q.) 1670–1679 (2016).
Sztompka, P. in Polish Essays in the Methodology of the Social Sciences (ed. Wiatr, J. J.) 173–194 (Springer, 1979).
Jaccard, J. & Jacoby, J. Theory Construction and Model Building Skills: A Practical Guide for Social Scientists (Guilford, 2010).
Lord, F. M. & Novick, M. R. Statistical Theories of Mental Test Scores (Addison-Wesley, 1968).
Allen, M. J. & Yen, W. M. Introduction to Measurement Theory (Waveland, 2002).
Joye, D., Wolf, C., Smith, T. W. & Fu, Y. in The SAGE Handbook of Survey Methodology (eds Wolf, C. et al.) 3–15 (Sage, 2016).
Strathern, M. ‘Improving ratings’: audit in the British university system. Eur. Rev. 5, 305–321 (1997).
Campbell, D. T. Assessing the impact of planned social change. Eval. Program Plann. 2, 67–90 (1979).
Festinger, L. A Theory of Cognitive Dissonance (Stanford Univ. Press, 1957).
Heider, F. Attitudes and cognitive organization. J. Psychol. 21, 107–112 (1946).
Cartwright, D. & Harary, F. Structural balance: a generalization of Heider’s theory. Psychol. Rev. 63, 277–293 (1956).
McPherson, M., Smith-Lovin, L. & Cook, J. M. Birds of a feather: homophily in social networks. Annu. Rev. Sociol. 27, 415–444 (2001).
Seitlinger, P., Kowald, D., Trattner, C. & Ley, T. Recommending tags with a model of human categorization. In Proc. 22nd ACM Intl Conf. on Information & Knowledge Management (eds He, Q. & Iyengar, A.) 2381–2386 (ACM, 2013).
Bowker, G. C. & Star, S. L. Sorting Things Out: Classification and its Consequences (MIT Press, 2000).
Healy, K. The performativity of networks. Euro. J. Sociol. 56, 175–205 (2015). This article argues that theories have the potential to reformat and reorganize the phenomena that models purport to describe.
Zuboff, S. The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power (Profile Books, 2018).
Pasquale, F. The Black Box Society: The Secret Algorithms that Control Money and Information (Harvard Univ. Press, 2015).
Nissenbaum, H. How Computer Systems Embody Values (IEEE Computer Society Press, 2001).
Seaver, N. Knowing algorithms. In DigitalSTS (eds Vertesi, J. & Ribes, D.) 412–422 (Princeton Univ. Press, 2013).
boyd, d. & Crawford, K. Critical questions for big data: provocations for a cultural, technological, and scholarly phenomenon. Inf. Commun. Soc. 15, 662–679 (2012).
Graham, S. & Wood, D. Digitizing surveillance: categorization, space, inequality. Crit. Soc. Policy 23, 227–248 (2003).
Benjamin, R. Catching our breath: critical race STS and the carceral imagination. Engaging Sci. Technol. Soc. 2, 145–156 (2016).
Lazer, D. et al. Computational social science. Science 323, 721–723 (2009). This landmark article discussed the early potential of computational approaches for the social sciences.
Lazer, D. M. J. et al. Computational social science: obstacles and opportunities. Science 369, 1060–1062 (2020).
Vosoughi, S., Roy, D. & Aral, S. The spread of true and false news online. Science 359, 1146–1151 (2018).
Del Vicario, M. et al. The spreading of misinformation online. Proc. Natl Acad. Sci. USA 113, 554–559 (2016).
Bail, C. A. et al. Exposure to opposing views on social media can increase political polarization. Proc. Natl Acad. Sci. USA 115, 9216–9221 (2018).
Blalock, H. M. Conceptualization and Measurement in the Social Sciences (Sage, 1982).
RatSWD Quality Standards Working Group. Quality Standards for the Development, Application, and Evaluation of Measurement Instruments in Social Science Survey Research RatSWD Working Papers 245 (German Data Forum (RatSWD), 2015). This paper proposes quality standards and guidelines for the development, application and evaluation of measurement instruments in social science survey research.
Jacobs, A. Z. & Wallach, H. Measurement and fairness. In FAccT’21: Proc. 2021 ACM Conf. Fairness, Accountability, and Transparency (eds Elish, M. C. et al.) 375–385 (ACM, 2019); https://doi.org/10.1145/3442188.3445901. This paper describes how validity issues can lead to fairness issues.
Adcock, R. & Collier, D. Measurement validity: a shared standard for qualitative and quantitative research. Am. Polit. Sci. Rev. 95, 529–546 (2001).
Hofman, J. M., Sharma, A. & Watts, D. J. Prediction and explanation in social systems. Science 355, 486–488 (2017).
Peters, J., Janzing, D. & Schoelkopf, B. Elements of Causal Inference: Foundations and Learning Algorithms (MIT Press, 2017).
Jungherr, A. in Digital Discussions: How Big Data Informs Political Communication (eds Stroud, N. J. & McGregor, S.) 9–35 (Routledge, 2018).
Donoho, D. 50 years of data science. J. Comput. Graph. Stat. 26, 745–766 (2017).
Blodgett, S. L., Lopez, G., Olteanu, A., Sim, R. & Wallach, H. Stereotyping Norwegian salmon: an inventory of pitfalls in fairness benchmark datasets. In Proc. 59th Annual Meeting of the Association for Computational Linguistics (ed. Zong, C.) (2021).
Ethayarajh, K. & Jurafsky, D. Utility is in the eye of the user: a critique of NLP leaderboard design. In Proc. 2020 Conf. Empirical Methods in Natural Language Processing (EMNLP) (eds Webber, B. et al.) 4846–4853 (2020).
Jungherr, A., Schoen, H., Posegga, O. & Jürgens, P. Digital trace data in the study of public opinion: an indicator of attention toward politics rather than political support. Soc. Sci. Comput. Rev. 35, 336–356 (2017).
Samory, M., Sen, I., Kohne, J., Flöck, F. & Wagner, C. Call me sexist, but…: Revisiting sexism detection using psychological scales and adversarial samples. In Intl AAAI Conf. Web and Social Media 573–584 (2021).
Gebru, T. et al. Datasheets for datasets. Preprint at https://arxiv.org/abs/1803.09010 (2018).
Olteanu, A., Castillo, C., Diaz, F. & Kıcıman, E. Social data: biases, methodological pitfalls, and ethical boundaries. Front. Big Data 2, 13 (2019).
Sen, I., Floeck, F., Weller, K., Weiss, B. & Wagner, C. A total error framework for digital traces of human behavior on online platforms. Public Opin. Q. (in the press). This paper gives a systematic overview of errors that may be introduced when analysing digital traces of human behavior.
Lazer, D. Issues of construct validity and reliability in massive, passive data collections. The City Papers: An Essay Collection from The Decent City Initiative http://citiespapers.ssrc.org/issues-of-construct-validity-and-reliability-in-massive-passive-data-collections/ (2015).
Perdomo, J., Zrnic, T., Mendler-Dünner, C. & Hardt, M. Performative prediction. In Proc. 37th Intl Conf. Machine Learning (eds Daumé III, H. & Singh, A.) 7599–7609 (PMLR, 2020).
Hannak, A. et al. Measuring personalization of web search. In Proc. 22nd Intl Conf. World Wide Web (WWW’13) (eds Schwabe, D. et al.) 527–538 (ACM, 2013).
Ali, M. et al. Discrimination through optimization: how Facebook’s ad delivery can lead to biased outcomes. Proc. ACM Hum.–Comp. Interact. 3, 199 (2019).
Thomas, P. & Brunskill, E. Data-efficient off-policy policy evaluation for reinforcement learning. In Intl Conf. Machine Learning (eds Balcan, M. F. & Weinberger, K. Q.) 2139–2148 (PMLR, 2016).
Sinha, A., Gleich, D. F. & Ramani, K. Deconvolving feedback loops in recommender systems. In Advances in Neural Information Processing Systems 29 (eds Lee, D. et al.) 3243–3251 (2016).
Tomašev, N. et al. AI for social good: unlocking the opportunity for positive impact. Nat. Commun. 11, 2468 (2020).
Mau, S. Das metrische Wir: Über die Quantifizierung des Sozialen (Suhrkamp, 2017). This book explores the implications of measurements on social systems.
D’Ignazio, C. & Klein, L. F. Data Feminism (MIT Press, 2020).
Muller, J. Z. The Tyranny of Metrics (Princeton Univ. Press, 2018).
Lee, M. K., Jain, A., Cha, H. J., Ojha, S. & Kusbit, D. Procedural justice in algorithmic fairness: leveraging transparency and outcome control for fair algorithmic mediation. Proc. ACM Hum.–Comp. Interact. 3, 182 (2019).
Weller, A. Challenges for transparency. Preprint at https://arxiv.org/abs/1708.01870 (2017).
Shokri, R., Strobel, M. & Zick, Y. Exploiting transparency measures for membership inference: a cautionary tale. In The AAAI Workshop on Privacy-Preserving Artificial Intelligence (PPAI) (eds Fioretto, F. et al.) 17 (AAAI, 2020).
Shokri, R., Strobel, M. & Zick, Y. On the privacy risks of model explanations. Preprint at https://arxiv.org/abs/1907.00164 (2019).
Lahoti, P. et al. Fairness without demographics through adversarially reweighted learning. In Advances in Neural Information Processing Systems 33 (eds Larochelle, H. et al.) 728–740 (2020).
Dwork, C., Hardt, M., Pitassi, T., Reingold, O. & Zemel, R. Fairness through awareness. In Proc. 3rd Innovations in Theoretical Computer Science Conf. (eds Goldwasser, S.) 214–226 (ACM, 2012).
Hardt, M., Price, E. & Srebro, N. Equality of opportunity in supervised learning. In Proc. 30th Intl Conf. on Neural Information Processing Systems (eds Lee, D. et al.) 3323–3331 (Curran, 2016).
Werner, D. Nothing About Us Without Us: Developing Innovative Technologies for, by and with Disabled Persons (Healthwrights, 1998).
Charlton, J. I. Nothing About Us Without Us: Disability Oppression and Empowerment (Univ. California Press, 1998).
Costanza-Chock, S. Design justice, A.I., and escape from the matrix of domination. J. Design Sci. https://doi.org/10.21428/96c8d426 (2018).
Scott, J. C. Seeing Like a State (Yale Univ. Press, 2008).
Lazer, D. et al. Meaningful measures of human society in the twenty-first century. Nature https://doi.org/10.1038/s41586-021-03660-7 (2021).
Goel, S., Anderson, A., Hofman, J. & Watts, D. J. The structural virality of online diffusion. Manage. Sci. 62, 180–196 (2016).
Aral, S. & Nicolaides, C. Exercise contagion in a global social network. Nat. Commun. 8, 14753 (2017).
Eagle, N., Macy, M. & Claxton, R. Network diversity and economic development. Science 328, 1029–1031 (2010).
Contractor, N. in The Oxford Handbook of Networked Communication (eds Welles, B. F. and González-Bailón, S.) (Oxford Univ. Press, 2018).
Contractor, N., Monge, P. R. & Leonardi, P. M. Multidimensional networks and the dynamics of sociomateriality: bringing technology inside the network. Int. J. Commun. 5, 682–720 (2011).
Rahwan, I. et al. Machine behaviour. Nature 568, 477–486 (2019). This paper argues that studies of machine behaviour are necessary to control AI-enabled systems and to avoid harm.
Salganik, M. J. Bit By Bit: Social Research in the Digital Age (Princeton Univ. Press, 2017). This book gives an overview of methods deployed in computational social science.
Baeza-Yates, R. Big data or right data? In Proc. 7th Alberto Mendelzon Intl Workshop on Foundations of Data Management (eds Bravo, L. & Lenzerini, M.) 14 (CEUR-WS, 2013).
Schwarz, G. & Stensaker, I. Time to take off the theoretical straightjacket and (re-)introduce phenomenon-driven research. J. Appl. Behav. Sci. 50, 478–501 (2014).
Mathieu, J. E. The problem with [in] management theory. J. Organ. Behav. 37, 1132–1141 (2016).
Watts, D. Should social science be more solution-oriented? Nat. Hum. Behav. 1, 15 (2017).
Stier, S., Breuer, J., Siegers, P. & Thorson, K. Integrating survey data and digital trace data: key issues in developing an emerging field. Soc. Sci. Comput. Rev. 38, 503–516 (2020).
Mellon, J. Internet search data and issue salience: the properties of google trends as a measure of issue salience. J. Elections Public Opin. Parties 24, 45–72 (2014).
Stier, S., Bleier, A., Lietz, H. & Strohmaier, M. Election campaigning on social media: politicians, audiences, and the mediation of political communication on Facebook and Twitter. Polit. Commun. 35, 50–74 (2018).
Bernard, H. R., Killworth, P. D. & Sailer, L. Informant accuracy in social-network data: V. An experimental attempt to predict actual communication from recall data. Soc. Sci. Res. 11, 30–66 (1982).
Prince, S. A. et al. A comparison of direct versus self-report measures for assessing physical activity in adults: a systematic review. Int. J. Behav. Nutr. Phys. Act. 5, 56 (2008).
Scharkow, M. The accuracy of self-reported internet use—a validation study using client log data. Commun. Methods Meas. 10, 13–27 (2016).
Boase, J. & Ling, R. Measuring mobile phone use: self-report versus log data. J. Comput. Mediat. Commun. 18, 508–519 (2013).
Revilla, M., Ochoa, C. & Loewe, G. Using passive data from a meter to complement survey data in order to study online behavior. Soc. Sci. Comput. Rev. 35, 521–536 (2017).
Elmer, T., Chaitanya, K., Purwar, P. & Stadtfeld, C. The validity of RFID badges measuring face-to-face interactions. Behav. Res. Methods 51, 2120–2138 (2019).
Sapiezynski, P., Stopczynski, A., Lassen, D. D. & Lehmann, S. Interaction data from the Copenhagen Networks Study. Sci. Data 6, 315 (2019).
Groves, R. M. & Lyberg, L. Total survey error: past, present, and future. Public Opin. Q. 74, 849–879 (2010).
Coravos, A., Chen, I., Gordhandas, A. & Stern, A. D. We should treat algorithms like prescription drugs. Quartz, https://qz.com/1540594/treating-algorithms-like-prescription-drugs-could-reduce-ai-bias/ (February 2019).
Arnold, M. et al. FactSheets: increasing trust in AI services through supplier’s declarations of conformity. Preprint at https://arxiv.org/abs/1808.07261 (2019).
Bender, E. & Friedman, B. Data statements for natural language processing: toward mitigating system bias and enabling better science. Trans. Assoc. Comput. Linguist. 6, 587–604 (2018).
Mitchell, M. et al. Model cards for model reporting. In Proc. Conf. Fairness, Accountability, and Transparency (eds Morgenstern, J. & boyd, d.) 220–229 (ACM, 2019). This paper argues for higher standards in documenting machine learning models.
Kuh, A., Petsche, T. & Rivest, R. Learning time-varying concepts. In Advances in Neural Information Processing Systems 3 (eds Lippmann, R. P. et al.) 183–189 (Morgan-Kaufmann, 1991).
Bartlett, P. L., Ben-David, S. & Kulkarni, S. R. Learning changing concepts by exploiting the structure of change. Mach. Learn. 41, 153–174 (2000).
Gama, J., Žliobaitundefined, I., Bifet, A., Pechenizkiy, M. & Bouchachia, A. A survey on concept drift adaptation. ACM Comput. Surv. 46, 44 (2014).
Abbasi, M., Friedler, S. A., Scheidegger, C. & Venkatasubramanian, S. Fairness in representation: quantifying stereotyping as a representational harm. In Proc. 2019 SIAM Intl Conf. Data Mining (eds Berger-Wolf, T. & Chawla, N.) 801–809 (SIAM, 2019).
Abebe, R. et al. Roles for computing in social change. In Proc. 2020 Conf. Fairness, Accountability, and Transparency (eds Castillo, C. & Hildebrandt, M.) 252–260 (ACM, 2020).
Hampton, L. M. Black feminist musings on algorithmic oppression. In Proc. 2021 Conf. Fairness, Accountability, and Transparency (eds Elish, M. C. et al.) 1 (ACM, 2021).
De-Arteaga, M., Fogliato, R. & Chouldechova, A. A case for humans-in-the-loop: decisions in the presence of erroneous algorithmic scores. In Proc. 2020 CHI Conf. Human Factors in Computing Systems (eds Bernhaupt, R. et al.) 1–12 (ACM, 2020).
Boyarskaya, M., Olteanu, A. & Crawford, K. Overcoming failures of imagination in AI infused system development and deployment. Preprint at https://arxiv.org/abs/2011.13416 (2020).
Nanayakkara, P., Diakopoulos, N. & Hullman, J. Anticipatory ethics and the role of uncertainty. Preprint at https://arxiv.org/abs/2011.13170 (2020).
Friedman, B. Value-sensitive design. Interaction 3, 16–23 (1996).
Olteanu, A., Diaz, F. & Kazai, G. When are search completion suggestions problematic? Proc. ACM Hum.–Comp. Interact. 4, 1–25 (2020).
Jiang, J. A., Wade, K., Fiesler, C. & Brubaker, J. R. Supporting serendipity: opportunities and challenges for human–AI collaboration in qualitative analysis. Proc. ACM Hum.–Comp. Interact. 5, 1–23 (2021).
Churchill, E., van Allen, P. & Kuniavsky, M. Designing AI: introduction. Interaction 25, 34–37 (2018).
Selbst, A. D. boyd, d., Friedler, S., Venkatasubramanian, S. & Vertesi, J. Fairness and abstraction in sociotechnical systems. In Proc. Conf. Fairness, Accountability, and Transparency (eds Morgenstern, J. & boyd, d.) 59–68 (ACM, 2019).
Barocas, S., Biega, A. J., Fish, B., Niklas, J. & Stark, L. When not to design, build, or deploy. In Proc. 2020 Conf. Fairness, Accountability, and Transparency (eds Castillo, C. & Hildebrandt, M.) 695–695 (ACM, 2020).
Monge, P. & Contractor, N. Theories of Communication Networks (Oxford Univ. Press, 2003).
Glaser, B. & Strauss, A. The Discovery of Grounded Theory: Strategies for Qualitative Research (Aldine de Gruyter, 1967).
Bryant, A. Re-grounding grounded theory. J. Inf. Technol. Theory Appl. 4, 25–42 (2002).
Charmaz, K. Constructing Grounded Theory: A Practical Guide Through Qualitative Analysis (Sage, 2006).
Timmermans, S. & Tavory, I. Theory construction in qualitative research: from grounded theory to abductive analysis. Sociol. Theory 30, 167–186 (2012).
Nelson, L. Computational grounded theory: a methodological framework. Sociol. Methods Res. 49, 3–42 (2020). This paper proposes a methodological framework to combine expert human knowledge and hermeneutic skills with the processing power and pattern recognition of computers.
McFarland, D., Lewis, K. & Goldberg, A. Sociology in the era of big data: the ascent of forensic social science. Am. Sociol. 47, 12–35 (2015).
Radford, J. & Joseph, K. Theory in, theory out: the uses of social theory in machine learning for social science. Front. Big Data 3, 18 (2020).
Macy, M. & Willer, R. From factors to actors: computational sociology and agent-based modeling. Annu. Rev. Sociol. 28, 143–166 (2002).
Smith, E. R. & Conrey, F. R. Agent-based modeling: a new approach for theory building in social psychology. Pers. Soc. Psychol. Rev. 11, 87–104 (2007).
Keuschnigg, M., Lovsjö, N. & Hedström, P. Analytical sociology and computational social science. J. Comp. Soc. Sci. 1, 3–14 (2018).
Hedström, P. & Bearman, P. in The Oxford Handbook of Analytical Sociology (eds Hedström, P. & Bearman, P.) (Oxford Univ. Press, 2011).
Lemmerich, F. et al. Mining subgroups with exceptional transition behavior. In Proc. 22nd ACM SIGKDD Intl Conf. Knowledge Discovery and Data Mining (eds Krishnapuram, B. & Shah, M.) 965–974 (ACM, 2016).
Singer, P. et al. Why we read Wikipedia. In Proc. 26th Intl Conf. World Wide Web (eds Barrett, R. & Cummings, R.) 1591–1600 (ACM, 2017).
Aguera y Arcas, B., Mitchell, M. & Todorov, A. Physiognomy’s new clothes. Medium, https://medium.com/@blaisea/physiognomys-new-clothesf2d4b59fdd6a (6 May 2017).
We thank the reviewers for their contributions to this work.
The authors declare no competing interests.
Peer review information Nature thanks Ceren Budak, Johan Ugander and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.
About this article
Cite this article
Wagner, C., Strohmaier, M., Olteanu, A. et al. Measuring algorithmically infused societies. Nature 595, 197–204 (2021). https://doi.org/10.1038/s41586-021-03666-1
This article is cited by
Scientific Reports (2022)
Nature Human Behaviour (2022)
Cognitive Research: Principles and Implications (2021)