Over the past 100 years, social science has generated a tremendous number of theories on the topics of individual and collective human behaviour. However, it has been much less successful at reconciling the innumerable inconsistencies and contradictions among these competing explanations, a situation that has not been resolved by recent advances in ‘computational social science’. In this Perspective, I argue that this ‘incoherency problem’ has been perpetuated by an historical emphasis in social science on the advancement of theories over the solution of practical problems. I argue that one way for social science to make progress is to adopt a more solution-oriented approach, starting first with a practical problem and then asking what theories (and methods) must be brought to bear to solve it. Finally, I conclude with a few suggestions regarding the sort of problems on which progress might be made and how we might organize ourselves to solve them.
As a sociologist who spends a lot of time in the company of physicists, computer scientists and other outsiders to my field, I am often asked a question of the sort: “What is the social science perspective on X?”, where X is some topic of interest. To a social scientist, the question sounds hopelessly naïve: for any topic X, social science has dozens, if not hundreds, of perspectives, but no single perspective on which there is anything close to universal agreement. Nevertheless, I would argue that it is worth taking the question seriously, if only because it highlights an important difference between the social and physical/engineering sciences.
Physicists disagree of course — for example, about the best way to reconcile general relativity with quantum mechanics, or the best explanation for the ‘missing mass’ problem in cosmology — but overall there is tremendous agreement both on what physicists know about the universe (Newtonian mechanics, thermodynamics, electromagnetism, optics, special and general relativity, statistical mechanics, particle physics and so on) and where the remaining areas of uncertainty lie. By contrast, any representative cross-section of social scientists would have difficulty agreeing on almost any question at all, including which questions were the most important to be agreed upon. It could be argued that in economics there exist certain specialized subfields, such as mechanism design applied to auctions1,2 and matching markets3,4, that comprise cumulative bodies of self-consistent, empirically validated theory that have even proven useful in practice. But no such claims can be sustained for economics in general, let alone for problems of interest to the social sciences broadly.
Comparing the social sciences unfavourably to physics is of course a game with a long and, I would argue, quite unproductive history5. However, my thesis differs from the usual critique that social science should strive to be more like physics by identifying general principles. I shall argue that the problem with social science is not so much that it has one theory for one thing and another theory for another thing6, but rather that it has many theories for the very same thing. Even worse, these theories — although often interesting and plausible when considered individually — are fundamentally incoherent when viewed collectively. I then argue that this incoherency problem arises not only because of a lack of appropriate data for evaluating social scientific theories, but also because of the institutional and cultural orientation of social-science disciplines, which have historically emphasized the advancement of particular theories over the solution of practical problems. Finally, I argue that one possible solution to the incoherency problem is to reject the traditional distinction between basic and applied science, and instead seek to advance theory specifically in the service of solving real-world problems.
Before proceeding, however, let me clarify two points of possible confusion. First, I am not arguing that all, or even most, of social science should become solution-oriented. Social science can serve many purposes — for example, the field can challenge common-sense assumptions about the nature of social reality7,8,9, provide rich descriptions of lived experience10,11,12, inspire new ways of thinking about human behaviour13,14 and shed light on specific empirical puzzles15,16 — that do not directly address practical problems but can still provide valuable insight. My argument is not that social scientists should stop pursuing these other objectives in favour of solving practical problems; only that collectively we should pay more attention than we do to the latter. Second, I am also not suggesting that social scientists do not already devote themselves to solving practical problems: many do, especially in policy-relevant areas like education17, health care18, poverty19 and government20. Rather, what I am suggesting is that social scientists can profitably view the solution of practical problems as a mechanism for improving the coherency of social science itself.
More data is not the answer
What accounts for this state of affairs and what, if anything, can be done about it? One popular conjecture is that historically social science has not had access to the right kind of data29,30. Social phenomena, the argument goes, are inherently emergent properties of complex, multi-scale networked systems. Simply observing networks and behaviour at multiple scales over extended intervals of time is therefore already an extraordinary undertaking, while establishing cause and effect through ‘macro’ social experiments is even more difficult, and often impossible. In other words, one possible reason why social science seems less ‘scientific’ than we would like is simply that our ability to propose theories has for so long outstripped our ability to test them.
If this were the whole of the problem, then the era of ‘big data’ should be the solution. For the first time in history, the digital traces of ordinary everyday interactions — sending e-mails, checking social media, buying goods and services online, consuming content, expressing opinions — in principle allow us to observe individual-level behaviour and interactions on a large scale and over extended periods of time. At the same time, we can also conduct experiments on greater scales and with increasing complexity, whether in ‘virtual labs’31,
To some extent this hope has been realized: among the countless papers that exploit digital data and experiments, there are many that are very good and some that establish genuinely new and interesting ideas. Some even explicitly set out to test existing theories against novel data with the goal of deciding among competing explanations or simply placing limits on what we can hope to explain. All of this work represents exciting progress. And yet, ten years into the era of what is now called computational social science, it seems to me that more data, and even better data, is not enough. Nor has the influx of physicists and computer scientists into the social sciences over the past two decades clearly ameliorated the coherency problem. Far from the social sciences acquiring a coherent physics-inspired core of empirically validated theoretical knowledge, they have instead acquired a whole new batch of physics-inspired models that have, if anything, added to the confusion.
Identifying Goldilocks problems
A major challenge for solution-oriented social science is, therefore, to identify a set of problems that are not so large and complex as to require a total theory of social, economic, and political life, but are still of sufficient difficulty to justify a genuinely scientific approach. Even better would be problems that are modular, in the sense that they can be expressed in a succession of increasingly ambitious versions. By starting with the most limited version of a problem and progressing up the hierarchy of complexity, one could hope to make concrete progress on a realistic timescale, while still maintaining a grand vision of ultimate progress. Finally, while the research itself would be understandable only to experts, it is important that no particular expertise be required to understand the problem statement or to check that proposed solutions work.
Identifying problems that have this ‘Goldilocks’ property of being neither too easy nor too hard is difficult, but one possible direction is to draw inspiration from engineering, and place more emphasis on building tangible devices and systems that have specific, well-defined properties. For example, the problem of building a driverless vehicle is easy to understand (a car that drives itself!) and relatively easy to evaluate (does it drive itself?), but is of sufficient difficulty to require fundamental advances in artificial intelligence (AI). By analogy, social scientists might propose building instruments for measuring social sentiment, or platforms for supporting political deliberation or economic exchange, or compilers that enable human workers and machines to collaborate on complex tasks. Solution-oriented social science, however, need not be restricted to solutions with direct engineering analogues. For example, systems of best practices could be developed, say for management or hiring, that are grounded in large-scale comparative observational studies, field experiments and algorithmic decision aids. Systems for generating and testing the policy implications of competing theories — with respect, say, to social influence or collective problem-solving — would also qualify.
Another potential approach is inspired by the ‘common-task framework’51,52, originally developed in AI research, according to which researchers compete to solve specific tasks (for example, machine translation), solutions are benchmarked using agreed-upon performance metrics (word error rate) and performance is evaluated on publicly available datasets (Canadian Hansards) by an independent referee (NIST, the National Institute of Standards and Technology). Perhaps surprisingly, by limiting the scope of the problems to be solved, the common task framework has yielded extraordinary advances in the performance of machine-learning algorithms over the past 30 years, ultimately producing working AI services such as Google and Skype translation systems, Siri and Cortana. Although adapting the common task framework to social science is not without complications — for example AI researchers may be satisfied with predictive accuracy whereas social scientists also typically seek to understand causal mechanisms — there is no reason in principle why ‘Netflix style’ contests could not be conducted using social datasets, potentially with important scientific and policy consequences.
To conclude, let me restate that I am not arguing that solving problems is the only productive mode of social-science research, nor am I am arguing that social scientists never take it upon themselves to solve practical problems. What I am arguing, however, is that placing more emphasis on use-inspired research would benefit social science in two ways. First, it would force social scientists to deal with the incoherency problem, thereby advancing fundamental scientific understanding of the social world. And second, it would help social science to be more visibly useful to the world, thereby improving its status with an increasingly sceptical public53, as well as generating excitement and interest among students who might otherwise choose the natural sciences, engineering or some other profession entirely. Finally, concrete progress need not require sweeping changes in the organization of social science. If one could identify even a handful of Goldilocks problems, even a single research centre or institute could make exciting progress within a decade. If that happened, other institutes and centres might be inspired to follow, much as a single institute — the Laboratory of Molecular Biology (LMB) in Cambridge — jump-started the field of molecular biology and inspired many other similar institutes to follow54. Social science is of course different from molecular biology, and analogies with past successes are always at risk of being overblown. Nevertheless, given the limited downside of just one group of people trying to do something different in just one place for a limited time, and the considerable upside if they succeed, my vote is that it is worth the risk.