After various breakthroughs in data availability, hardware capabilities and algorithm innovations, deep neural networks began to outperform other computational approaches on benchmarks in computer vision and speech recognition around 2012. Arguably, the next big step was around 2018, with the development of the transformer architecture that lies at the heart of large language models such as BERT and GPT-3. In simple terms, transformers take a sequence as input and produce a new sequence as output. An essential trick is the use of an ‘attention’ mechanism that means the algorithm looks at an input sequence and decides which parts of the sequence are more important than others. This is not unlike how humans approach language processing: a reader does not process a paragraph word-for-word but might remember certain words as more important than others. This selective view allows for more complex long-range connections to be learned by the model. As a result, the approach is extremely versatile, as many tasks can be mapped as a language translation problem, and transformer models can be spotted in several exciting research developments. Examples are in identifying and classifying chemical reactions, in protein sequencing and design, and in genomics. Another important aspect of transformer-type approaches is that they can be self-supervised; after pre-training on large amounts of unlabelled data, they can tackle new tasks by fine-tuning.

Beyond scientific applications, transformer-based models have turned into powerful tools for the information age as companies such as Google, Facebook and OpenAI have the resources to massively scale up neural networks including transformers and to train these models on all of the internet. OpenAI’s GPT-3, a language model with 175 billion parameters, is so good at producing text upon simple prompts that OpenAI makes it available only via a controlled API given the risk of misuse. But large language models already power widely used search and recommender systems (such as Google’s), and are affecting what information many of us are exposed to.

A new centre formed by researchers at the Human-Centered Artificial Intelligence group at Stanford — the Center for Research on Foundation Models (CRFM) — calls for an investment in academic efforts and resources to create large neural network models to study their capabilities, limitations and impact on society. A 200-page white paper calls these large neural networks ‘foundation models’, as they are trained on broad data at scale and can be adapted to a wide range of downstream tasks. Indeed, GPT-3 can be used and adapted for a surprising range of tasks, such as producing images from text or code from simple text instructions.

However, these so-called foundation models also show many failures, even ones that are clearly nonsensical to humans. The models are brittle, and the larger they are, the less clear it is how they work. For such multi-purpose models, interpretability is even trickier as each task might require distinct explanations of how the model determines a specific answer. Cherry-picked examples can show impressive abilities, but small changes to the input can result in complete failure without a clear reason when or why the model would fail.

Many other problems emerge for foundation models and call for ethical reflection. Given that large language models are trained on data available on the web, they can propagate existing stereotypical biases in society. A recent study shows, for example, that GPT-3 contains persistent anti-Muslim bias. Another major problem is the environmental cost — a paper from Emma Strubell et al. in 2019 already highlighted the exploding carbon emissions for training large language models. Emily Bender and others also discussed such problems in the now famous paper ‘On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?’

In a timely workshop organized by the new centre at Stanford, many of these issues were debated. A question that came up — echoing concerns from the paper by Bender et al. — is whether large language models should be built at all? As panel speaker Su-Lin Blodgett said, it is concerning how rapidly these large models have become entrenched and that there is a sense of inevitability about this, while we hardly have an idea of what risks may arise. The white paper from CRFM covers many practical and ethical concerns about foundation models. However, the issue is whether there is currently enough focus on the sheer human and environmental cost of further scaling up large language models, involving scraping data at scale from the internet without knowledge or explicit consent, containing all of society’s biases, and using massive computational resources, all for the development of tools that empower the tech industry.

Large neural network models are already under development and deployed by big tech companies, and any effort to address this power asymmetry — which seems the intention of the new centre at Stanford — seems welcome. There is a need to develop professional norms for responsible research in large language (or foundation) models, which should include, among others, guidelines for data curation, auditing processes and an evaluation of environmental cost. These big questions should not be left to the tech industry.