A montage of small drawn and painted pictures of cows.

Pictures of cows repeated across a range of paintings attributed to Jan Brueghel the Elder.Credit: Eric Monson and Elizabeth Honig

When art thieves broke into a church in northwestern Italy this March, they thought they were stealing a seventeenth-century work by the Flemish painter Pieter Brueghel the Younger. In fact, police in the small Ligurian town of Castelnuovo Magra had been tipped off, and swapped The Crucifixion, valued at €3 million (US$3.3 million), for a cheap copy.

In fairness to the gang, a number of Brueghel’s works do look nearly interchangeable. A similar Crucifixion attributed to the same artist hangs in the Philadelphia Museum of Art in Pennsylvania. And Brueghel probably copied both paintings from another by his trailblazing father, Pieter Bruegel the Elder — whose work also heavily influenced his other son, Jan (known as Jan Brueghel the Elder). With a dynasty of prolific artists, some replicating each other’s and their own works, attribution can be nightmarish.

Elizabeth Honig studies these complexities to build up a better picture of who was painting what and influencing whom in northern Renaissance art. And now, she has turned for help to the untiring eye of a computer.

Honig — an art historian at the University of California, Berkeley — has a database of more than 1,500 digitally reproduced Brueghel pictures, most attributed to Jan. In 2016, she initiated an unusual collaboration with artificial intelligence (AI) researchers in France and the United States, deploying state-of-the-art computer vision to help in analysing similarities and tracing them from work to work. Other art historians are also seeing opportunities in harnessing machine learning to provide empirical support for theories and ideas previously confined to the subjective eyes of the beholders.

The computer, says Honig, can pick up “so many more details, so much more easily”. Take windmills: hundreds of pictures featuring them fill her Brueghel database. The algorithm has picked up identical images of the structures in multiple paintings. It can even show when a replica has been flipped. And it has helped to pinpoint exact copies of lions, dogs and other figures. The workshops of many Renaissance artists were co-working spaces, so the computer technique helps Honig to piece together how different artists, in the family or not, might collaborate. “Rubens comes in and does some figures, and then Jan Breughel comes in and does the horses, the dog and the lion, because he’s ‘Mister Animal’,” Honig says. “And so they fit the things together.”

Many art historians surmised, on the basis of records and close observation, that this is what happened with numerous paintings by the younger Brueghels. The computer helps to prove it. Hong says: “It addresses a lot of questions about the process of production.”

The computer scientists bring their own questions to the project. To them, Honig’s collection is a perfect data set with which to stretch their algorithms. Working with paintings challenges a program’s pattern-matching capacity, says Mathieu Aubry, a specialist in computer vision and deep learning at École des Ponts ParisTech in France. The difficulty hinges on differences in media and colour. Computer vision can’t, he explains, “recognize that a house is the same in a drawing and an oil painting if it has not been trained to do so”. The sharp linearity of draughtsmanship and relatively blurred edges in oil painting can confound algorithms.

It would take too long to annotate identical objects or teach the computer to look for certain similarities, such as shape. So Aubry and his colleagues used a technique called unsupervised deep learning, in which the algorithm is shown the pictures and finds similarities for itself. The results could feed into more practical applications of AI vision, he says, such as self-driving cars.

His team posted the results — for instance, a cannon and a chandelier both repeated across five separate pictures — on the arXiv preprint server in March (X. Shen et al. Preprint at https://arxiv.org/abs/1903.02678; 2019). And next week, they will present them at the 2019 Conference on Computer Vision and Pattern Recognition in Long Beach, California. Although unsupervised deep learning typically takes a lot of computer power, Aubry says, it is mostly immune from human preconceptions. So it’s a good way to avoid biases such as the tendency to focus on the main features of a picture.

Telling trends

Similar technology is being used at Rutgers University in Piscataway, New Jersey, to map how style is defined and develops over time in artists as diverse as Rembrandt van Rijn and the Russian avant-garde artist Kazimir Malevich. “We had theories but they’re not provable,” says art historian Marian Mazzone, a member of the Rutgers Art and Artificial Intelligence Laboratory. “Computer science may be a tool that can help me empirically answer some of these questions.”

Working with lab head Ahmed Elgammal, she has produced a digital analysis of 77,000 of works of art spanning five centuries, from the Renaissance to pop art (A. Elgammal et al. Preprint at https://arxiv.org/abs/1801.07729; 2018). To the team’s astonishment, the computer — also using unsupervised learning — put the artworks into chronological order.

The project confirmed a theory of eminent twentieth-century art historian Heinrich Wolffin. He argued that shifts in artistic style could be analysed and categorized according to five binary characteristics. One was whether the work was ‘linear’ (contour-led, as in the work of Sandro Botticelli) or ‘painterly’ (reliant more on brushstrokes denoting light and shadow, as in the paintings of Tintoretto). Elgammal argues that AI allows art history to be treated, for the first time, as a predictive science that compares theory with observations.

Elsewhere, AI is being harnessed to address a perennial problem of material legacy that underpins art history: deterioration. For instance, the Verus Art system from start-up Arius Technology in Vancouver, Canada, is deploying a 3D scan–print system — initially devised to study damage to Leonardo da Vinci’s Mona Lisa — to replicate artworks precisely, down to textured brushstrokes and pigment hues. Intended for education, outreach and archives, the ‘backed-up’ paintings might have another use: foiling thieves more discerning than those fooled by Castelnuovo Magra’s cheap copy.