When A Machine Learning Algorithm Studied Fine Art Paintings, It Saw Things Art Historians Had Never Noticed
The task of classifying pieces of fine art is hugely complex. When examining a painting, an art expert can usually determine its style, its genre, the artist and the period to which it belongs.
Art historians often go further by looking for the influences and connections between artists, a task that is even trickier.
So the possibility that a computer might be able to classify paintings and find connections between them at first glance seems laughable.
And yet, that is exactly what Babak Saleh and pals have done at Rutgers University in New Jersey.
These guys have used some of the latest image processing and classifying techniques to automate the process of discovering how great artists have influenced each other.
They have even been able to uncover influences between artists that art historians have never recognized until now.
The way art experts approach this problem is by comparing artworks according to a number of high-level concepts such as the artist’s use of space, texture, form, shape, color and so on.
Experts may also consider the way the artist uses movement in the picture, harmony, variety, balance, contrast, proportion and pattern.
Other important elements can include the subject matter, brushstrokes, meaning, historical context and so on. Clearly, this is a complex business.
So it is easy to imagine that the limited ability computers have for analyzing two-dimensional images would make this process more or less impossible to automate. But Salah and co show how it can be done.
At the heart of their method, is a new technique developed at Dartmouth College in New Hampshire and Microsoft research in Cambridge, UK, for classifying pictures according to the visual concepts that they contain.
These concepts are called classemes and include everything from simple object description such as duck, frisbee, man, wheelbarrow to shades of color to higher-level descriptions such as dead body, body of water, walking and so on.
Comparing images is then a process of comparing the words that describe them, for which there are a number of well-established techniques.
For each painting, they limit the number of concepts and points of interest generated by their method to 3000 in the interests of efficient computation.
This process generates a list of describing words that can be thought of as a kind of vector. The task is then to look for similar vectors using natural language techniques and a machine learning algorithm.
Determining influence is harder though since influence is itself a difficult concept to define. Should one artist be deemed to influence another if one painting has a strong similarity to another?
Or should there be a number of similar paintings and if so how many?
So Saleh and co experiment with a number of different metrics. They end up creating two-dimensional graphs with metrics of different kinds on each axis and then plotting the position of all of the artists in this space to see how they are clustered.
The results are interesting. In many cases, their algorithm clearly identifies influences that art experts have already found.
For example, the graphs show that the Austrian painter Klimt is close to Picasso and Braque and indeed experts are well acquainted with the idea that Klimt was influenced by both these artists.
The algorithm also identifies the influence of the French romantic Delacroix on the French impressionist Bazille, the Norwegian painter Munch’s influence on the German painter Beckmann and Degas’ influence on Caillebotte.
The algorithm is also able to identify individual paintings that have influenced others.
It picked out Georges Braque’s Man with a Violin and Pablo Picasso’s Spanish Still Life: Sun and Shadow, both painted in 1912 with a well-known connection as pictures that helped found the Cubist movement.
And yet a visual inspection shows a clear link. The yellow circles in the images below show similar objects, the red lines show composition and the blue square shows a similar structural element, say Saleh and co.
That is interesting stuff. Of course, Saleh and co do not claim that this kind of algorithm can take the place of an art historian.
After all, the discovery of a link between paintings in this way is just the starting point for further research about an artist’s life and work.
But it is a fascinating insight into the way that machine learning techniques can throw new light on a subject as grand and well studied as the history of art.
Please like, share and tweet this article.
Pass it on: New Scientist