This site uses AI to find Tate Britain's paintings that look like photos from today's news
| on September 05, 2016
Tate Britain has launched an artificial intelligence (AI) programme that compares and matches Reuters photojournalism with British art from the Tate collection.
Does the painting above look like the photo above it? They're not an exact match, but considering an AI managed to work out that the photo shows a woman (even though it can't see her face) with bare arms and legs who's not standing up - and match it to a painting from the Tate Britain that has many of the same characteristics and tonal range - it's an undeniably impressive achievement.
A winning team from Fabrica designed the project for the IK Prize - awarded by the Tate to a team, company or individual who can create a way in which art at the Tate could be explored using digital technology. This year’s brief was to focus on artificial intelligence.
In collaboration with Microsoft, digital creatives, researches and software developers, Fabrica put their heads together to come up with the result – aptly named Recognition.
Images with close similarity, decided by four categories of recognition - object recognition, facial recognition, composition recognition and context recognition - are selected as a match and displayed in Recognition's gallery.
The project will be displayed at Tate Britain as well as in a constantly expanding virtual gallery. It represents a rare splice of modern and archaic images causing a strange reflection on humanity, current affairs and art.
Below you can see the criteria for how the AI has matched a photo of eunuchs from Mumbai from last month to a c1660 painting by Sir Peter Lely.
Participants are encouraged to compare the machine’s matches with their own, and invited to help retrain the algorithm to see if AI can learn from the numerous human interpretations of an image.
The results will be showcased on the virtual gallery website at the end of the project – which runs from September 2 to November 27.
Fabrica - a research centre/art school in Treviso, Italy that's owned by Benetton - also worked alongside AI specialists such as JoliBrain. They were given a £15,000 prize and £90,000 production budget.
"Ongoing advances in AI technologies are enabling new partnerships between people and machines," says Eric Horvitz from the Microsoft Research Lab at Redmond.
"The Recognition project was motivated by the promise of this kind of collaboration. We envision a world where humans and machines work together in new ways to do amazing things."
But how does Recognition work?
Artificial intelligence sounds complex right? We thought so, so let's try and break it down.
Recognition processes the Reuters images in four different ways – through object, composition, context and facial recognition.
Algorithms rely on matching, learning, or pattern recognition using appearance-based analysis to determine object recognition. Developed by JoliBrain using DeepDetect and Densecap, the neural network finds objects from the image and tries to label them by crafting a short sentence.
Composition recognition uses the same technology by JoliBrain to focus on shapes and structures, visual layout and colours.
Facial recognition is provided by Microsoft’s Cognitive Services’ Computer Vision and Emotion APIs.
Context composition analyses titles, dates, tags, and descriptions associated with each image. By reading this text, Recognition learns how to write a caption for each match.
Images with close similarity in these four categories are selected as a match and displayed in Recognition’s gallery.
Viewers can learn why each matching artwork was chosen and can share their favourite matches online – deciding whether a machine can help to shed light on a new perspective of art.
You can see what images Recognition has matched so far here.