Is Artificial Intelligence capable of restoring lost art masterpieces?

Is AI able to give us a glimpse of lost masterpieces?

Machine learning was used in recent projects to recover Rembrandt and Klimt paintings. These projects raise questions about how computers can comprehend art.

Fire claimed three Gustav Klimt's most controversial paintings in 1945. The "Faculty Paintings", as they were called, were commissioned in 1894 by the University of Vienna. They were unlike any other Austrian symbolist's work. They were immediately rejected by critics who were shocked at their radical departure from the original aesthetics. Klimt quit the project after the university professors rejected them. The works were soon accepted into other collections. They were stored in a castle north Vienna during World War II for safekeeping. However, the castle was destroyed and the paintings probably went with it. Today, only a few black-and-white photos and writings are left. They are staring at me.

The paintings are not what they seem. Franz Smola, a Klimt expert and Emil Wallner (a machine learning researcher), spent six months combining their skills to restore Klimt's work. It was a tedious process that began with black-and-white photographs and then included artificial intelligence and scores more information about Klimt's art in an effort to recreate the lost paintings. These are the results that Wallner and Smola are showing me, and even they are stunned by the stunning technicolor images produced by the AI.

Let's be clear: This AI is not bringing back Klimts original works. Smola quickly points out that "it's not a process to recreate the actual colors, but it is recolorizing the photos." "The medium of photography has already been an abstraction from the actual works." Machine learning provides a glimpse into something that was thought to have disappeared for many decades.

Wallner and Smola find this charming, but not all people support AI filling these voids. Machine learning is able to recreate lost or damaged works, but this idea is controversial, just like the Faculty Paintings. Ben Fino-Radin, an art conservator, says that machine learning in conservation is his main concern. "This is because of the many ethical and moral questions that have plagued the machine learning field."

There are many questions surrounding the technology used to revive human art. No algorithm can generate authorial intent, even if it was perfect AI. This topic has been debated for centuries. Before Klimt's paintings were damaged, Walter Benjamin, an essayist, opposed mechanical reproduction.

Yet, AI has a lot of potential. Klimt's growth as an artist was influenced by the Faculty Paintings. They were a bridge between his earlier, more traditional paintings and his later, more radical works. However, the mystery surrounding what they looked like in full colour has remained. This was the problem Wallner and Smola were trying to solve. Their project was not about creating perfect reproductions, but about showing a glimpse at what was missing.

Wallner created and trained a three-part algorithm to accomplish this. The algorithm was first fed a hundred thousand images from the Google Arts and Culture library. This enabled it to understand composition, artwork, and objects. The next step was to study Klimt's artworks. Wallner says that this creates a bias towards Klimt's colors and his motifs of the period. Finally, the AI was given color clues that pointed to particular parts of the paintings. These clues were not based on color references. Smola, a Klimt expert, was amazed at the amount of detail that the writings from the time revealed. The paintings were so bizarre and egregious that critics were compelled to write detailed descriptions of them, down to the artist’s choice of colors. Simon Rein, project manager, says that it is an irony in history. The fact that the paintings were rejected and caused scandal puts us in a better place to restore them, because we have so much documentation. These data points can be fed into the algorithm to create a more accurate picture of the way these paintings looked at the time.

That accuracy is possible by combining the algorithm with Smola’s expertise. He discovered that Klimt's work from this time period has strong patterns and consistency. The Faculty Paintings were a study of Klimt's paintings before and after they were completed. This provided clues as to the common themes and colors that were prevalent in his work. Historical evidence supports even the surprising discoveries Wallner and Smola made. Critics noted Klimt's use of a rare red in Klimt's palette when he first displayed his paintings. The Three Ages of Woman was painted shortly after the Faculty Paintings. It boldly uses a red that Smola believes is the same as the one that caused a stir when it was first shown in the Faculty Paintings. A number of writings from that time raise concerns about the green sky in a Faculty Painting. Combining these writings and Smola’s knowledge about Klimt’s specific palette of greens is what created one of the most surprising images from the AI.

Klimt is not the only work that is being revived by AI. Robert Erdmann is a senior scientist at Rijksmuseum Amsterdam who uses machine learning to solve the mystery surrounding Rembrandt van Rijn’s 1642 masterpiece The Night Watch. This is part of an ongoing conservation and research program called Operation Night Watch. The current painting measures approximately 15 feet in width and 12 feet high, but it is much smaller than Rembrandt van Rijn's original. To fit into a new place, it was trimmed on four sides in 1715. The deepest cut was two feet from the left. Erdmann believed machine learning could help Rembrandt to decode the original vision of the painting. However, they were not found.

Erdmann's strongest data point was a Gerrit Lundens 17th-century copy. This painter is known for faithfully reproducing old masters and included Rembrandt parts that were missing. Erdmann used three neural networks in his design. Erdmann used the first to map out visually matching points across the two paintings. The Rembrandt was faithfully represented by the Lundens when they were viewed side-by-side and scaled to the exact same size. Erdmann switched between the digital overlays of the two paintings to see how much distortion and stretching were in the copy. This is where the second network comes in. The second network warped Lundens' image by stretching and compressing some parts, until the majority of the spatial distortion was gone.

The Rembrandt and the Lundens were thus very closely linked. These are two works that were created by artists who have their own style. Rectifying this required a third step, which Erdmann calls "sending the neural net to art school". Through backpropagation, it learned to render Lundens in Rembrandt's style. Iteration after iteration was made, moving closer to its goal until it reached its plateau. It was a perfect match. It wasn't a perfect match.

AI and machine-learning raise ethical and usage questions, just like any new technology. This includes decades-old artworks. Richard Rinehart is the director of Bucknell University's Samek Art Museum. He points out that technology has been used to determine our social contracts. AI may be unique in this aspect. He says that although techno-social contracts have been made unilaterally so far, AI might be able negotiate for itself. Technology has been at the core of conservation for centuries, in all material sciences, chemistry and color science. Rinehart says that although AI may be a significant change, the idea of applying technology to art is an accepted part of conservation, and self-criticism is a healthy part.

Fino-Radin, an art conservator, would love to see more self-criticism within the industry. But their concerns go deeper. While they are excited about the new creative possibilities this technology offers, they are concerned that it could be confused with conservation and restoration. Fino-Radin states that AI is not a restoration process. It's more like bringing back the art to life. This kind of work belongs to the field of Digital Art History.

Wallner and Smola are well aware of criticisms and will explain the Klimt project's limitations and scope. Wallner says, "We used the photos exactly as they were to ensure that we didn't depart too much from the original paintings." Erdmann explains that his reconstruction had the purpose of letting the public see Rembrandt's original composition. He emphasizes that "When I translate the Lundens copy to the style of Rembrandt the AI doesn’t have the ability put the life and genius that Rembrandt back into a painting." I'm not trying. It's not something I want to do. What you see today at the Rijksmuseum is the cropped Rembrandt painting. The extended composition printouts were temporarily displayed at the Rijksmuseum from June 2021 to October 2021. They were placed in front of Rembrandt's painting and not flush with it so that no one could mistake them for the original.

Both projects, according to Rinehart, are excellent case studies in how artificial intelligence might be used effectively in the art industry. Instead of being afraid of what the future holds for this technology, he wants increased participation from everyone—curators, conservators, museums, and the general people. "What's crucial is to invite the public to follow museums down that continuum," he argues, "so that we may learn to discern the shades of nuance and utility between 'real' and 'simulacrum.'" 

Is the aura of the art or artist diminished when technology provides credible answers to age-old mysteries? If you ask the staff at Google Arts and Culture, the answer is a simple and pragmatic "no." If anything, they believe their work draws attention to the Faculty Paintings and adds to the enigma surrounding Klimt, a revolutionary painter best renowned for his less rebellious Golden Period. Erdmann's AI reconstruction allows viewers to see Rembrandt's night watchman's original and dynamic vision. This ability to see what has been lost is unquestionably a net profit. 

Perhaps everything comes down to the aura. Many holes in art history can be filled using AI, but it cannot duplicate masterpieces. Nothing is possible for me. "There is no binary option between 'genuine original' and 'false synthetic' in Aura," argues Reinhart. Standing in front of a painting or gazing at it on a computer screen can both be enjoyable, but they are two quite different experiences on multiple levels. It's how we feel when we see them that matters.