The empty frames in the Isabella Stewart Gardner Museum are reminiscent of the largest unsolved art theft in the world. While the original masterpieces may never be found again, a team from MIT‘s Computer Science and Artificial Intelligence Laboratory (CSAIL) led by Liang Shi has now developed a system for designing reproductions of paintings that may help in this matter.
The Motivation
Mike Foshey, repaint developer:
“The value of the visual arts has increased rapidly in recent years, with the result that there is an increasing tendency to confine them in warehouses outside the public sphere. We are developing a technology to reverse this trend and create affordable and accurate reproductions that are accessible to all.”
The Reproduction Tool
The CSAIL team’s tool is called RePaint and uses a combination of 3D printing and deep learning to authentically recreate favorite images regardless of lighting conditions or placement. RePaint can be used to redesign works of art for a home, protect originals from wear and tear in museums, or even help businesses create prints and postcards of historical items.
Successful Test Runs
To test RePaint, the team reproduced a series of oil paintings created by a fellow artist. The team found that RePaint was more than four times more accurate than the most advanced physical models to produce the exact hues for different artworks.
2D Printing vs. 3D Printing
Although 2D printers are most commonly used for the reproduction of paintings, they only have a fixed set of four inks (cyan, magenta, yellow and black). The researchers found a better way to capture a more complete spectrum of Degas and Dali. They used a special technique called “Color-Contoning” where a 3D printer is used and 10 different transparent inks are stacked on top of each other in very thin layers, similar to wafers and chocolate in a kit cat bar. They combined their method with a decades-old technique called halftoning, in which an image is produced by many small colored dots rather than by continuous tones. This combination had better grasped the nuances of the colors.
Which Colors for which Image
However, with the larger palette of colors it was still not clear which colors should be used for which paintings. Instead of using more complex physical approaches, the team trained a deep learning model to predict optimal color palettes. When this worked, the system was then lined with pictures of paintings to determine which colors should be used in which areas for certain paintings.
Changil Kim, Repaint developer:
“If you only reproduce the color of a painting as it looks in the gallery, it might look different in your house. Our system works in all lighting conditions that have a much higher color rendering capacity than almost any other work to date.”
Current Status and Outlook
Despite the progress made so far, the team must continue to make improvements before they can create a brilliant duplicate of Vincent van Gogh’s “Starry Night”. Mike Foshey, for example, said that they could not fully reproduce certain colors such as cobalt blue due to a limited ink library. In the future they plan to expand this library and develop a painting specific algorithm to select the colors. The surface structure and reflection must also still be worked on in order to achieve effects such as glossy or matt surfaces.
Currently, the reproductions are only about the size of a business card because the printing is time consuming. In the future, the team hopes that more advanced commercial 3D printers will help make larger paintings more efficient.
Source: MIT
