Discovering hidden connections in art with deep, interpretable visual analogies
21Image retrieval systems allow individuals to find images that are semantically similar to a query image. This serves as the backbone of reverse image search engines and many product recommendation engines. Restricting an image retrieval system to particular subsets of images can yield new insights into relationships in the visual world.
In this webinar, Microsoft Research Development Engineer Mark Hamilton presents a novel method for specializing image retrieval systems called conditional image retrieval. When applied over large art datasets in particular, conditional image retrieval provides visual analogies that bring to light hidden connections among different artists, cultures, and media. This aims to encourage a new level of engagement with creative artifacts and inspire people to imagine new works of art. Hamilton will demonstrate how conditional image retrieval systems can efficiently find shared semantics between works of vastly different media and cultural origin. He’ll also demonstrate how this approach can improve generative adversarial networks (GANs), algorithms that create novel images from scratch, by identifying where GANs fail to model the true data distribution. Finally, Hamilton will introduce a generalization of Additive Shapley Values (SHAP) for better understanding why neural networks view images as similar.
Together, you’ll learn how to:
■ Build a reverse image search engine
■ Generalize these engines to handle extra requirements on the retrieved images, such as a filter for a particular artistic culture or time period
■ Find shared inspiration and semantics in the visual arts
■ Interpret and explain a reverse image search engine
■ Use conditional image retrieval to discover blind spots in GANs
𝗥𝗲𝘀𝗼𝘂𝗿𝗰𝗲 𝗹𝗶𝘀𝘁:
■ Conditional Image Retrieval (Publication): https://www.microsoft.com/en-us/research/publication/conditional-image-retrieval/
■ MIT blog: https://www.csail.mit.edu/news/algorithm-finds-hidden-connections-between-paintings-met
■ Interactive Website: www.aka.ms/mosaic
■ GitHub Code: www.aka.ms/visual-analogy
■ Mark Hamilton at Microsoft Research: https://www.microsoft.com/en-us/research/people/marhamil/
■ Cognitive Services Research (Group Page): https://www.microsoft.com/en-us/research/group/cognitive-services-research/
*This on-demand webinar features a previously recorded Q&A session and open captioning
This webinar originally aired on July 30, 2020
Explore more Microsoft Research webinars: https://aka.ms/msrwebinars