Duration: 1:03:34

3,431 views

73

Image captioning is an interesting problem in the intersection between computer vision and natural language processing, and it has attracted great attention from their respective research communities. Recent image captioning models have achieved impressive results on the tasks where large amounts of paired image-caption training data is available. However, they generalize poorly to images in the wild, where there are a wide variety of visual objects that are unseen in the caption corpora for training. This raises the challenge of Novel Object Captioning (NOC), that is, generating captions to describe novel objects unseen in paired image-caption training data, which is especially pertinent in real-world applications.

This webinar will focus on some of the recent vision-language pretraining (VLP) approaches for image captioning. We will cover our latest approaches, including object-semantics aligned pretraining (OSCAR) and visual-vocabulary pretraining (VIVO). We will also discuss their key principles and how we address the core challenges in image caption generation. Join us to learn how our discovery leads to a new image captioning framework that achieves state-of-the-art performance on the nocaps benchmark (developed to evaluate NOC at scale) and surpasses human CIDEr scores on nocaps for the first time.

Visual-vocabulary pretraining (VIVO) conducts pretraining with vision data only. As the method does not need paired image-caption data, it opens the possibility of leveraging large amounts of images, paired with either human-labeled or machine-generated tags. By using VIVO pretraining, the performance of the captioning model, especially on novel objects, has been substantially improved.

What you’ll learn:

■ How latest VLP approaches help to improve captioning performance by pretraining on large-scale image-text pairs, then fine-tuning on task-specific small data.
■ How VIVO pretraining is conducted in the absence of image-text pairs, leading to state-of-the-art performance on NOC.
■ How visual-text alignment is learned during VLP and significantly contributes to the downstream vision-language tasks.
■ How to use our open-source model and code in your research and how to use our Azure Cognitive Services cloud API for your own development.

𝐑𝐞𝐬𝐨𝐮𝐫𝐜𝐞 𝐥𝐢𝐬𝐭:

■ Azure Florence Project page: https://www.microsoft.com/en-us/research/project/azure-florence-vision-and-language
■ Oscar on Github: https://github.com/microsoft/Oscar
■ Oscar Publication: https://www.microsoft.com/en-us/research/publication/oscar-object-semantics-aligned-pre-training-for-vision-language-tasks
■ VIVO Publication: https://www.microsoft.com/en-us/research/publication/vivo-surpassing-human-performance-in-novel-object-captioning-with-visual-vocabulary-pre-training
■ Novel object captioning surpasses human performance on benchmarks (MSR Blog): https://www.microsoft.com/en-us/research/blog/novel-object-captioning-surpasses-human-performance-on-benchmarks
■ Objects are the secret key to revealing the world between vision and language (MSR Blog): https://www.microsoft.com/en-us/research/blog/objects-are-the-secret-key-to-revealing-the-world-between-vision-and-language
■ Azure AI, describes images as well as people do (AI Blog): https://blogs.microsoft.com/ai/azure-image-captioning
■ Lijuan Wang (Researcher Profile): https://www.microsoft.com/en-us/research/people/lijuanw
■ Xiaowei Hu (Researcher Profile): https://www.microsoft.com/en-us/research/people/xiaowh

*This on-demand webinar features a previously recorded Q&A session and open captioning.

Explore more Microsoft Research webinars: https://aka.ms/msrwebinars