Recent Advances in Image Captioning, Image-Text Retrieval andโฆ
90๐ง๐ถ๐๐น๐ฒ: Recent Advances in Image Captioning, Image-Text Retrieval and Visual Question Answering using Scene Graph Parsing, What Next?
๐ฆ๐ฝ๐ฒ๐ฎ๐ธ๐ฒ๐ฟ: Hamid Palangi
๐๐ฎ๐๐ฒ: July 9, 2019
Creating appropriate representation of data is the key for many recent breakthroughs in both language and vision. In natural language, from the structured representations like parse trees to BERT and Transformers pretrained on large scale data. In computer vision, with a slightly different trend, from scale-invariant feature transform (SIFT) to CNNs pretrained on large scale data back to more structured representation of images using scene graphs. Building appropriate models for parsing scene into graphs has unique challenges which has led to the task of Scene Graph Generation (SGG) with various subtasks from object detection, to scene graph classification and detection. An orthogonal challenge to SGG task is the effectiveness of generated scene graphs in downstream language and vision tasks that can benefit from these pretrained models. In this talk, we present our recent work to pretrain large scale SGGs, and two new models to exploit them which has resulted in significant improvement for downstream tasks of image captioning and image-text retrieval. We further present the challenges and opportunities ahead for SGGs and new downstream tasks like visual question answering.
๐ฆ๐น๐ถ๐ฑ๐ฒ๐: https://www.microsoft.com/en-us/research/uploads/prod/2021/07/43842.pdf