Learning Commonsense Understanding through Language and Vision
22As humans, we parse language and visual scenes -- often together -- into a rich understanding of what is going on in the world. Given even a still image or a sentence describing an event, like "my friends eating at a restaurant", we can infer who's doing what, where they're at, and what might happen next. Though existing models seem strong at tasks involving language or vision, these models often struggle at combining these two modalities into such a unified commonsense understanding.
In this talk, I will cover two recent works that seek to help bridge this gap. First, I'll introduce a model named PIGLeT that learns physical commonsense understanding by interacting with the world in simulation, and uses this knowledge to ground language. PIGLeT learns linguistic form and meaning, together, and outperforms text-to-text only models that are orders of magnitude larger. I'll then introduce a model named MERLOT, which learns multimodal script knowledge by watching millions of YouTube videos with transcribed speech. MERLOT learns the layered inferences that go beyond recognition at the level of individual scenes, and towards cognition-level reasoning that understands what is happening globally over time; it gets SOTA performance on 12 vision-and-language datasets.
I'll conclude with a sketch of future directions for how we can better learn multimodal commonsense understanding, as well as on the social impacts of this work.
Speaker: Rowan Zellers, University of Washington
MSR Deep Learning team: https://www.microsoft.com/en-us/research/group/deep-learning-group/