Learning to Understand Natural Language in Physically-Grounded Environments
6We all want computers to understand natural language. Whether it be to command a robot, or answer a question by reading the web, language understanding is a fundamental problem for natural language processing. Physically-grounded settings are an important special case of this problem, with applications in robotics and interactions with embodied systems. This talk presents Logical Semantics with Perception (LSP), a model for understanding natural language statements within a physically-grounded environment. For example, given an image, LSP can map a description such as "the blue mug to the left of the monitor," to a set of image segments containing blue mugs left of monitors. Importantly, LSP can be trained directly from natural language / object pairs, which is a natural form of supervision that can be easily obtained from human interaction. I will present experiments applying LSP to several domains, including image understanding (using Microsoft Kinect data) and geographical question answering. The talk may also include some additional related work on semantic parsing at web-scale for information extraction and question answering. (time permitting)