Duration: 1:28:01

75 views

1

Towards understandable neural networks for high level AI tasks - Part III Overview of talk series: Current AI software relies increasingly on neural networks (NNs). The universal data structure of NNs is the numerical vector of activity levels of model neurons, typically with activity distributed widely over many neurons. Can NNs in principle achieve human-like performance in higher cognitive domains - such as inference, planning, grammar - where theories in AI, cognitive science, and linguistics have long argued that abstract, structured symbolic representations are necessary? The work I will present seeks to determine whether, and precisely how, distributed vectors can be functionally isomorphic to symbol structures for computational purposes relevant to AI - at least in certain idealized limits such as unbounded network size. This work - defining and exploring Gradient Symbolic Computation (GSC) - has produced a number of purely theoretical results. Current work at MSR is exploring the use of GSC to address large-scale practical problems using NNs that can be understood because they operate under the explanatory principles of GSC. Part I is available at http://resnet/resnet/fullvideo.aspx?id=36339. Part II (Oct. 12), should cover Review from Part I how recursive structures built of symbols can be compositionally encoded as distributed numerical vectors: tensor product representations (TPRs) - review and extend the discussion from Part I of how TPRs can be used to compute recursive symbolic functions with massive parallelism - argue for the crucial role of distributed representations in neural networks, and show how distributed TPRs (unlike localized representations) generalize based on similarity of different structural positions/roles - present a generative model that can in principle be used to reverse-engineer a trained network to determine whether that network has learned a TPR scheme - illustrate, through the example of wh-questions, how networks deploying TPRs go beyond the capabilities of symbol processing because their representations include TPRs not only of purely discrete structures, but also structures in which discrete positions are occupied by blends of numerically-weighted ('gradient') symbols - or equivalently, structures built of discrete symbols that occupy numerically-weighted positions (or roles). Part III, on Friday, October 16th, will include, among other topics, discussion of - how certain symbolic constraint-based grammars can be encoded as interconnection-weight matrices which asymptotically compute the TPRs of grammatical structures - how certain symbolic Maxent models can be encoded as weight matrices of networks that produce the TPRs of alternative structures with a log-linear probability distribution