Symmetry-Based Learning
54Learning representations is arguably the central problem in machine learning, and symmetry group theory is a natural foundation for it. A symmetry of a classifier is a representation change that doesn't change the examples' classes. The goal of representation learning is to get rid of unimportant variations, making important ones easy to detect, and unimportant variations are symmetries of the target function. Exploiting symmetries reduces sample complexity, leads to new generalizations of classic learning algorithms, provides a new approach to deep learning, and is applicable with all types of machine learning. In this talk I will present three approaches to symmetry-based learning: (1) Exchangeable variable models are distributions that are invariant under permutations of subsets of the variables. They subsume existing tractable independence-based models and difficult cases like parity functions, and outperform SVMs and state-of-the-art probabilistic classifiers. (2) Deep symmetry networks generalize convolutional neural networks by tying parameters and pooling over an arbitrary symmetry group, not just the translation group. In preliminary experiments, they outperformed convnets on a digit recognition task. (3) Symmetry-based semantic parsing defines a symmetry of a sentence as a syntactic transformation that preserves its meaning. The meaning of a sentence is thus its orbit under the semantic symmetry group of the language. This allows us to map sentences to their meanings without pre-defining a formal meaning representation or requiring labeled data in the form of sentence-formal meaning pairs, and achieved promising results in a paraphrase detection problem. (Joint work with Rob Gens, Chloe Kiddon and Mathias Niepert.)