Should Neural Network Architecture Reflect Linguistic Structure?

ABSTRACT:

I explore the hypothesis that recurrent neural networks are incorrectly biased for making linguistically sensible generalizations when modeling language, and that a better model class is based on architectures that reflect linguistic structure. I focus on two problems: learning to represent words and learning to generate sentences. By computing representations of words as compositions of sub-word units (characters, morphemes), we find better generalization with fewer parameters, as well as more elegant handling of "out of vocabulary" words, in comparison to traditional word embedding models. In the second part of the talk, I introduce recurrent neural network grammars (RNNGs), a new joint, generative model of phrase-structure trees and sentences. RNNGs operate via a recursive syntactic process reminiscent of probabilistic context-free grammar generation, but decisions are parameterized using RNNs that condition on the entire (top-down, left-to-right) syntactic derivation history, thus relaxing context-free independence assumptions, while retaining a bias toward explaining decisions via "syntactically local" conditioning contexts. Experiments show that RNNGs obtain better results in generating language than state of the art RNN models that ignore linguistic structure. Rather surprisingly, generative RNNGs significantly outperform discriminatively trained RNNGs for parsing, obtaining the best-reported parsing accuracies in both Chinese and English.

This is joint work with Adhi Kuncoro, Austin Matthews, Wang Ling, Lingpeng Kong, Miguel Ballesteros, Isabel Trancoso, Alan W Black, and Noah A. Smith.

BIO:

Chris Dyer is a research scientist at Google DeepMind and an assistant professor in the School of Computer Science at Carnegie Mellon University. His work has occasionally been nominated for best paper awards in prestigious NLP venues and has, much more occasionally, won them. He lives in London and, in his spare time, plays cello.