Learning simpler language models with the differential state framework

Alexander G. Ororbia, Tomas Mikolov, David Reitter

Research output: Contribution to journalLetterpeer-review

42 Scopus citations


Learning useful information across long time lags is a critical and difficult problem for temporal neural models in tasks such as language modeling. Existing architectures that address the issue are often complex and costly to train. The differential state framework (DSF) is a simple and high-performing design that unifies previously introduced gated neural models. DSF models maintain longer-term memory by learning to interpolate between a fast-changing data-driven representation and a slowly changing, implicitly stable state.Within theDSF framework, a new architecture is presented, the delta-RNN. This model requires hardly any more parameters than a classical, simple recurrent network. In language modeling at the word and character levels, the delta-RNN outperforms popular complex architectures, such as the long short-term memory (LSTM) and the gated recurrent unit (GRU), and, when regularized, performs comparably to several state-of-the-art baselines. At the subword level, the delta-RNN's performance is comparable to that of complex gated architectures.

Original languageEnglish (US)
Pages (from-to)3327-3352
Number of pages26
JournalNeural computation
Issue number12
StatePublished - Dec 1 2017

All Science Journal Classification (ASJC) codes

  • Arts and Humanities (miscellaneous)
  • Cognitive Neuroscience


Dive into the research topics of 'Learning simpler language models with the differential state framework'. Together they form a unique fingerprint.

Cite this