LMS learning algorithms: Misconceptions and new results on convergence

Zi Qin Wang, Michael T. Manry, Jeffrey L. Schiano

Research output: Contribution to journalArticlepeer-review

59 Scopus citations


The Widrow-Hoff delta rule is one of the most popular rules used in training neural networks. It was originally proposed for the ADALINE, but has been successfully applied to a few nonlinear neural networks as well. Despite its popularity, there exist a few misconceptions on its convergence properties. In this paper we consider repetitive learning (i.e., a fixed set of samples are used for training) and provide an in-depth analysis in the least mean square (LMS) framework. Our main result is that contrary to common belief, the nonbatch Widrow-Hoff rule does not converge in general. It converges only to a limit cycle.

Original languageEnglish (US)
Pages (from-to)47-56
Number of pages10
JournalIEEE Transactions on Neural Networks
Issue number1
StatePublished - Jan 2000

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence


Dive into the research topics of 'LMS learning algorithms: Misconceptions and new results on convergence'. Together they form a unique fingerprint.

Cite this