LMS learning algorithms: Misconceptions and new results on convergence

Zi Qin Wang; Michael T. Manry; Jeffrey L. Schiano

doi:10.1109/72.822509

LMS learning algorithms: Misconceptions and new results on convergence

Zi Qin Wang
, Michael T. Manry
, Jeffrey L. Schiano

Electrical Engineering

Research output: Contribution to journal › Article › peer-review

65 Scopus citations

Abstract

The Widrow-Hoff delta rule is one of the most popular rules used in training neural networks. It was originally proposed for the ADALINE, but has been successfully applied to a few nonlinear neural networks as well. Despite its popularity, there exist a few misconceptions on its convergence properties. In this paper we consider repetitive learning (i.e., a fixed set of samples are used for training) and provide an in-depth analysis in the least mean square (LMS) framework. Our main result is that contrary to common belief, the nonbatch Widrow-Hoff rule does not converge in general. It converges only to a limit cycle.

Original language	English (US)
Pages (from-to)	47-56
Number of pages	10
Journal	IEEE Transactions on Neural Networks
Volume	11
Issue number	1
DOIs	https://doi.org/10.1109/72.822509
State	Published - Jan 2000

All Science Journal Classification (ASJC) codes

Software
Computer Science Applications
Computer Networks and Communications
Artificial Intelligence

Access to Document

10.1109/72.822509

Cite this

@article{78960f8c41ea46febeb98c75fa3216dd,

title = "LMS learning algorithms: Misconceptions and new results on convergence",

abstract = "The Widrow-Hoff delta rule is one of the most popular rules used in training neural networks. It was originally proposed for the ADALINE, but has been successfully applied to a few nonlinear neural networks as well. Despite its popularity, there exist a few misconceptions on its convergence properties. In this paper we consider repetitive learning (i.e., a fixed set of samples are used for training) and provide an in-depth analysis in the least mean square (LMS) framework. Our main result is that contrary to common belief, the nonbatch Widrow-Hoff rule does not converge in general. It converges only to a limit cycle.",

author = "Wang, \{Zi Qin\} and Manry, \{Michael T.\} and Schiano, \{Jeffrey L.\}",

note = "Funding Information: Manuscript received April 27, 1997; revised January 26, 1999 and August 31, 1999. This work was supported by the Advanced Technology Program of the state of Texas under Grant 003 656-063. Z.-Q. Wang is with FAS Technologies, Dallas, TX 75238 USA (e-mail: [email protected]). M. T. Manry is with the Department of Electrical Engineering, The University of Texas at Arlington, Arlington, TX 76019 USA (e-mail: [email protected]). J. L. Schiano is with the Department of Electrical Engineering, The Pennsylvania State University, University Park, PA 16802 USA (e-mail: [email protected]). Publisher Item Identifier S 1045-9227(00)01193-0.",

year = "2000",

month = jan,

doi = "10.1109/72.822509",

language = "English (US)",

volume = "11",

pages = "47--56",

journal = "IEEE Transactions on Neural Networks",

issn = "1045-9227",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "1",

}

TY - JOUR

T1 - LMS learning algorithms

T2 - Misconceptions and new results on convergence

AU - Wang, Zi Qin

AU - Manry, Michael T.

AU - Schiano, Jeffrey L.

N1 - Funding Information: Manuscript received April 27, 1997; revised January 26, 1999 and August 31, 1999. This work was supported by the Advanced Technology Program of the state of Texas under Grant 003 656-063. Z.-Q. Wang is with FAS Technologies, Dallas, TX 75238 USA (e-mail: [email protected]). M. T. Manry is with the Department of Electrical Engineering, The University of Texas at Arlington, Arlington, TX 76019 USA (e-mail: [email protected]). J. L. Schiano is with the Department of Electrical Engineering, The Pennsylvania State University, University Park, PA 16802 USA (e-mail: [email protected]). Publisher Item Identifier S 1045-9227(00)01193-0.

PY - 2000/1

Y1 - 2000/1

N2 - The Widrow-Hoff delta rule is one of the most popular rules used in training neural networks. It was originally proposed for the ADALINE, but has been successfully applied to a few nonlinear neural networks as well. Despite its popularity, there exist a few misconceptions on its convergence properties. In this paper we consider repetitive learning (i.e., a fixed set of samples are used for training) and provide an in-depth analysis in the least mean square (LMS) framework. Our main result is that contrary to common belief, the nonbatch Widrow-Hoff rule does not converge in general. It converges only to a limit cycle.

AB - The Widrow-Hoff delta rule is one of the most popular rules used in training neural networks. It was originally proposed for the ADALINE, but has been successfully applied to a few nonlinear neural networks as well. Despite its popularity, there exist a few misconceptions on its convergence properties. In this paper we consider repetitive learning (i.e., a fixed set of samples are used for training) and provide an in-depth analysis in the least mean square (LMS) framework. Our main result is that contrary to common belief, the nonbatch Widrow-Hoff rule does not converge in general. It converges only to a limit cycle.

UR - https://www.scopus.com/pages/publications/0033640656

UR - https://www.scopus.com/pages/publications/0033640656#tab=citedBy

U2 - 10.1109/72.822509

DO - 10.1109/72.822509

M3 - Article

C2 - 18249738

AN - SCOPUS:0033640656

SN - 1045-9227

VL - 11

SP - 47

EP - 56

JO - IEEE Transactions on Neural Networks

JF - IEEE Transactions on Neural Networks

IS - 1

ER -

LMS learning algorithms: Misconceptions and new results on convergence

Abstract

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this