TY - JOUR
T1 - On the convergence of formally diverging neural net-based classifiers
AU - Berlyand, Leonid
AU - Jabin, Pierre Emmanuel
N1 - Publisher Copyright:
© 2018 Académie des sciences
PY - 2018/4
Y1 - 2018/4
N2 - We present an analytical study of gradient descent algorithms applied to a classification problem in machine learning based on artificial neural networks. Our approach is based on entropy–entropy dissipation estimates that yield explicit rates. Specifically, as long as the neural nets remain within a set of “good classifiers” we establish a striking feature of the algorithm: it mathematically diverges as the number of gradient descent iterations (“time”) goes to infinity but this divergence is only logarithmic, while the loss function vanishes polynomially. As a consequence, this algorithm still yields a classifier that exhibits good numerical performance and may even appear to converge.
AB - We present an analytical study of gradient descent algorithms applied to a classification problem in machine learning based on artificial neural networks. Our approach is based on entropy–entropy dissipation estimates that yield explicit rates. Specifically, as long as the neural nets remain within a set of “good classifiers” we establish a striking feature of the algorithm: it mathematically diverges as the number of gradient descent iterations (“time”) goes to infinity but this divergence is only logarithmic, while the loss function vanishes polynomially. As a consequence, this algorithm still yields a classifier that exhibits good numerical performance and may even appear to converge.
UR - http://www.scopus.com/inward/record.url?scp=85043773021&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85043773021&partnerID=8YFLogxK
U2 - 10.1016/j.crma.2018.03.003
DO - 10.1016/j.crma.2018.03.003
M3 - Article
AN - SCOPUS:85043773021
SN - 1631-073X
VL - 356
SP - 395
EP - 405
JO - Comptes Rendus Mathematique
JF - Comptes Rendus Mathematique
IS - 4
ER -