Empirical comparison of flat-spot elimination techniques in back-propagation networks

Back-Propagation (BP)[Rumelhart et al, 1986] is a popular algorithm employed for training multilayer connectionist learning systems with nonlinear activation function (sigmoid). However, BP is plagued by excruciatingly slow convergence for many applications, and this drawback has been partly attributed to the Flat-Spots problem. Literature defines flat-spots as regions where the derivative of the sigmoid activation function approaches zero, and in these regions the weight changes become negligible, despite the presence of considerable classification error. Thus learning slows down dramatically. Several researchers have addressed this problem posed by flat-spots [Fahlman, 1988, Balakrishnan & Honavar, 1992]. In this paper we present a new way of dealing with the flat-spots in the output layer. This new method uses a Perceptron-like weight-modification strategy to complement BP in the output layer. We also report an empirical evaluation of the comparative performances of these techniques on some data-sets that have been used extensively in bench-marking inductive learning algorithms.