There exists a neural network that does not make avoidable mistakes

The authors show that a multiple-input, single-output, single-hidden-layer feedforward network with (known) hardwired connections from input to hidden layer, monotone squashing at the hidden layer and no squashing at the output embeds as a special case a so-called Fourier network, which yields a Fourier series approximation properties of Fourier series representations. In particular, approximation to any desired accuracy of any square integrable function can be achieved by such a network, using sufficiently many hidden units. In this sense, such networks do not make avoidable mistakes.