Improving neural network robustness through neighborhood preserving layers

Bingyuan Liu, Christopher Malon, Lingzhou Xue, Erik Kruus

Research output: Contribution to journalArticlepeer-review

4 Scopus citations


High-dimensional embeddings are often projected via fully connected layers while training neural networks. A major vulnerability that makes neural networks fail to be robust against adversarial attack is their use of overparameterized fully connected layers. We present a dimension reducing layer which preserves high-dimensional neighborhoods across the entire manifold. Atypically, our neighborhood preserving layer operates on non-static high dimensional inputs and can be trained efficiently via gradient descent. Our interest is in developing a trainable manifold representation, whose low-dimensional embeddings can be re-used for other purposes, and in investigating its robustness against adversarial attack. Our layer internally uses nearest-neighbor attractive and repulsive forces to create a low dimensional output representation. We demonstrate a novel neural network architecture which can incorporate such a layer, and also can be trained efficiently. Our theoretical results show why linear layers, which have many parameters, are innately less robust. This is corroborated by experiments on MNIST and CIFAR10 replacing the first fully-connected layer with a neighborhood preserving layer by our proposed model.

Original languageEnglish (US)
Article number104469
JournalImage and Vision Computing
StatePublished - Jul 2022

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'Improving neural network robustness through neighborhood preserving layers'. Together they form a unique fingerprint.

Cite this