Hybrid autoencoder with orthogonal latent space for robust population structure inference

Meng Yuan, Hanne Hoskens, Seppe Goovaerts, Noah Herrick, Mark D. Shriver, Susan Walsh, Peter Claes

Research output: Contribution to journalArticlepeer-review


Analysis of population structure and genomic ancestry remains an important topic in human genetics and bioinformatics. Commonly used methods require high-quality genotype data to ensure accurate inference. However, in practice, laboratory artifacts and outliers are often present in the data. Moreover, existing methods are typically affected by the presence of related individuals in the dataset. In this work, we propose a novel hybrid method, called SAE-IBS, which combines the strengths of traditional matrix decomposition-based (e.g., principal component analysis) and more recent neural network-based (e.g., autoencoders) solutions. Namely, it yields an orthogonal latent space enhancing dimensionality selection while learning non-linear transformations. The proposed approach achieves higher accuracy than existing methods for projecting poor quality target samples (genotyping errors and missing data) onto a reference ancestry space and generates a robust ancestry space in the presence of relatedness. We introduce a new approach and an accompanying open-source program for robust ancestry inference in the presence of missing data, genotyping errors, and relatedness. The obtained ancestry space allows for non-linear projections and exhibits orthogonality with clearly separable population groups.

Original languageEnglish (US)
Article number2612
JournalScientific reports
Issue number1
StatePublished - Dec 2023

All Science Journal Classification (ASJC) codes

  • General


Dive into the research topics of 'Hybrid autoencoder with orthogonal latent space for robust population structure inference'. Together they form a unique fingerprint.

Cite this