Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics

Andrew Cwiek, Sarah M. Rajtmajer, Bradley Wyble, Vasant Honavar, Emily Grossner, Frank G. Hillary

Research output: Contribution to journalArticlepeer-review

8 Citations (SciVal)


In this critical review, we examine the application of predictive models, for example, classifiers, trained using machine learning (ML) to assist in interpretation of functional neuroimaging data. Our primary goal is to summarize how ML is being applied and critically assess common practices. Our review covers 250 studies published using ML and resting-state functional MRI (fMRI) to infer various dimensions of the human functional connectome. Results for holdout (“lockbox”) performance was, on average, ~13% less accurate than performance measured through cross-validation alone, highlighting the importance of lockbox data, which was included in only 16% of the studies. There was also a concerning lack of transparency across the key steps in training and evaluating predictive models. The summary of this literature underscores the importance of the use of a lockbox and highlights several methodological pitfalls that can be addressed by the imaging community. We argue that, ideally, studies are motivated both by the reproducibility and generalizability of findings as well as the potential clinical significance of the insights. We offer recommendations for principled integration of machine learning into the clinical neurosciences with the goal of advancing imaging biomarkers of brain disorders, understanding causative determinants for health risks, and parsing heterogeneous patient outcomes.

Original languageEnglish (US)
Pages (from-to)29-48
Number of pages20
JournalNetwork Neuroscience
Issue number1
StatePublished - Feb 1 2022

All Science Journal Classification (ASJC) codes

  • General Neuroscience
  • Computer Science Applications
  • Artificial Intelligence
  • Applied Mathematics


Dive into the research topics of 'Feeding the machine: Challenges to reproducible predictive modeling in resting-state connectomics'. Together they form a unique fingerprint.

Cite this