A comparison of bayesian and frequentist model selection methods for factor analysis models

We compare the performances of well-known frequentist model fit indices (MFIs) and several Bayesian model selection criteria (MCC) as tools for cross-loading selection in factor analysis under low to moderate sample sizes, cross-loading sizes, and possible violations of distributional assumptions. The Bayesian criteria considered include the Bayes factor (BF), Bayesian Information Criterion (BIC), Deviance Information Criterion (DIC), a Bayesian leave-one-out with Pareto smoothed importance sampling (LOO-PSIS), and a Bayesian variable selection method using the spike-and-slab prior (SSP; Lu, Chow, & Loken, 2016). Simulation results indicate that of the Bayesian measures considered, the BF and the BIC showed the best balance between true positive rates and false positive rates, followed closely by the SSP. The LOO-PSIS and the DIC showed the highest true positive rates among all the measures considered, but with elevated false positive rates. In comparison, likelihood ratio tests (LRTs) are still the preferred frequentist model comparison tool, except for their higher false positive detection rates compared to the BF, BIC and SSP under violations of distributional assumptions. The root mean squared error of approximation (RMSEA) and the Tucker-Lewis index (TLI) at the conventional cut-off of approximate fit impose much more stringent "penalties" on model complexity under conditions with low cross-loading size, low sample size, and high model complexity compared with the LRTs and all other Bayesian MCC. Nevertheless, they provided a reasonable alternative to the LRTs in cases where the models cannot be readily constructed as nested within each other. Translational Abstract Quantitative research utilizes various models to draw information and conclusions from data. Appropriate model selection is thus a crucial determinant of the validity and usefulness of the research. Factor analysis is a popular statistical technique for consolidating or reducing the dimension of multivariate observed variables through their linkages to some lower-dimensional latent factors.Weconducted a Monte Carlo study to compare the performances of well-known frequentist model fit indices and several Bayesian model selection criteria as tools for model selection in factor analysis. Advantages and disadvantages of these model selection tools are noted, and guidelines for using these tools are summarized. This study offers valuable insights to methodologists on the strengths and limitations of current model selection tools, and practical suggestions to applied researchers on ways to capitalize on the strengths of particular Bayesian model comparison criteria in performing model selection in factor analysis, especially in scenarios where common frequentist model fit indices fail to provide a definitive answer. In an empirical example, we applied these model selection tools to evaluate the quality of the Motivated Strategies for Learning Questionnaire in measuring the rehearsal, elaboration, and effort regulation of 2,000 college students.