Multilevel Versus Single-Level Factor Analysis: Differentiating Within-Person and Between-Person Variability Using the CCAPS-34

Objective: Although most self-report measures of distress are intended to assess time-varying constructs, they are usually developed using between-person data. They are therefore vulnerable to misspecification due to measurement nonequivalence at the between-person and within-person levels. In recent years, multiple studies have found that self-report distress may not be the same when considered over time versus between people: what changes over time may not be the same as what makes individuals different from one another. Method: In this study, we present a multilevel factor analysis (MFA) of a widely used multidimensional self-report measure of psychological symptoms, the Counseling Center Assessment of Psychological Symptoms-34 (CCAPS-34), in two samples (Ns: 1,223 and 757) of individuals with 10 or more observations each. We compare the results to traditional factor analysis. Results: Single-level factor analyses converged with the established seven-factor structure, regardless of sample or data subset. The MFA largely, but not entirely, recovered the existing factor structure of the CCAPS-34 at the withinperson level in both samples, but not at the between-person level. The between-person factor structure was simpler than the within-person factor structure, particularly in the nonclinical sample in which only two factors were sufficient. Conclusions: The factors of this instrument that change over time appear to be narrow, while differences between people are broader. This argues against using general distress measures when assessing treatment outcomes. MFA is a promising method for measure development, even in data with relatively few observations per person. This method may clarify how self-report psychopathology manifests over time.