Selected preprints or publications

Survey Item Validation

Abstract In the social sciences, validity refers to the adequacy of a survey (or other mode of assessment) for its intended purpose. Validation refers to the activities undertaken during and after the construction of the survey to evaluate and improve validity. Item validation refers here to procedures for evaluating and improving respondents’ understanding of the questions and response options included in a survey. Verbal probing techniques such as cognitive interviews can be used to understand respondents’ response process, that is, what they are thinking as they answer the survey items. Although cognitive interviews can provide evidence for the validity of survey items, they are time-consuming and thus rarely used in practice. The Response Process Evaluation (RPE) method is a newly-developed technique that utilizes open-ended meta-surveys to rapidly collect evidence of validity across a population of interest, make quick revisions to items, and immediately test these revisions across new samples of respondents. Like cognitive interviews, the RPE method focuses on how participants interpret the item and select a response. The chapter demonstrates the process of validating one survey item taken from the Inventory of Non-Ordinary Experiences (INOE).

Survey Uses May Influence Survey Responses

Abstract When validating psychological surveys, researchers tend to concentrate on analyzing item responses instead of the processes that generate them. Thus, the threat of invalid response on validity is neglected. Such invalid responses occur when participants unintentionally or intentionally select response options that are otherwise inaccurate. In this paper, we explore the effect of survey use on survey responses under the hypothesis that participants may intentionally give invalid responses if they disagree with the uses of the survey results. Results show that nearly all participants reflect on the intended uses of an assessment when responding to items and most decline to respond or modify their responses if they are not comfortable with the way the results will be used. We introduce methods to prevent and detect invalid responses, thus providing researchers with more confidence in the validity of their inferences.

Dynamic Fit Index Cutoffs for Confirmatory Factor Analysis Models

Abstract Model fit assessment is a central component of evaluating confirmatory factor analysis models. Fit indices like RMSEA, SRMR, and CFI remain popular and researchers often judge fit based on suggestions from Hu and Bentler (1999), who derived cutoffs that distinguish between fit index distributions of true and misspecified models. However, methodological studies note that the location and variability of fit index distributions – and, consequently, cutoffs distinguishing between true and misspecified fit index distributions – are not fixed but vary as a complex interaction of model characteristics like sample size, factor reliability, number of items, and number of factors. Many studies over the last 15 years have cautioned against fixed cutoffs and the faulty conclusions they can trigger. However, practical alternatives are absent, so fixed cutoffs have remained the status quo despite their shortcomings. Criticism of fixed cutoffs stem primarily from the fact that they were derived from one specific confirmatory factor analysis model and lack generalizability. To address this, we propose dynamic cutoffs such that derivation of cutoffs is adaptively tailored to the specific model and data being evaluated. This creates customized cutoffs that are designed to distinguish between true and misspecified fit index distributions in the researcher’s particular context. Importantly, we show that the method does not require knowledge of the “true” model to accomplish this. As with fixed cutoffs, the procedure requires Monte Carlo simulation, so we provide an open-source, web-based Shiny application that automates the entire process to make the method as accessible as possible.

Shiny App

Thinking Twice About Sum Scores

Abstract A common way to form scores from multiple-item scales is to sum responses of all items. Though sum scoring is often contrasted with factor analysis as a competing method, we review how factor analysis and sum scoring both fall under the larger umbrella of latent variable models, with sum scoring being a constrained version of a factor analysis. Despite similarities, reporting of psychometric properties for sum scored or factor analyzed scales are quite different. Further, if researchers use factor analysis to validate a scale but subsequently sum score the scale, this employs a model that differs from validation model. By framing sum scoring within a latent variable framework, our goal is to raise awareness that (a) sum scoring requires rather strict constraints, (b) imposing these constraints requires the same type of justification as any other latent variable model, and © sum scoring corresponds to a statistical model and is not a model-free arithmetic calculation. We discuss how unjustified sum scoring can have adverse effects on validity, reliability, and qualitative classification from sum score cut-offs. We also discuss considerations for how to use scale scores in subsequent analyses and how these choices can alter conclusions. The general goal is to encourage researchers to more critically evaluate how they obtain, justify, and use multiple-item scale scores.