Mixed item response theory models for adjusting response styles in cross-cultural datasets

Bruce William Austin

This study examined the use of mixed item response theory (IRT) models to identify and adjust data for multiple response styles within and across cultures and for large and small sample sizes. Specifically, the results support that IRT models provide a level of accuracy and ease-of use that will benefit researchers attempting to study large-scale cross-cultural data. Response style effects are a source of error that result from how individuals interpret response scales on a survey. They are likely independent of the underlying content being measured. Such styles can result from culture, carelessness, fatigue, and intentionally misleading responses. The comparison of groups with different response styles can lead to erroneous results and faulty conclusions. Adjustments to survey data for response styles can remove sources of error due to non-content differences. Accurate adjustment methods can be complex and difficult to use while easy methods can be inaccurate and biased. IRT models can balance ease-of-use and accuracy. The mixed IRT Partial Credit Model (PCM) was used to identify and adjust data for response styles in the 2012 PISA dataset consisting of 8,367 respondents. Items comprising scales measuring eight student content domains were examined for response styles using statistical and graphical information not used in previous studies. Multilevel regression analysis was implemented to compare outcomes from models using adjusted data with models using unadjusted data. PISA math achievement was the outcome variable. Results showed that removing non-contingent response (NCR) cases influenced the fit and accuracy of subsequent models as well as the estimation of response styles. Mixed PCM IRT models identified multiple latent response style classes for each scale examined. Adjusting the data for response styles improved multilevel model estimates compared to estimates using unadjusted data. Results were inconsistent, however, when attempting to duplicate findings comparing the mixed Rating Scale Model (RSM) with the mixed PCM model using a small sample of 150 respondents. The ability to compare results across national and cultural boundaries is helpful for maximizing the benefits of using large-scale international datasets. Ease-of-use and accuracy are important characteristics of response style adjustment methods that will encourage their use by researchers.

Mixed item response theory models for adjusting response styles in cross-cultural datasets

Files and links (1)

Abstract

Metrics

Details