Socio-economic status predicts response patterns in surveys of learner experiences and beliefs

Large-scale international education assessments such as the Trends in Mathematics and Science Study (TIMSS) and the Progress in Reading Literacy Study (PIRLS) contain rich data about learners’ experiences of school and beliefs about themselves. These data are often used to compare the experiences and beliefs of learners who attend different types of schools, and inferences are made about the relationship between learner experiences and beliefs and achievement. However, such inter-group comparisons may be misleading. Of chief concern is that learners’ schooling experiences may shape how they interpret and respond to questionnaire items about school life and self-beliefs – a phenomenon often described as “reference bias” (Lira et al., 2022). This implies that variation in reported experiences and beliefs may partly reflect differences in interpretation rather than true differences in the underlying constructs. This brief shares findings about the accuracy and reliability with which learner experiences and beliefs are measured in TIMSS and PIRLS data for South Africa.1 We present evidence that learners attending different types of schools (proxied for by socio-economic status) interpret the TIMSS and PIRLS survey items that ask about their experiences and beliefs in systematically different ways. Based on this evidence, we caution against making inter-group comparisons of learner-reported experiences and beliefs in highly unequal societies such as South Africa.