How much value do Value Added Models provide for the quality of schools?

Rachel Outhred, Education Theme Lead, Itad

The anticipation of the value that Value Added Models (VAM) might bring to developing an accountability measure for school quality has been palpable in the development sector over the past two years. They were a focus for discussion at this year’s 2017 UKFIET Education and Development conference. The technical limitations of VAM were highlighted by the University of Cambridge’s Anna Vignoles, and the use of them in a presentation on the evaluation of Promoting Equality in African Schools (PEAS), commissioned by Ark and PEAS in Uganda. So what are VAMs? And how valuable are they?

What are Value Added Models?

VAMs use student test scores to monitor and evaluate teachers and schools in order to increase the accountability of these actors in the education of children. As VAMs are thought to control for almost everything other than teacher and school effectiveness, it is claimed that they can accurately measure increases in school quality. Simply put, VAMs produce a score for pupils based on gains since their previous test. In Uganda, for example, Ark calculated student scores by comparing the results of the same sample of students in their examinations at both the end of primary and secondary school. Ark’s Phil Elks argues that with VAMs, ‘schools get credit when a student performs better than expected, given their prior attainment’.

VAMs were first hinted at by Eric Hanushek at Stanford University, where he argued that the use of micro-data to understand gains in student achievement can help understand the performance of teachers. Some have described calculating learning gains in test scores as part of a ‘Growth Mindset’, where talent and ability are the starting point to success, rather than the markers of success. In a Growth Mindset, motivation and resilience lead to accomplishment – parents, teachers and students are encouraged to focus on what children need to learn in order to grow, rather than focusing on a fixed level of attainment as the definition of success.

VAMs encourage a Growth Mindset as they report test scores by growth since the previous test, rather than innate ability or talent. It is argued that VAMs lead to equity-based assessment in school and provide more useful data about the quality of schools, rather than the ‘quality’ of the school’s student intake. For example, in Uganda, Ark proposes to use value added scores ‘to improve secondary school accountability at low cost’.

How valuable are Value Added Models?

On the face of it, VAMs sound very valuable but it’s worth reflecting on some of the lessons from other education systems on using them for accountability purposes. There are three main criticisms of using VAMs for accountability purposes:

1. At the teacher level, measures from VAMs are not stable enough to use for accountability purposes.

VAMs have been used in a variety of education systems globally. Chicago Public Schools, New York City and the District of Columbia Public Schools use VAMs to assign teacher bonuses, identify teachers for training and to identify ‘weak’ teachers. However, there is considerable concern regarding the use of this modelling for making high-stakes decisions about individual teachers. The National Research Council of the National Academy of Sciences, leading global testing house Education Testing Services and RAND Corporation, all agree that the estimates from VAMs are simply too imprecise and not sufficiently ‘causal’ to use for high-stakes accountability purposes.

2. What the test measures tends to be ignored in the application of VAMs.

Working on the DFID commissioned Road-map for Enhancing Assessment in Uganda, the team found that much of the knowledge and skills learners need has little or no role in formal tests and examinations in Uganda. We found a ‘significant mismatch between what students actually learn, and the knowledge and skills valued and emphasised by the current assessment system, and what employers see as key skills or competencies for successful lives and careers now and in the future‘.

The report highlights that many actors within the education system, and within society, are incentivised to behave in ways that do not result in the teaching and learning of the knowledge and skills learners need. But it is the test scores from these examinations that have been used in VAM in Uganda. Therefore, the scores assigned to schools and teachers in this context have little to do with school quality and more to do with the extent to which schools and teachers are able to increase students’ likelihood of passing examinations. Certainly passing examinations is important, but school quality does and must mean more than passing examinations which are known to have little to do with the knowledge and skills learners need.

3. Negative consequences are greatest for the most vulnerable children.

High-stakes testing using assessments that measure basic rote skills will have the largest consequences for the most vulnerable children. As Resnick and Berger explain in their paper ‘An American Examination System’, in order to increase test scores, teachers that serve vulnerable children tend to take the most direct route to raising scores – which usually entails highly repetitious test-taking practice. In addition, it is well known that in contexts where secondary examinations are not compulsory, but are used for accountability purposes, schools and teachers are incentivised to discourage perceived ‘weak learners’ from sitting the examination. Such practices put poor and vulnerable children most at risk of dropout and exclusion.

Overall, VAMs in and of themselves are neither valuable nor without value. The extent to which they are useful depends on (1) what is being measured by tests and examinations used in the model and (2) what decisions the VAM score is likely to inform. Accountability for increasing learning in educational systems is important. Focusing on the continued growth of individual students is important. However, some of the concerns around the use of VAMs in education systems globally need to be weighed and considered carefully when using VAM scores for teacher and school accountability purposes in low and middle income countries.