Outcome theory and education

19:15, Oct 01 2012

If you were looking for a farmer who was good at raising stock in difficult conditions you'd not just focus on their raw outcomes. Rather, you'd look at how much they improve the stock they're working with.

What's this got to do with national standards in education? Many words have been spilled on the politics of national standards - supporting or rejecting them.

It's complex and I don't want to take a position on the overall merit or otherwise of having a system of national standards.

However, as an outcomes measurement specialist, I want to shed some light on a technical, rather than political, aspect of national standards.

Outcomes theory, the general theory of measuring stuff like this, has a principle called equality of input, equality of outcome or the ''level playing field'' principle.

When you want to measure the performance of anything (teachers, schools, whatever) it's important to distinguish between two very different situations.


In the first situation you have rough equality of input. What's coming into the system is roughly the same for all units whose performance is being measured. In this case we're talking about the academic level of children entering the school system.

Where you have equality of input, it makes perfect technical sense to just rank performance on the basis of final outcomes. In the measurement business we call these raw outcomes. Raw outcomes are ones that haven't been adjusted for varying levels of input between different units whose performance is being assessed (schools, in this case). In a case of equality of input, it's reasonable to assume that the thing that has caused differences in raw outcomes is the level of skill and effort put into the process.

But contrast this with a second situation which is more like our school system. In this situation you have inequality of input.

Here, just measuring raw outcomes won't accurately measure the level of skill and effort put into the process. The picture will be confused by the different starting level of the input that teachers are working with.

In our education system, we have a situation where the kids coming into different schools differ on their level of academic performance and the level of outside resources available to them to promote their academic growth.

Education Minister Hekia Parata has noted that she ultimately wants to include national standards results in the mix of factors used to reward teachers. So she sees the current national standards as measuring teacher skill and effort in improving students' academic performance.

If one wants to use national standards for this, it seems to me that the standards, in their current form, are just measuring raw outcomes. Therefore they'll be providing a confused picture of teacher skill and effort confounded by the academic level where the students in the particular school started from.

If you were wanting to find, and reward, farmers who are good at raising stock in difficult conditions, you would not just look at raw outcomes.

That is likely to highlight those farmers who are well resourced, on ideal farms and who are starting with fresh high-performing stock.

What you'd be looking for would be improvement over baseline levels. In the private sector this is often referred to as measuring of ''value added''.

Of course, it's not trivial to measure value added in teaching.

Measuring just raw outcomes can create the wrong incentives for teachers in the system. Much of the talk about national standards is that they're intended to give a better chance to kids who are struggling because of lack of opportunity.

However, from a purely technical point of view, a measurement system that just rewards raw outcomes has a tendency to create the opposite incentives for teachers.

Those teachers who seek rewards will tend to go to the places where raw outcomes will be highest - well-off schools in which the kids are high performing to start with.

Schools with students who start at a lower academic level and have fewer outside resources to enhance their academic performance will generally end up at a lower level on raw outcomes.

Measuring those raw outcomes does not seem to provide the incentives for teachers to move into exactly those schools where we want good teachers to work their teaching magic and help redress the disadvantage of the students in such schools face.

So, separate from any political considerations, a purely technical lens would suggest that the next place for the minister to push national standards work is to focus on whether a way can be identified to reliably measure improvement.

It may, or may not, turn out to be possible to get a feasible measure of value added in our schools.

But at the moment, as a technical expert in the measurement business, I can't see how using raw outcomes is going to deliver the minister the measure she really wants - information on who are the best teachers, and which are the best schools at improving the academic performance of less advantaged kids.

- Dr Paul Duignan, of Parker Duignan Consulting is an honorary Research Fellow at Massey University, and an expert in outcomes measurement. www.outcomescentral.org

The Dominion Post