National standards: this data is not for ranking

Comparing individual schools by comparing their national standards data is a waste of time. Students are assessed by overall teacher judgement, not standardized testing. Getting some sensible moderation in place is one of the biggest challenges in the implementation of the standards, but people need to accept that moderation will never be as rigorous as it is under standardized testing regimes. For reasons I’ll touch on later, this is by design.

A school’s national standards data is not useful information for a parent choosing which school to pack a kid off to. (Given zoning and the high fixed costs of moving, there is limited scope for school choice in the first place, so that might be a moot point.) Far better is knowledge of what a school does with its data.

If I was at the Ministry of Education (or—heaven forbid!—a parent), here’s what I’d be looking for.

I’d look for signs that schools are using the standards sensibly on a student level. Especially for students not at standard, I’d want to see individualized learning plans, with achievable benchmarks/milestones. Ideally, these plans would be designed in and as a collaboration between teacher, student, and caregiver. Give the student a sense of direction and ownership: here’s what we want you to be able to do, here’s our plan for getting you there, and here’s how you’ll be able to feel your own progress along the way.

Over time, I’d look for signs that schools are using standards, in conjunction with the learning plans, to do some value-added appraisal of teachers. I would incorporate this formally into professional development structures. I accept that there’s only so much a teacher can be reasonably expected to do for kids who turn up hungry, have caregivers with significant reading difficulties, or who switch between schools a lot. (These are, incidentally, things that are thought to correlate pretty tightly with decile.) Placing appropriate weight on factors like this is something that I think standards will be able to do over time, even if they’re fairly messy.

These things don’t really need tightly moderated standards. If comparison between schools is what you’re really wanting, then standardized tests would produce more useful data. But standardized testing was defeated politically on grounds I think were actually quite sensible.

Standardized tests are just not a good cultural fit with New Zealand. There’s a national egalitarian streak: we don’t like putting anyone higher or lower than anyone else, and we especially don’t like it when it’s kids we’re sorting. We don’t like to pressure children academically to the point of causing stress. We worry about labelling effects on kids who are at risk of falling behind: effects that are real, powerful, long-lasting, and can be very destructive. These traits of ours are admirable, and saying “little Timmy got 35 percent in his Year 2 reading test” is offensive to just about all of them.

Besides, the literature on standardized tests is a minefield. As an example, unless they’re made meaningful to students (ie have consequences) then there’s not much to suggest they help much at all with anything. But the point of the New Zealand variant on standards is that they’re snapshots of where students are at and tools for planning. They’re not meant to be consequential for students. This next point is important so I’m going to italicize it: most students probably don’t even know when their teacher is assessing them against the standards.

Where cross-school comparisons might come in useful is in identifying stand-out schools so that successful and/or innovative practices can be, where appropriate, replicated more widely. Obviously, what works in a school within one particular cultural, social, and economic context won’t necessarily work for a school that’s in a totally different one. You do your data analysis and your case studies, then you devise sensible categories and work within them.

So long as moderation is at least better than hopeless, in time, we’ll also learn quite a bit more than we already know about the impact of social and economic factors on academic attainment during the early years of school. This is important, because it’s these early years that are assumed to matter most. National standards data, for all its flaws, is or can be made rich enough to support meaningful research that will help us improve how we teach our children.

Depending on whether or not I can be bothered, I might write up some stuff on why I don’t like deciles as analytical tools. But not this week.

Advertisements

National Standards

Familiarize yourself with the standards themselves (reading and writing [pdf], mathematics [pdf]). Notice that assessment against the standards is done by overall teacher judgement and not entirely, or even principally, by standardized testing.

If you want to play with the shiny data set, Luis A Apiolaza has done you a favour by putting it into something sensible and R-friendly. I recommend reading all of the case studies on Stuff before diving in. Context matters.

My own prejudices. I am a policy analyst with nil expertise in education. I support publication of national standards data. I am agnostic as to whether national standards should have been introduced in the first place, but furious at those “public servants” who sought to obstruct implementation of a lawful Government policy: no integrity.

I’m mainly interested in what the standards data can and can’t tell us, and how they might be used to improve education outcomes. The lens I’m trying to look through is how would I tackle this if I were at the Ministry of Education?

Concern trolling

An open letter to @Megan_Woods

Dear Dr Woods

I understand you’re pretty new in town, so let me explain how things work. When someone like you from the hinterlands first comes to Wellington all starry eyed and ambitious and desirous to change the world, more experienced caucus colleagues, if they have your best interests at heart, will give you some version of the talk. This is not intended to be a pleasant conversation.

Typically, someone who’s seen a dozen brighter stars flame out will gently explain your responsibilities over the coming three years (summary: work hard for your electorate constituents, and other than that, say nothing at all to anybody under any circumstances ever) and provide a brief introduction to the concept of a single-term MP. Judging from your tweet last night…

RT @Megan_Woods Hitler has a pretty clear manifesto that he campaigned and won on. Question: does this make what he did ok @NZNationalParty? #SaveOurAssets

First-term MP for Wigram, covering herself in glory.

…you either did not listen to the talk or your colleagues didn’t care enough about you to make you sit down and shut up through it. This was a poor start to your career in Parliament.

I have no desire to correct your trampling all over human misery for petty partisan ends by taking umbrage and trampling all over human misery for petty partisan ends. I am not easily offended, and in truth your tweet did not upset me personally.

But, for the love of all that is good and holy in this world, I urge you to stop saying stupid things, or, failing that, to stop saying anything at all.

In fewer than 140 characters, you conceded your own party’s central contention on the government’s proposed programme of partial SOE floats, and then godwinned yourself out of reasonable discourse. I believe I am being objective when I use the word stupid to describe this performance.

There are reasonable objections to the government’s intended course of action, and there are objections that are not reasonable.

Early criticism about foregoing future dividend streams was misguided. Nobody has provided any evidence to suggest that the government will not recoup, on average, the market’s weighted estimate of this stream’s present value at IPO. If you believe you have a better estimate of the true value of these dividends, then I suggest you participate in the IPO and in the sharemarket in the days and years following. If you’re right, you’ll get rich. Use the money to spend on whatever you think the Government should have been spending it on instead. Seriously: you’ll be rich enough to do this.

I guess it’s not your fault your superiors later opted for the silly line that “the Government has no mandate”. Putting aside that this policy simply could not have taken the electorate by surprise – it was announced nearly a year before the general election and your party made opposition to it a central campaign plank – the Government, along with its confidence and supply partners, can command a majority in the House of Representatives. That is what a mandate to govern looks like in Westminster democracies. To put this into terms relevent to your current position: it’s why you are an Opposition backbench MP and not a Government backbench MP.

I accept that Parliamentary majorities can and do pass manifestly unjust laws, and that sometimes, as you allude, these laws can be monstrous and evil. Putting aside that this is obviously not the case in this instance (partial floats are, at worst, unwise, and nobody is suggesting the mass murder is part of the policy package at issue here), you have managed to miss a crucial point. The government has never argued that the partial floats are a good idea because it has a mandate to govern. The government thinks that they’re a good idea anyway, and that its mandate merely weakens moralistic-sounding arguments opposing the peaceful policy. Not unreasonable.

In truth, mandate talk bores me. Most New Zealanders oppose this piece of government policy, but clearly a large potion of those opposing did not, taking all other matters into account when it mattered on 26 November 2011, care about it so much as to alter their votes. Democracy seems to be working as advertised. From my point of view, it’s positively a good thing about this Government that it seems determined to teach voters the meaning of buyer’s remorse. It’s a valuable lesson.

Finally, since I’ve dismissed some your party’s standard arguments against partial SOE floats and not even deigned to comment much about others (notably the outright racist invocation of the foreign-ownership bogey generally in the same breath as racist hectoring over the private Crafar farm sales), I think it’s only fair that I offer a suite of arguments against the policy that I think are more reasonable. Your mileage on these, like mine, may vary.

  1. The Government should consider more carefully alternative means of raising or releasing capital. Options include increasing tax, raising debt through bond issues, or reducing outlays in other areas of government spending. There are others. (I presume for now the option you prefer is part of a broader and coherent package for the management of the government’s books, but this isn’t something I’m prepared to take on faith for much longer.) In the present global economic environment, some options are more easily defensible than others, but none of the avenues I have mentioned are no-hopers in economic terms.
  2. The government shouldn’t be reducing its exposure to the energy industry. The Government has contended that its programme is an exercise in shifting the capital assets side of its balance sheet away from energy and towards “social assets” like schools, hospitals, and somewhat strangely, roads and fibre-optic cables. You may wish to argue that the state’s current level of exposure to energy is ideal. I’m not sure how you’d do this without the conversation drifting back towards the unenlightening dividend-stream argument, but you’re welcome to go ahead and try.
  3. The Government will misallocate the capital it releases. I totally agree. Roads, fibre-optic cables, schools, and hospitals are very nice, but it’s not at all clear to me that they’re necessarily the best use of government capital. I’d like to see more evidence, and a higher standard of business cases offered.
  4. A partial float will not do much to improve the firms’ standard of governance and profitability. This argument, made often and well by economist Paul Walker, fundamentally undermines the Government’s claims about the partial floats’ potential benefits to the New Zealand economy. As an added bonus for your team, it is more or less a slam dunk. Walker’s other objections to the Government’s policy are also strong.
  5. The state’s interests, shareholders’ interests, and the public interest may not be well aligned. Can we really trust governments, present and future, to let these companies fail if they’re mismanaged into loss land? Can we really trust them not to hobble potential competition from new and innovative market entrants? Can we really trust them not to saddle the companies with majority shareholder directives aimed not at increasing profits but at scratching some other political itch?

See how I did that? I raised, in an honest way, reasons I’m not wild about the government’s programme. My arguments didn’t resort to partisan hackery, and didn’t resort to wildly inappropriate historical analogies. It’s possible. You should try it one day. After your first term. Until then, you should probably just watch how the grown-ups in your caucus do it.

With less disrespect than the above would suggest,
Emil.

Update: I wrote this last night and schedules. Between the time of writing and the time of publication, Dr Woods apologized gracefully. Bravo.