Don't let dubious Pisa league tables dictate how we educate our children

http://www.theguardian.com/commentisfree/2013/dec/01/dont-let-pisa-league-tables-dictate-schooling

Version 0 of 1.

The triennial results from the Programme for International Assessment (Pisa), due on Tuesday but trailed in the Sunday press, have become education's equivalent of the football World Cup. And the performance of the British teams is just as mediocre, giving more leverage to politicians determined to get a few easy cheers from slagging off teachers.

In the 2009 tests the UK was around average, with England, Scotland and Northern Ireland roughly on a par, and Wales doing even worse. The English and Scots did better than Peruvians, as you'd expect, but they were not up there with the Chinese, Singaporeans, Koreans, Japanese or Estonians and Poles. Little change this time, apparently. So much for Tony Blair's "education, education, education", for it was under his regime that the 15-year-olds who took the latest tests in maths, science and reading in 2012 got most of their schooling. Michael Gove will take that as vindication of his policies, which involve transforming everything from curriculum to ownership of schools.

The judgments of Pisa, which tests more than 500,000 pupils in 66 countries – and also looks at factors that might influence the scores such as education spending and school autonomy – are treated as authoritative and unquestionable. One of the few areas where UK children are above average is in being happy at school. Those world-class Koreans are bottom of that league table, and the Estonians and the Poles aren't far above. Gove will probably announce a national drive to raise misery standards in schools.

As they say in sport, you can't argue with the scoreboard. But in Pisa's case, we can and should. There are ample reasons not only to question whether average scores from written tests can adequately assess the quality of school systems across the planet, but also to argue that international testing regimes pose a threat to national sovereignty and cultural diversity.

For an international test to work, all students have to answer the same questions, or at least questions of similar difficulty. In one obvious sense, they don't: the questions are translated into different languages which, according to one Norwegian academic, "results in rather strange prose" in his country. Besides, "literacy" in Finnish or Korean, where words are consistently written as they are spoken, is different when compared with literacy in English.

Danish academics, when they analysed the 2006 Pisa tests, found that eight of the 28 reading questions were deleted from the final analysis in some countries. Moreover, about half the students participating that year weren't tested on reading at all. The OECD, which runs Pisa, says it calculates "plausible values" for the missing scores, and this is a standard statistical device. But it's a hard idea for most of us to get our heads round, and many statisticians dispute its validity, suggesting that the results are nonsensical and meaningless.

Problems also arise from different cultures, and different attitudes to education in general and tests in particular. For example, French students won't guess the answers to multiple-choice questions; they decline to answer, though a guess gives at least a 25% chance of being right. East Asian countries always do well; critics argue that's not because their schools are brilliant but because of deference to authority and an anxiety for success that leads parents to seek intensive out-of-school tutoring. There's scope for gaming, even cheating, since the tests are supervised by research institutes in each country. Some countries are suspected of excluding their weaker performers.

To be fair, the OECD reports contain numerous caveats and warn of margins of error in their league tables. But that won't stop education ministers such as Gove and his opponents such as Labour's Tristram Hunt from mining the data to make political capital. Disgracefully, Gove used the 2009 results to claim that English schools had gone "down, down, down" since 2000, when test results were better and 15-year-olds had mostly been educated under the Tories. He ignored clear warnings from the OECD that the 2000 results were flawed and shouldn't be used for comparisons.

Indeed, the Pisa results provide little support for Gove's ideas, and still less for Boris Johnson's. Finland, the most consistent high performer, has the least selective, most comprehensive system in the world, and it has no inspectors, no exams before 18 and a national curriculum that is confined to broad outlines. Sweden really has gone "down, down, down" since it introduced the free schools that Gove has imitated. The US, with its charter schools also a model for Gove's free schools, does no better and, in mathematics, it does much worse than England.

But we should be wary of cherry-picking the results or, for that matter, paying any heed to them at all. Andreas Schleicher, the OECD man in charge of Pisa and once described by Gove as "the most important man in English education", wrote last year that "deficiencies" in UK education would lead to £4.5 trillion in lost economic output over a lifetime. To which we should respond: how does he know and what does it matter anyway? Behind the Pisa tests lies an ideology that accepts economic growth and competitiveness as the sole aims of schooling. The definition of educational success is being standardised and it is being narrowed.

For fear of falling behind we must all adopt "best practice" as revealed by the OECD. By focusing on economic imperatives, schools risk losing sight of their roles in nurturing social solidarity, passing on cultural heritage and promoting civic engagement. Might justice, social harmony and a clean environment be just as important for our children's future as economic prosperity?

By all means, learn from what others are doing in their own schools. But don't allow league tables of dubious provenance to dictate how we decide to educate our children.

Our editors' picks for the day's top news and commentary delivered to your inbox each morning.