What past party elections reveal about next Labour leader's chances of success

http://www.theguardian.com/politics/blog/2015/aug/10/what-past-party-elections-reveal-next-labour-leader-success-jeremy-corbyn

Version 0 of 1.

As we all contemplate the prospect of Jeremy Corbyn as Britain’s next prime minister, it’s worth remembering it is little more than 50 years since Sir Alec Douglas-Home emerged as the new Tory party leader and claimed the keys of No 10 on the basis of a secretive process denounced at the time as “the magic circle”.

Are there clues as to a new leader’s prospects for success with the wider electorate to be found in the way their party goes about choosing between rival candidates? Then as now, I think there are. After all, the rules have changed greatly, the party franchise expanded – as the grassroots Corbyn insurgency dramatically shows.

As it usually is in matters of internal party democracy and accountability, Labour was ahead of the Tories in 1963. It elected its leaders by a series of exhaustive ballots among MPs at Westminster, the contender with least votes eliminated until a winning candidate emerged with more than 50%.

The previous winter Harold Wilson had just beaten the volatile George Brown and Jim Callaghan (Wilson’s successor in 1976) in the contest to succeed Hugh Gaitskell – who had died suddenly – as opposition leader, soon to be PM.

Contrast that with the last gasp of the patrician Tories. Shaken by the Profumo scandal and other difficulties, including a prostate operation, Harold Macmillan bailed out just ahead of the October 1963 party conference in Blackpool. Predictable mayhem ensued, from which a handful of party grandees advised the Queen to call for the foreign secretary, Douglas-Home (a 14th earl at the time) instead of the assumed favourite, Rab Butler.

The process was denounced by modernisers like Iain Macleod (who coined the “magic circle” phrase) and Enoch Powell, who both refused to serve. When Douglas-Home narrowly lost to Wilson in 1964, the Tories finally copied Labour – as they would again much later over female candidates – by letting MPs pick the new man. Abrasive Ted Heath, currently in the news, prevailed over the charming-but-idle Reggie Maudling and also-ran Powell.

Two years later, Jeremy Thorpe was elected leader of the Liberal party to succeed the gallant Jo Grimond. As with Heath, Thorpe won 50% of the votes but was left slightly shy of victory under the rules, though his rivals conceded.

Life was about to get more complicated. Labour’s five turbulent years of office from 1974 to 1979 came during the global oil crisis and also saw domestic unrest – from militant unions, inflation, IRA and coup talk – which stimulated growing attacks on the Westminster leadership for betrayal and incompetence in the face of harsh choices.

The mood in the Labour party then was much like it has been since 2010, albeit in an age with no social media and – as Patrick Wintour’s excellent analysis shows here – crucial ballast in the shape of dictatorial union bosses and obliging Scottish MPs who helped steady the ship.

In 1979-80 Jim Callaghan had stayed on as leader a year after his defeat (the grownup thing to do) in the hope of helping Denis Healey to the succession. It backfired when Labour MPs (some of them about to defect to the breakaway SDP) narrowly elected Michael Foot instead. Whereas Healey had led by 112 MP votes to 83 in round one, the also-ran votes fell to Foot by 139 to 129 in an atmosphere where many MPs were eager to appease the unions and the left, thereby avoiding the new mechanism known as “mandatory reselection”.

But the game was up for the parliamentary version of the magic circle. Callaghan and Foot had conceded the principle of a wider “electoral college” in which the unions would have 40% of the votes (then cast in winner-takes-all union blocs), and the parliamentary party (PLP) and activists 30% each.

All that was denounced as disgraceful and irresponsible at the time, but versions of the wider franchise have since extended to all three main Westminster parties – even the Tories, who let MPs pick the finalists but party members chose between them.

The case against MP-only ballots was and remains that it is elitist. The case against the ever-wider franchise which now allows Tories and Trots to pay £3 and join Labour’s open primary season is that it is both anti-elitist and irresponsible in the deeper meaning of the word. But neither argument is watertight.

In 1981 Tony Benn had used the new electoral college to challenge for Healey’s job as Foot’s deputy, albeit against the advice of many leftwing colleagues, who saw a left-left ticket (Benn’s disloyalty had alienated Foot) as less attractive than a left-right one. He was defeated by less than 1% and lost his seat in the Thatcher landslide of 1983.

That left the road open to Neil Kinnock’s “coronation” victory in all three selections of the electoral college to succeed Foot, a 71.27 % share of the ballot. He got almost half the MPs’ vote (14.778% of their 30% share), with Roy Hattersley, his rival and elected deputy, getting a quarter. The pair became the “dream team” which lost to Thatcher (1987) and John Major (1992).

What is conspicuous about Labour leadership contests in the new era is that, unlike those confined to the PLP only – let alone Tory elections – they have more often than not been foregone conclusions. In 1992 it was clear from the start that John Smith would defeat Bryan Gould and he did in all three sections, including winning over almost three-quarters of MPs: 90% of the college in total.

With Gordon Brown having refused to stand against Smith, Tony Blair faced him down when Smith dropped dead in 1994 and went on to win both the leadership and three general elections. In view of the way Blair is now seen by many activists, including those who – as Patrick Wintour says – either left in 2003 or were too young, it is worth noting that John Prescott and Margaret Beckett, both on the left, did better: they got 43% of the college between them, including almost half the union votes. Blair got two-thirds of the MPs. The ballot was more democratic because the union bloc votes were replaced (Smith’s legacy) by “one member one vote” (OMOV) rules.

Related: Jeremy Corbyn: in the new politics only the now matters | Matthew d’Ancona

Is it tempting to say that MPs are the better judges of a leader’s potential to win elections and govern Britain? Certainly the Tory record points that way. In 1975 Thatcher unexpectedly beat both Heath and the likely magic circle candidate, Willie Whitelaw, and went on to be a dominant vote-winning leader until her own backbenchers – much more ruthless than Labour – pushed her out via a leadership challenge in 1990. They then picked the emollient John Major over the charismatic but divisive Michael Heseltine to lead them to a fourth term. Major was backed by 185 MPs to Hezza’s 131.

When the Tories finally lost power after 18 years in 1997 the party did what parties often do in the wake of defeat. Backbenchers and activists relish the chance to embrace old principles and passions, tempered by the realities of office: they pick the candidate they want, not the one the voters warm to.

As with loveable bruiser Denis Healey in 1980, so with blokeish Ken Clarke now. In 1997 he led the field on the first two ballots but lost by 90 votes to 72 when John Redwood’s 38 votes went to rightwinger William (“Save the pound”) Hague, in round three. As Hague later warned David Miliband, he was too much of a “geek” to win elections. Clarke might have dented Blair’s 170-seat majority as Hague failed to do (he gained one net seat in 2001). MPs are not infallible.

By 2001 the Tories finally embraced a wider electoral system with disastrous immediate results. Party members got the final pick, as Julian Glover explains here. Michael Portillo, the bookies’ favourite, topped the first ballot but managed to be eliminated – doomed to linger on as a TV railway buff – leaving Iain Duncan Smith and Clarke, who had won the third ballot, to face the party voters. Again they rejected the pro-EU liberal Tory in favour of a certified loser. Activists are not infallible either.

IDS didn’t even make it to polling day, a fate which some predict will befall whichever candidate becomes the next Labour leader (if Labour finally gets ruthless). When he was forced out in 2003, Michael Howard enjoyed a Gordon Brown-style coronation and went on to reduce the Blair landslide to 67 seats in 2005 with some help from Iraqi Sunni insurgents and a disastrous US-UK occupation.

Howard stayed on to secure an orderly election (and promote his protege, David Cameron). He reviewed the widely criticised rules but decided not to change them. This time David Davis entered the race as favourite, a working class boy from a tough background, but got overtaken by the Etonian after a smooth conference speech. Clarke was eliminated first and Cameron picked up most of his votes to emerge with 90 MPs to Davis’s 57 and a significant 42 to the rightwing populist Liam Fox – arguably the nearest Tory equivalent to Corbyn in outlook and instinct.

Cameron won power by beating Gordon Brown, the man whose own “coronation” in 2007 undermines MPs otherwise better claim to electoral wisdom. When Blair finally quit after 10 years as PM, Brown succeeded in winning the nomination of no less than 313 Labour MPs of the 355 elected in 2005. Many felt sceptical but there was also a sense of inevitability.

No heavyweight challenger could be found, still on their feet and willing to face the Brown machine. Leftwinger John McDonnell’s plucky effort did not gather enough support to make the contest, as Diane Abbott did in 2010 and Jeremy Corbyn (“It’s my turn,” he joked at the outset) did in 2015.

Matthew D’Ancona offers a smart explanation here as to why he’s doing better than they did: in the age of social media it’s all about NOW.

As everyone knows “Jez We Can” Corbyn has only modest support among fellow MPs and is on the ballot paper because some colleagues thought he would enliven the contest. They certainly got that bit right. What the record of a wider franchise seems to show is that MPs are better at picking winners, but far from infallible.

After all, the Liberals, who like their modern incarnation the Liberal Democrats prided themselves on a fastidious electoral process, picked Thorpe when the contest was confined to MPs. He was very lucky not to go to jail for conspiracy to murder. In the 2007 Lib Dem leadership election, the activists narrowly picked Nick Clegg over Chris Huhne. One led them into a coalition trap, the other really did go to jail. It’s never easy.