Scottish independence: yes, no, or maybe?

http://www.theguardian.com/politics/2014/sep/12/scottish-independence-yes-no-maybe-voters-opinion-polls

Version 0 of 1.

It has been a week of polling whiplash. On Sunday, YouGov shocked the unionist campaign by finding Yes in the lead for the first time, by two points. Then on Thursday night, the same company had No back in front, this time by four points.

A day later, just as some were speculating that the wind had gone out of the nationalist sails, came the Guardian/ICM poll. That survey, the first telephone poll of the campaign, found the No camp's lead to be only two points at 51% to 49%, once the don't knows are excluded.

On a poll-by-poll basis, these are relatively big changes over a short period of time. But looking over a longer timescale, the overall trend is quite clear to discern. A 20-point gap a year ago had crept to 15 at the beginning of 2014, was 10 points a month or so ago, and is now averaging less than five points.

Voters have moved towards yes over the year, and this is reflected for the first time in a telephone poll too.

Yes leads, or is in a statistical tie, across all age groups except those aged over 60. Older people are of course historically more likely to vote but, in an election where turnout is expected to be exceptionally high, this could become less important.

Recent polls are all showing turnout well above 80%. In the ICM poll, 87% say they are certain to vote, and the figure is above 80% across all age groups. TNS surveys over the past six months consistently showed that 70%-75% said they were certain to vote. This leapt to 84% in the latest poll released this week.

Other factors are shifting as the polling date approaches. The one most likely to shift the outcome, if anything will, is that the gender gap is narrowing. Past data had women firmly in the no camp.

A YouGov poll on 4 August found 67% of women saying they would vote no, but this had fallen to 58% by 1 September. Recent TNS figures reflect a similar pattern: 41% of women surveyed in its latest poll intend to vote no, down from 49% a month ago, while the proportion who intend to vote yes has increased from 27% to 35%. ICM shows the yes vote among women at 45%.

Where there is not agreement across polls is in the number of voters who have not yet decided, in the final week before the referendum, how they are going vote.

Online polls have the don't knows on less than 10%, while ICM shows 17% and TNS 23%. The fact that there are marginally more women undecided than men is likely to benefit Better Together but, at the same time, the greater number of those unemployed or in part-time work who still don't know how they will vote – if at all – is potentially to the benefit of yes.

Though we should be cautious when looking at small subgroups, the ICM poll suggests those out of work who have made up their mind are firmly in the yes camp.

The trend of recent weeks doesn't though mean that yes will end up ahead. Since 1 August, 18 polls have been made public, of which only one had yes ahead.

Once margins of error and fluctuations within polls taken by the same company in short succession (the latest YouGov isn't too dissimilar to figures released just before the poll showing yes ahead) are considered, the overall figures and trends are in fact relatively consistent: the no lead still exists, but has got markedly smaller.

Aside from the disproportionate attention on a single standout poll, there have actually been only a small number of polls carried out. If there were 50 sets of polls released, the way we viewed anomalies – such as, perhaps, the YouGov poll that showed a yes lead – would most likely be different.

So, what's going on in Scotland? There is still time for politicians and events to change minds, but that lessens each day – more than 18% of Scotland's 4.2 million voters have registered to send their ballots by post, so many will already have made their big decision.

The other big unknown is what effect the record turnout might have. A record 97% of eligible voters are registered to vote. This forecasts what is likely to be a historic level of participation, far higher than the turnout levels of most UK elections. Many that have never voted, or haven't done so in a long time, will head to the polls on Thursday, and no-one's sure what effect that will have.

Also, polling referendums is inherently complex, and pollsters have often been wrong because of the lack of historical data, and uniqueness of questions.

Polls rely on past behaviour to interpret the present, and historic data in this case is scarce. How can we adequately analyse the behaviour of so many that lack an electoral track record? This difficulty is again compounded by the small number of polls being carried out in Scotland.

Given the trends of the past month, the narrow gap and these fundamental challenges, it's advantage no – if only just – for now.