This article is from the source 'guardian' and was first published or seen on . It last changed over 40 days ago and won't be checked again for changes.
You can find the current article at its original source at http://www.guardian.co.uk/commentisfree/cifamerica/2012/apr/27/which-pollster-performed-best-gop-primaries
The article has changed 10 times. There is an RSS feed of changes available.
| Version 1 | Version 2 |
|---|---|
| Which pollster performed best in the GOP primaries? | Which pollster performed best in the GOP primaries? |
| (30 days later) | |
| One question often asked is "who is the most accurate pollster?" My usual response is along the lines of "with a few noticeable exceptions, for most pollsters, one is as accurate as another on average. Average each pollster in a particular contest, and you'll likely end up to the truth." That inevitably leads to the next question: "So, which aggregate should I be watching?" | One question often asked is "who is the most accurate pollster?" My usual response is along the lines of "with a few noticeable exceptions, for most pollsters, one is as accurate as another on average. Average each pollster in a particular contest, and you'll likely end up to the truth." That inevitably leads to the next question: "So, which aggregate should I be watching?" |
| So, the major three players: FiveThirtyEight, HuffPollster (formerly Pollster.com), and Real Clear Politics. They all use slightly different approaches to aggregating polling data, but all are looking at pretty much the same information. My past investigation into their accuracies during the 2010 elections found that none of the three was statistically significantly more accurate than either other. | So, the major three players: FiveThirtyEight, HuffPollster (formerly Pollster.com), and Real Clear Politics. They all use slightly different approaches to aggregating polling data, but all are looking at pretty much the same information. My past investigation into their accuracies during the 2010 elections found that none of the three was statistically significantly more accurate than either other. |
| But what about the 2012 primary season? | But what about the 2012 primary season? |
| Primary polling is a tricky proposition. This year, perhaps more than any other in modern political history, had ups and downs. One day, Newt Gingrich was up. The next, it was Mitt Romney. And there were some differences in how each firm came to their projections. | Primary polling is a tricky proposition. This year, perhaps more than any other in modern political history, had ups and downs. One day, Newt Gingrich was up. The next, it was Mitt Romney. And there were some differences in how each firm came to their projections. |
| FiveThirtyEight took into account momentum and past pollster accuracy; HuffPollster utilized Kalman Filtering; and Real Clear Politics kept its simple averaging technique. Maybe, we might think, the differences in technique would create differences in accuracy. | FiveThirtyEight took into account momentum and past pollster accuracy; HuffPollster utilized Kalman Filtering; and Real Clear Politics kept its simple averaging technique. Maybe, we might think, the differences in technique would create differences in accuracy. |
| So, let's look back at the 12 2012 primary contests where all three outfits posted a final vote estimate.* | So, let's look back at the 12 2012 primary contests where all three outfits posted a final vote estimate.* |
| In order to compute an "accuracy" measure, I had to standardize each outlet's data. FiveThirtyEight aims to allocate undecided voters, while HuffPollster and Real Clear Politics do not. I have allocated undecideds proportionally and any residual vote assigned to candidates no longer running or not viable (such as Buddy Roemer) to the viable still-running candidates at the time of the contest. [Mathematically, in Wisconsin, for example, this would be: Ron Paul's expected percentage of the vote divided by (Ron Paul's expected percentage of the vote plus Newt Gingrich's expected percentage of the vote plus Mitt Romney's expected percentage of the vote plus Rick Santorum's expected percentage of the vote) multiplied by 100. I used this forumula for the results also, and took the absolute difference between the results and the undecided allocated percentage.] | In order to compute an "accuracy" measure, I had to standardize each outlet's data. FiveThirtyEight aims to allocate undecided voters, while HuffPollster and Real Clear Politics do not. I have allocated undecideds proportionally and any residual vote assigned to candidates no longer running or not viable (such as Buddy Roemer) to the viable still-running candidates at the time of the contest. [Mathematically, in Wisconsin, for example, this would be: Ron Paul's expected percentage of the vote divided by (Ron Paul's expected percentage of the vote plus Newt Gingrich's expected percentage of the vote plus Mitt Romney's expected percentage of the vote plus Rick Santorum's expected percentage of the vote) multiplied by 100. I used this forumula for the results also, and took the absolute difference between the results and the undecided allocated percentage.] |
| After calculating each candidate's error, I measured the average candidate error in each state. I also took note of the greatest candidate error for each caucus or primary. | After calculating each candidate's error, I measured the average candidate error in each state. I also took note of the greatest candidate error for each caucus or primary. |
| The results demonstrate that 2010 was not a fluke: all three aggregates were, on average, about as accurate as each other. Some were more accurate in one state than the others, but it wasn't really consistent. HuffPollster either was, or was tied for, the most accurate measure in the most states, numbering seven. HuffPollster's aggregate candidate error in the 12 states was also tied for the "lowest", with an average of all the aggregates, at 2.2 points, but that was statistically insignificant. | The results demonstrate that 2010 was not a fluke: all three aggregates were, on average, about as accurate as each other. Some were more accurate in one state than the others, but it wasn't really consistent. HuffPollster either was, or was tied for, the most accurate measure in the most states, numbering seven. HuffPollster's aggregate candidate error in the 12 states was also tied for the "lowest", with an average of all the aggregates, at 2.2 points, but that was statistically insignificant. |
| In each state, there was usually at least one candidate whose performance the aggregates missed by a wide margin. | In each state, there was usually at least one candidate whose performance the aggregates missed by a wide margin. |
| The average greatest candidate error for all the outfits was 4.3 points. Interestingly enough, an average of all the aggregates produced the lowest greatest candidate error on average, at 4.2 points, but that was not statistically significant. | The average greatest candidate error for all the outfits was 4.3 points. Interestingly enough, an average of all the aggregates produced the lowest greatest candidate error on average, at 4.2 points, but that was not statistically significant. |
| All of these facts confirm what we already know: primary polling is good, but not great. Most candidates will fall within a few percentage points of the expected result (as seen with the average error). But with a greater number of candidates than in general election contests, there seems to be a greater chance that one candidate's percentage of the vote will be significantly under- or over-estimated. | All of these facts confirm what we already know: primary polling is good, but not great. Most candidates will fall within a few percentage points of the expected result (as seen with the average error). But with a greater number of candidates than in general election contests, there seems to be a greater chance that one candidate's percentage of the vote will be significantly under- or over-estimated. |
| Sometimes, we take note of an error such as in New Hampshire, where Ron Paul's result was greatly underestimated by all the polls and aggregates. We are less likely to take much account of a wide Newt Gingrich miss in Wisconsin. | Sometimes, we take note of an error such as in New Hampshire, where Ron Paul's result was greatly underestimated by all the polls and aggregates. We are less likely to take much account of a wide Newt Gingrich miss in Wisconsin. |
| I should note that other techniques of measuring error can deliver different results. In a 1 March post, FiveThirtyEight's Nate Silver looked at the margin between the top two candidates in all the contests through Arizona and Michigan. He found that the average difference between FiveThirtyEight's final forecast and Real Clear Politics on the average error of margin between the two top candidates was 1.6 percentage points – with FiveThirtyEight ahead. My own method of assessment in those contests finds FiveThirtyEight still ahead of Real Clear Politics, but by only 0.3 of a point; and I have HuffPollster ahead of FiveThirtyEight by 0.1 of a point. | I should note that other techniques of measuring error can deliver different results. In a 1 March post, FiveThirtyEight's Nate Silver looked at the margin between the top two candidates in all the contests through Arizona and Michigan. He found that the average difference between FiveThirtyEight's final forecast and Real Clear Politics on the average error of margin between the two top candidates was 1.6 percentage points – with FiveThirtyEight ahead. My own method of assessment in those contests finds FiveThirtyEight still ahead of Real Clear Politics, but by only 0.3 of a point; and I have HuffPollster ahead of FiveThirtyEight by 0.1 of a point. |
| The reason for such variance depends on whether we look at all the candidates or just the top ones, and on whether we allocate undecideds. In a contest with more than two major viable candidates, I see no real reason not to examine the accuracy for all the candidates' projected results. In terms of the undecideds' allocation, most authorities in academia and most respected methods of judging accuracy, such as the widely praised Traugott, Martin and Kennedy, do allocate undecideds proportionally, in one way or another. That's why I've done so here. | The reason for such variance depends on whether we look at all the candidates or just the top ones, and on whether we allocate undecideds. In a contest with more than two major viable candidates, I see no real reason not to examine the accuracy for all the candidates' projected results. In terms of the undecideds' allocation, most authorities in academia and most respected methods of judging accuracy, such as the widely praised Traugott, Martin and Kennedy, do allocate undecideds proportionally, in one way or another. That's why I've done so here. |
| The truth, however, is that the differences in how one measures "accuracy" come down to a few decimal points here or there. The FiveThirtyEight, HuffPollster and Real Clear Politics averages will get you about as close to the final outcome as one another. I don't recommend looking at one over another. I, in fact, read and average all three. | The truth, however, is that the differences in how one measures "accuracy" come down to a few decimal points here or there. The FiveThirtyEight, HuffPollster and Real Clear Politics averages will get you about as close to the final outcome as one another. I don't recommend looking at one over another. I, in fact, read and average all three. |
| * In Iowa, FiveThirtyEight posted two Iowa estimates. One "standard" and one "Iowa-specific" model. I've measured the accuracy of the Iowa-specific forecast because it was the final one posted, and readers of FiveThirtyEight were told it was likely to be more accurate. In Iowa, New Hampshire, South Carolina and Florida, HuffPollster posted its "sensitive" trend as the one it followed for its caucuses and primaries. I've measured the accuracy of the "sensitive" trends because they were the ones posted in the final estimate article. | * In Iowa, FiveThirtyEight posted two Iowa estimates. One "standard" and one "Iowa-specific" model. I've measured the accuracy of the Iowa-specific forecast because it was the final one posted, and readers of FiveThirtyEight were told it was likely to be more accurate. In Iowa, New Hampshire, South Carolina and Florida, HuffPollster posted its "sensitive" trend as the one it followed for its caucuses and primaries. I've measured the accuracy of the "sensitive" trends because they were the ones posted in the final estimate article. |
| Our editors' picks for the day's top news and commentary delivered to your inbox each morning. | |
| Our editors' picks for the day's top news and commentary delivered to your inbox each morning. Enter your email address to subscribe. |