Margin of Error

I don't think I've ever followed an election where the polls were as horribly off the mark as they were in Alberta.

Last May, when the media jumped on the "pollsters blew it" bandwagon for not projecting a Tory majority, most companies were still within the margin of error on the final vote intent numbers. Even during the 2004 federal election, the case study in pollsters "missing" a late swing, there wasn't a poll the final week of the campaign that had the Liberals behind (even if seat projections did), and most only under-estimated Liberal support by 3-5 points.

But last night? This wasn't just a case of shanking a field goal "wide right", but booting it in the complete opposite direction of the goal posts. Here's how the final polls stacked up with the results.

As most have commented, Forum's Sunday afternoon poll picked up part of the late swing but, even then, to go from a 2-point Wildrose lead and 10-point PC win is under 24 hours is shocking. It wasn't just a case of last second "strategic voting", since most polls in the final week correctly pegged Liberal and NDP support levels.

So what went wrong? I can think of 6 possibilities:


1. The polls made little effort to screen out the 43% of Albertans who didn't bother to vote on election day. Just asking respondents if they were absolutely certain to vote would have been a good start, even if few followed through on those intentions. But there are other attitudes and demographics that can help predict intent (i.e. older people are more likely to vote), and because of a lack of transparency in how these questions were asked or weighted, we have no way of knowing what steps were taken to screen out unlikely voters.


2. Building on the above point, the Big Blue Machine may have had a superior get out the vote operation than the relatively new Wildrose Party. I suspect this is part of the reason the federal Conservatives have "over performed" the polls on election day in recent years. Still, the best GotV operation will only bump you up a few percentage points, and it's not like the Wildrose Party was short of former Tory organizers, money, or volunteers.


3. The PCs had better candidates and more incumbents. Even though local candidates rarely have a big impact on the results (see Quebec, 2011), it's possible Albertans "voted" for the party and leader they wanted when asked that question on the survey, then considered the local candidates when they saw the names on the ballot. Still, once again, I can't imagine this would translate to more than a point or two at the province-wide level.


4. With voters growing increasingly disengaged and disinterested in the political process, it's possible many simply made up their mind in the voting booth. Since most polls only asked vote intent, there was little analysis in terms of strength of support, or where undecideds and soft voters might break before election day.


5. The most popular theory is that there was a "late swing" back to the PCs. This is born out by the Sunday Forum poll but, even then, a 20-point swing in the margin over the course of 5 days, or a single day 12-point swing is almost unheard of in politics. I don't doubt there was a late shift, but from what I hear, the PC Party's internal numbers showed them in much better shape than any of the media polls, suggesting that Smith's lead was never as big as it was reported.


6. So how could all those polls have been wrong? Well, if you look at that table above, you'll notice that Leger was one of the closest to the final mark, despite leaving field a week prior to the vote, before any "swing back" to the PCs was fully felt. The pollsters who overshot Wildrose support the most all used robo-diallers and online panels.

Both of those methodologies have inherent problems. You often need to make 50 to 100 robo calls to find one sap willing to complete the survey. So we know the Wildrose Party was popular with shut-ins, but that's about it. Moreover, since robo calls can only ask 5 simple questions before respondents drop off, you rarely have the opportunity to collect enough demographic information to judge how representative the sample is.

You can get those demographics using online panels, but while a national panel will have hundreds of thousands of Canadians on it, you're fishing from a much smaller pool when you get down to the Alberta level. Companies who don't frequently conduct political polling in Alberta might not have a good understanding of the biases inherent to the panel they're using, opening up the risk of skewed results.

If you're looking for more background on some of the problems associated with robocalls and online polls, I'd suggest this excellent letter by Darrel Bricker and John Wright, or this article featuring blunt comments from Allan Greg and Andre Turcotte.

The blame doesn't rest solely on the polling companies. The fact is robocalls and online polls are cheap to produce, and that's all the media is willing to pay for. The internal Tory polls used live callers, and asked more demographic and attitudinal questions than just vote intent - this no doubt let them verify the validity of their sample, and provided direction on what levers could cause the public to swing back to the Tory fold. There's something to be said about the old "you get what you pay for" adage, and most newspapers simply don't have the budget to invest in getting the job done right.


We'll probably never know which of the above factors were actually in play. And hell, this being Alberta, it could just be part of the deal with the devil the Alberta PCs signed long ago that ensures PC victory after PC victory.