Tuesday, May 03, 2011

Predicting the Unexpected

I'll announce the winners from my election pool later this week. One of the questions there asked which polling company's final poll numbers would hit closest to the mark.

You can browse the numbers here. To pick a winner, I simply added up the difference between their numbers and the results, producing the following:

Angus Reid: 5.2%
Nanos: 5.5%
Ipsos: 6.0%
Decima: 6.4%
Leger: 7.2%
Abacus: 9.0%
Forum: approx 9% (BQ and Green numbers extrapolated)
Ekos: 10.3%
Compass: 14.0%

So congrats to those of you who picked Angus. The top 6 companies on that list were within the margin of error on their numbers, so they too deserve a round of applause.

As for the seat projections, here's the total seat miss:

Riding by Riding: 52
LISPOP: 56
Calgary Grit: 56
Ekos: 58
Democratic Space: 58
Trendlines: 59
308.com: 98
Election Prediction Project: 118

So a similar performance by all the mathematical models, except for 308 who has already offered a brief post mortem. I will add that my prediction was further off the mark from my projection - I made the same faulty assumptions as the EPP did, assuming strong incumbents could hold their seats.

The largest problem with my projection was the polls it fed off - specifically the low Conservative numbers (which I did foresee as a potential problem). If I plug the actual numbers in, my model projects: CPC 168.8, NDP 94.6, Lib 34.0, Bloc 10.1. The regional splits break down nicely too, except for Quebec where I'm a bit high on the Bloc and low on the NDP.

But this model was supposed to handle pollsters missing the mark. A few of the results fell outside the 95% confidence interval so this is, as Jack Layton would say, a hash tag fail.

I'll put this one to bed for a bit and start tinkering again over the summer, but I think this speaks to the limitations of any seat projection model. They're useful tools, but it's incredibly naive to assume they can predict the total seat count, much less individual riding results.

But that's ok. If they worked, it would make election nights a bore.

Labels: ,

14 Comments:

  • All of the polls failed, particularly with the Conservative vote, which is the one that counted the most.

    But honestly, wasn't that clear from the start.

    By Anonymous Anonymous, at 1:46 p.m.  

  • Not to gloat, but my projection (cdnelectionwatch.blogspot.com) was closer than all the ones you listed. However, The Mace (themace.ca) did even better.

    The polls were fairly accurate in Québec, but they were a disaster outside of it. Taking a polling average would have underestimated the Conservative vote by at least 5% in every region outside Québec.

    By Blogger Election Watcher, at 1:57 p.m.  

  • Yesterday in your "what to watch for" post, you wrote that the NDP performance in Quebec would indicate how important it is to have a ground game, so clearly the ground game doesn't matter at all. Today you note that incumbency doesn't count for what you thought it does. I think these observations are true, and have the same root cause - disengaged voters who don't know or don't care about their local representative. It makes me a little sad.

    By Blogger Don, at 3:32 p.m.  

  • Dan, thanx for your seat projections during the campaign. We've added all 14 2011 model results to the 2004/2006/2008 Scoreboard @ http://www.trendlines.ca/free/elections/Canada/electcanada.htm

    By Anonymous Freddy Hutter, Trendlines Research, at 3:57 p.m.  

  • Maybe the problem is exiting the field too early. Nanos Sunday only numbers were closer to the mark:

    CPC 38.7% NDP 30.5% LIB 20.9% BQ 5.0% GP 3.7%

    But they did not publish their regional breakdowns for Sunday only. Not living there the extreme inefficiency of the BQ vote was a surprise.

    By Anonymous mikey, at 4:03 p.m.  

  • The problem isn't the polls - they give an fairly accurate snapshot of where people are on any given day.

    The problem is modeling how people are going to break when they actually step into the voting booth.

    Which support is soft? Which party has momentum - upward OR downward? Which candidate will voters flee from at the last minute, and which will they hold their noses but vote for anyway?

    I attempt to answer some of those questions in my election post-mortem: "You have to outrun the bear" and other iron laws of politics" (http://redgreenandblue.org/2011/05/03/canadian-election-wtf-happened-an-analysis/)

    By Blogger Jeremy, at 4:31 p.m.  

  • The Ekos projection of the popular vote was actually dead on (possibly the most accurate pollster) IF you only include voters that said they were certain to not change their minds:

    CPC: 38.64%
    NDP: 30.05%
    LPC: 18.94%
    Green: 5.1%
    Bloc: 5.7%
    Other: 1.5%

    This is the second election in a row that I have seen markedly lower errors from their numbers by using that correction. I think the general problem is that polls have too soft a likely voter screen, which underestimates the Conservatives and overestimates the others.

    By Anonymous hosertohoosier, at 5:46 p.m.  

  • Throughout this election I've been insisting that the pollsters that I trust the most are Nanos and Angus Reid. They were the two most accurate last time, and Nanos was the most accurate in 2006.

    Ekos is consistently among the worst, and I've been dismissing Ekos polls throughout this campaign. One of their biggest limitations is that they consistently inflate Green party support.

    By Blogger Michael Fox, at 9:32 p.m.  

  • Ekos was not that far off in 2008, however, they had a 40,000 person sample back then.

    By Anonymous hosertohoosier, at 10:00 p.m.  

  • It's sad but this is the crowning achievement of my entire life. Sadly, after being "right" about the election, my readership jumped from 17 (people, not thousand people) to 23. Then I fell into a depression, didn't pay the bills, and now my website is no more. :(

    By Blogger Unknown, at 5:31 p.m.  

  • The guy is absolutely fair, and there is no suspicion.

    By Anonymous disfrutar del sexo, at 8:54 a.m.  

  • By Blogger raybanoutlet001, at 2:03 a.m.  

  • By Blogger raybanoutlet001, at 2:00 a.m.  

  • By Blogger Unknown, at 10:30 p.m.  

Post a Comment

<< Home