Take us seriously. Or else...

Elizabeth May, facing internal dissent and leadership challenges from within, has picked up an enforcer:

The federal Green Party is hoping a former NHL tough guy will help beef up the party's presence in Quebec.

Green Leader Elizabeth May on Saturday named Georges Laraque as a deputy leader of the Green Party of Canada.

The 260-pound former Montreal Canadiens enforcer joined the Greens last February but his role has been undefined until now.

The announcement is part of an attempt to gain traction in the province.

Thinking that Georges Laraque will help the Greens in Quebec only shows how out of touch with Quebecers (or, at the very least, Habs fans) May is.

So You Think You Can Dance

The first rule of political events is usually "don't dance" (followed closely by "avoid hairnets").

But, then again, if playing the piano can humanize Stephen Harper, why shouldn't Ignatieff bust a move or two? The man looks like he's enjoying himself on the Liberal Express, which is a refreshing change for a politician who hasn't seemed overly comfortable with politics since jumping into the game 4 years ago.

Quick! Someone show these guys some polling data...

Ignatieff's proposed asbestos ban may have killed deal

A mining company says Michael Ignatieff might be responsible for stalling the reopening of one of Canada's last asbestos mines.

The Jeffrey Mine says it had a Chinese investor lined up and willing to invest $40 million, but that its would-be partner became scared over comments from the Liberal leader.

Ignatieff has called for a ban on exports of the potentially deadly substance. His spokesman, however, expressed skepticism Thursday that the Liberal leader's remarks might have torpedoed the deal.

[...]

"A week after Mr. Ignatieff's comments, they started questioning me, saying, 'What happens if we decide to invest in you and Mr. Ignatieff becomes prime minister and decides to stop production of chrysotile asbestos?"'

I too, am skeptical that Ignatieff killed this deal, but if he did...well, then, good for him. It's unconscionable to export a product we won't use here for safety reasons when we know it's killing millions of people in the developing world.

The Comeback Kid?

The good people at The Mark asked me to pen a short essay on a Canadian historical comeback Michael Ignatieff could learn from. So I took the most recent example, focusing on Stephen Harper circa 2005 - that infamous "Glad as Hell" tour when Harper went coast to coast to try and humanize himself, and ended up a laughing stock. But here's the take home message from that:

I would argue the real lesson to be learned in Harper’s comeback wasn’t that he rode the wave of scandal to 24 Sussex. It was that he only won after presenting a clear vision that resonated with Canadians. Harper’s “all scandal, all the time” campaign in 2004 didn’t work, so in December 2005 he started to campaign with a policy announcement every morning. He had priorities that showed he stood for something: lower taxes, cleaner government, and getting tough on crime. Suddenly, Stephen Harper didn’t look so scary.

You can read the rest of the article, and the mandatory leather vest jokes here. Also on The Mark are comparisons to Jean Chretien (by Misters Jedras and Silver), Robert Borden, and Pierre Trudeau.

To that list, I'd add one more case study, which I considered writing about - Mike Pearson. Think about it. Before the 1963 election, he'd already lost twice. He certainly wasn't charismatic. He was an academic who had lived in Europe and the US.

Yet somehow, he won. And he only won after promising "60 days of decision".

I know I'm beating a dead horse here (hey, it could be worse, this could be another post about the Census) but I'll make the point yet again - to win, Ignatieff needs to show Canadians what he stands for.

Everything You Always Wanted To Know About The Census (But Were Afraid To Ask)

The Great Census Crisis of 2010 has its first music video. It’s only a matter of time before angry mobs of statisticians take to the streets holding “I’m with Fellegi signs” and chanting “What do we want? Mandatory long form Census! When do we want? Every 5 years!”. Don’t worry, if you can’t make the rallies, you can always show your support symbolically by wearing a Census-themed pocket protector.

Now, I fully recognize that during the summer the only random sampling Canadians are concerned about involves six packs. So for those of you who haven’t been grabbed by the witty “senseless Census” headlines, I present a summary of what the !%@# is going on.


WHAT EXACTLY IS THIS ALL ABOUT?

Governments have been conducting censuses for thousands of years – and the good ones are never voluntary. As a Charlie Brown Christmas taught us, Mary and Joseph didn’t have much of a choice when it came to being counted.

In Canada, the Census takes place every 5 years. All Canadians get the short form which asks the basic questions – name, age, gender, marital status, language. One in five houses get the long form which asks you about your sexual history, voting habits, and embarrassing High School nicknames. I’m kidding of course, but that’s what the government would have you believe. In reality, it asks questions about everything from your income to your ethnicity to your daily commute. On average, you’ll have to fill out 2 or 3 long forms in your life.

The debate focuses on the long form. Those trying to axe the Census argue these questions are an invasion of privacy. “Why the hell should the government know what time I leave to go to work?” they shout angrily on their twitter accounts and in Toronto Sun editorials.

“Well,” the other side argues “so that cities can build roads and public transit to help you get to work on time. Duh.”

The reality is we live in an information age, and long form Census data is a valuable source of information. Governments use it to help plan communities and programs. Hospitals need it to provide the right kind of services and fight pandemics. Researches use it to track demographic trends over time. Masters students, like Stephen Harper, use it to write thesis papers. Think tanks, like the Fraser Institute, use it to prove their kooky right wing theories. And businesses use it all the time – just think of restaurants and grocery stores that sell ethnic foods or cater to specific client demographics.


THE PROBLEM WITH MAKING IT VOLUNTARY

As soon as you make a survey self-selecting (i.e. voluntary), certain types of people are more likely to fill it out. That’s why you really don’t want to put too much stock in the web polls on the Globe & Mail’s website.

Studies in the past have shown low income Canadians, visible minorities, and aboriginals are less likely to fill out voluntary surveys (I might add, these studies could only show this because we have Census data as a point of comparison). So if you’re trying to ensure government programs to help aboriginals are working...and low income aboriginals aren’t filling out the long form...you have a problem.

That’s why the US quickly scrapped plans to use a voluntary census after experimenting with the idea in 2003. Imagine that! Making sure it works first, instead of making the change because of a few angry e-mails.


THE GREAT CENSUS CRISIS OF 2010

Three weeks ago, it was quietly announced that the 2011 long form Census would become voluntary. Instead of being sent to every 5th household, it would be sent to every 3rd household. The cost of this? An extra 30 million dollars.

At first, the government’s response was limited 140 characters – a few tweets exchanged between Tony Clement and angry economists. After all, as Tony has since let it be known, the government that spends millions promoting Canada's Economic Action Plan doesn’t believe government decisions “need to be shouted from every rooftop”.

I’m sure it never occurred to the man who said “only elites care about prorogation” that anyone would care about something as dull as survey methodology. Hell, Tony didn’t understand it himself and he was the Minister of Industry – how could normal non-elite Canadians be expected to understand the issue, much less care about it?

But slowly, people began taking notice and speaking out against the move.

The former head of StatsCan. The Canadian Medical Association. The Canadian Association of Business Economists. The Canada West Foundation. Municipal governments. Newspaper editorials (including those pro-government coercion communists at the National Post and Calgary Herald). Alex Himelfarb. Don Drummond (who sits on the StatsCan advisory panel which was...never asked for advice on the change). It’s a long list, and you can view it here. And here.

Oh, and all the provinces except Alberta now oppose the move – but even there, the City of Calgary has been critical saying it would “cripple” their decision making ability. And Edmonton Conservative MP James Rajotte (who is now assured of never getting into a Harper Cabinet) has broken ranks, demanding an explanation from Clement.

Faced with this backlash, the empire struck back this week. After all, the party which sends Happy Hanukkah cards to swing voters feels you have a right to your privacy. First, there was the Star Wars themed press release, and accusations that Census officials would break down your door at 10 pm while you were "trying to read" (reading? That sounds awfully elitist to me). Then, the Toronto Sun wrote an editorial comparing the long form Census to communism (you know who else conducted a Census? Hitler!).

Maxime Bernier has emerged as the government’s point man on this, which is understandable – if there’s anyone who understands how easy it is for confidential information to leak out, it’s Max. Bernier has laid out the government’s position which is, in short, that it’s wrong to “coerce” Canadians into filling out “intrusive” questions, under the threat of imprisonment.


LOOKING AT THE GOVERNMENT’S ARGUMENT

The problem with this argument, is that it’s inconsistent with the government’s actions.

If there are specific questions they feel are intrusive, they can be removed. Personally, I don’t find the Census any less intrusive than an income tax form, and data is only reported in averages and totals – there’s no way anyone anywhere will know what I wrote on my Census form.

If they don’t see any value in the long form, then they should axe it altogether and save the money.

If they feel it’s “coercive” to force Canadians to fill out a form, then why are they still being coerced into completing the short form? And why are farmers being coerced into filling out an equally intrusive agricultural long form? Furthermore, does the “tough on crime” party actually believe that threatening to put people in jail for breaking the law amounts to “coercion”? Oh, and I should add that no one has ever gone to jail in Canada for not filling out their Census form. Just thought I'd mention that.

In short, Clement has come up with a more expensive and less effective alternative.

Now, in fairness, the government has also claimed there's a groundswell of Canadians who feel the Census is intrusive. And, if voters did feel this way, there might be an argument to make for changing it. I’d argue the benefits of the Census still justify it (I mean, who likes paying taxes or being called for jury duty), but it would be a fair argument.

Maxime Bernier claims “thousands of e-mails” were sent to him complaining about the long form Census in 2006. Perhaps, but we have no proof of this. I guess it's possible Bernier misplaced them. What we do have proof from is 22 "expressions of concern" sent to StatsCan during the 2006 Census process, and 3 complaints to the privacy commissioner over the past decade. At the same time, the privacy commissioner has raised concerns about other programs which the government shows no interest in scrapping.

Still, the government’s decision appears to have found some support. Tony Clement has personally thanked 10 people by name on Twitter for their words of encouragement. By the way, Tony Clement has 3500 followers on Twitter.


WHAT NOW?

The government appears unlikely to back down and there’s no indication this issue has captivated the hearts and minds of Canadians.

Still, the Industry Committee will look into this. The head of StatsCan has resigned in protest. And groups opposed to the change will continue to raise a little hell.

So like it or not, expect a lot more news on the Census for the rest of this summer.

"In light of today's media coverage..."

This morning, Tony Clement gives an in depth Q & A in the Globe, where he talks about the relationship between StatsCan and the government, their role in the Census changes, and his relationship with chief statistician Munir Sheikh.

Clement ends the Q & A by assuring us that both him and Mr. Sheikh will release statements today in support of the changes:

Q: Ok. What is the point of the statement tomorrow? Why more statements?

A: I think Munir wants to assure Canadians that Statscan is going to do its job -- and [explain] the nature of what that job is -- and then I will want to assure Canadians that we have confidence in Statscan -- that it can do its job.

And now...

Head of StatsCan mulls future over census crisis

OTTAWA - The head of Statistics Canada says he's "reflecting" on his position at the agency, the latest twist in the crisis over the government's decision to scrub the mandatory long-form census.

Munir Sheikh issued a terse email to all agency employees today cancelling a planned staff meeting and saying he would comment soon — sparking speculation from insiders that he might resign.

[...]

"In light of today's media coverage, I am cancelling the scheduled Town Hall meeting," wrote Sheikh, a respected economist with service in both Liberal and Conservative governments.

"I am reflecting on my position and that of the agency and will get back to you soon."

UPDATE - The Head of StatsCan has resigned in protest, saying the voluntary Census will not work. But hey, Tony knows best...

Leaving the Hill

Jay Hill announces he won't run in the next election.

On the list of things I never thought I'd write on this blog...

A group of Canadians sing in defence of the mandatory long form Census.

Fifth Annual Politicians in Cowboy Hats

For a brief history of Stampede fashion, you can read the 2006, 2007, 2008, and 2009 round-ups.

You never quite now what you'll get at the Stampede. The boy from Calgary became a national laughing stock when he mistook the Stampede for a Village People convention in 2005. And the geeky Liberal pushing the carbon tax was named "best dressed" by both this blog and CP in 2008.

Of course, 6 months later Dion had been pushed out as Liberal leader, and 6 months after leathervestgate, Harper was Prime Minister. So it's best not to read too much into this (unlike, say, a broken down bus which we all KNOW is directly linked to electoral fortunes).


Federal Politicians

Once again, we have a surprise winner this year in the "best dressed" category. Why, it's none other than Mr. Ivory Tower Michael Ignatieff. The urban legend in Calgary is that no one had ever been able to find a cowboy hat big enough to fit Ignatieff's head, but for the first time in five years he has moseyed into town in full gear, hat included, and he doesn't look the least bit out of place.



Wait, I thought he said he liked the smell of barns! Flip flop!



Layton and Harper have been to enough stampedes that we know what to expect from them. Layton always winds up looking surprisingly at ease for a mustached Toronto socialist. And Harper? Well, since leather vest gate, he always plays it safe and gets a C+/B- grade. The man never looks comfortable, but at least he doesn't embarrass himself.





You know, despite all the talk of Harper playing chess, I've always thought his tactics were more of the "detonator" variety. Kelowna Accord? KABOOM! Census? KABOOM! Promises not to appoint senators? KABOOM!



But if Harper and Layton have become too predictable, let's all raise a glass of sillabub to toast Elizabeth May. Worst dressed in 2008. Best dressed in 2009. And this year? I'll reserve judgement until I can find a picture of her before she tumbled into the tie dye trough.




And, in the interests of by partisanship, a special shout out to stampede superstar Jim Prentice - yeah, he's got a bad record as environment minister, but he rides a horse in the parade every year and looks like he could handle himself in the rodeo.

Prentice has a wide range out outfits and he makes them all work. And the ladies can't get enough of Jim's talk of voluntary 40 year emission reduction targets.




Provincial Politicians

Ed Stelmach disappoints every year at the Stampede. In 2007, he had us all smacking our heads on the ground when he called it "the Alberta Stampede". The last two years, he has celebrated the Alberta Stampede by wearing a suit jacket over top of his cowboy duds. A major no-no.

And this year, it's more of the same. On the weekend, he went with a rather unflattering vest that Ed's mom appears to have sewn his name into, and on Tuesday he dusted off the suit for the third straight year.




But despite my criticism of Stelmach, I'm going to give his Finance Minister the title of "Worst Dressed" this year. I'm hesitant to do this based on a profile picture, but I can't imagine anything going on below the shoulders that could salvage this outfit.

Once again, Ted is just trying to hard. When he ran for leadership, he drafted a catchy little country music jingle. He holds "golf and gun" fundraisers. But, really, he's just a university professor from the big city trying to pass himself off as a good 'ol country boy. And, in this case, it shows.



Which brings us to David Swann who is an academic from the big city and doesn't pretend to be anything else. In fact, he rode his bicycle in the Stampede parade this year.

But you know what? Even though Swann looks completely out of place every year, he puts on the hat, the boots, and the belt buckle. He hosts one of the largest stampede breakfasts in Calgary. And he always looks like he's having a great time. That's something Albertans can respect.




Municipal Politicians

This is Dave Bronconnier's last stampede as mayor and I'll give the man props, he has always dressed, looked, and acted just like the Mayor of Calgary should during stampede. Hell, the man even rides a horse for crying out loud.

Clement to Farmers: Prepare to be Coerced

Farmers still face mandatory agricultural census

Canadian farmers will still be required to fill out a questionnaire about their farming methods, even though the federal government is scrapping the mandatory long-form personal census because it says it is too intrusive.

In conjunction with the regular census, Statistics Canada also surveys farmers across the country, asking detailed questions about crops and farming techniques.

For example, farmers are asked about the area of land fertilized with manure, whether that manure was spread on the soil, injected into it, or fell randomly from a cow. [ed note: speaking of random sampling...]

CBC News has learned that this survey of farms remains mandatory, although the national long-form household census is being replaced with a voluntary version in 2011.

UPDATE - Clement was on The House yesterday, and he defended the farm Census:

In the radio interview, Clement did not address the concern that the change will cost more money as the new voluntary survey will be sent to more households than before. But he did explain why the mandatory long-form agricultural census was not scrapped.

Clement says the agricultural census is used for valuable measures "that will help farmers," adding, "The argument obviously to farming associations and to farmers is, 'You fill out the form, it'll help the government help you in your farming activities.'"

He went on to say that the farm Census is the Minister of Agriculture's decision - so any farmers who feel coerced, can take up their grievances with Gerry Ritz.

The Fraser Institute: Truthiness Over Fact Based Research

Warren Kinsella defends Tony Clement, and a National Post editorial rips into the Tories:

This is profoundly undignified governance. If, as it seems, the government cannot defend changing the census on any logical, resonant or particularly urgent grounds, it should abandon the undertaking until it’s prepared to do so.

My world is upside down! Luckily, the Fraser Institute is here to restore the natural order of things into place:

The chief economist of the Fraser Institute supports the Conservative government’s decision to scrap the long-form mandatory census, saying voluntary surveys will yield enough accurate information about the country and critics saying otherwise are “vested interest groups.”

The Fraser Institute uses long-form data in its school report card rankings, Mr. Veldhuis said, but he believes the short-form census — which will remain mandatory — the voluntary census and other polling, market research and voluntary surveys conducted by private companies will provide more than enough data on Canadians.

The think-tank recently conducted a survey asking people about their tax returns, he said, and they used short-form census data to ensure they had even geographical distribution across Canada. When asked how they would ensure even representation across different income and ethnic groups on such surveys without reliable long-form data, Veldhuis questioned why those differences would matter.

“This is what should be worrying average Canadians — this information is used by central planners to plan how to tinker with the lives of Canadians,” he said of the ways in which census data is used.

Now, I could try to discredit Veldhuis - it certainly strikes me as odd that he doesn't think one's income would impact their opinions on tax returns. But no, I'll leave it to Tony Clement to attack him.

Instead, I ran a search of articles on the Fraser Institute's website for the word "Census". I got 313 responses. Here are some highlights from from articles that came up on the first search page:

1. Is Toronto in decline? Compares income numbers for Toronto from the 2001 and 2006 Census (long form question!). Also looks at Torontonians' employment sectors (long form question!).

2. Is there Really a Looming Labour Shortage in Canada and, if there is, can Increased Immigration Fill the Gap? Looks at Census data to see role of immigrants (long form question!) in the Canadian labour force (long form question!).

3. BC's Agricultural Land Reserve a costly failure responsible for the most expensive housing costs in North America. Cites Census data showing what percentage of income (long form question!) is spent on housing (long form question!) for different age groups.

4. To Fix Health Care, Follow the Money. Looks at detailed wage data (long form question!) by employment (long form data!).

Just think of all the G8 summits we could host for the same price!

This government is fiscally conservative...except when it comes to spending money:

Ottawa to spend $16-billion on fighter jets

he Harper government is confirming it will spend $16-billion for the latest generation of fighter jets from Lockheed Martin.

But the government is fending off complaints that making one of the biggest military purchases in Canadian history without a single competing bid is a waste of taxpayer money.

The Liberals and a former public servant who once headed the purchase project say the massive F-35 Joint Strike Fighter purchase should have been subjected to competitive bids.

Sadly, all my posts can't be about sample selection bias, and on the topic of military fighter jets, my expertise is limited to what I've seen in Top Gun. But, to me, 16 billion feels like a lot of money to spend outside of a competitive bid process.

And I suspect a lot of voters would feel the same way.

This Week in Alberta: Dance Off!

The Wildrose Alliance releases their response to "Ted Morton is the Man". I had my mocking muscles all ready to go on this one, but it's actually not bad.



No word yet on whether or not we'll be treated to a Swann song...

Seat Projections

For those who recall, I experimented with a seat projection model last election.

By and large, it worked as well as any of the other models out there. But the rub is, my model, like every other pundit, seat projection and octopus out there, underestimated the Conservative seat total and overestimated the Liberal one. The reason? The Conservatives over performed their polling numbers at the ballot box.

And that’s the problem with the seat projection game – it’s only as good as the polling data it uses. I think I’ve found a way to work around this, although I’m sure some tweaking will be required. But for now, I’m quite pleased with the model I’ve produced.

To read the big and long methodology behind it, click here. The short version of it is this: the model projects seats, taking the following into account:

-Publicly released polling data
-2004, 2006, and 2008 election results
-Riding demographics
-The historical variance in riding results, compared to regional results
-Accuracy of Canadian pollsters in predicting recent provincial and federal elections
-Incumbency
-By election results

For all these factors, I’ve done quite a bit of trial and error research – usually using the ’04 and ’06 elections to “project” the ’08 one. The end result is a projection system that has a few key advantages:

1. It has built in safeguards against “election day swing”, when certain parties over or under perform all the polls. It’s no use pretending this doesn’t happen, so it should be reflected in projection models, especially ones which rely on simulations. This may not make my “best guess” any more accurate, but it makes the model far more adept at calculating the probability of things like a Liberal win, or a Tory majority.

2. It’s robust. By looking at the past three elections and the riding demographics, this model smoothes out some of the fluky results that tend to plague seat projection models due to “weird stuff” like Elizabeth May running in Central Nova or surprise nudity.

3. It’s entirely objective. I plug the numbers into my excel sheet, run some simulations in R, and get the output.

4. It produces a probability in each riding. Most projection models that show the Liberals up by 1% in a riding will project the riding to go Liberal – in reality, if the Liberals are projected to be up by 1%, it really means you might as well flip a coin to pick the winner.

So, if you want the long explanation, click here. Otherwise, here are my projections, based on polling data released in June and July. Remember, this is what would happen if the election were held today - it's not a prediction of what would happen if the election were held in a few months. Campaigns matter.









Probability of Conservative Win: 100%
Probability of Conservative Majority: 1.6%

Seat Projection Methodology

Before I jump in, let me be clear on two points:

1. This model has evolved over the past 3 years (though Stockwell Day will deny it has) and each step in it involved a lot of thought, testing, and consideration.

2. I’m completely 100% open to criticism and suggested improvements. I’m sure it can be improved.

That said, let’s jump in.


THE PROJECTION FRAME

At the basic level, this is a uniform swing projection using regional data (Atlantic Canada, Quebec, Ontario, Prairies, Alberta, BC). If the Liberals go up by 5 points in Alberta, the model swings every riding up 5 points. If the NDP drops 8 points in Atlantic Canada, the model drops every riding down 8 points.

I experimented with geometric swings (including the variation 538 used for the UK election) and mixed models, but they simply didn’t work as well on any of the elections in my data set.

The reason for this is simple – if the Liberals swing up 5 points in Ontario, the geometric swing is going to give them almost all of their newfound support in rural areas, since that’s where there are the most votes available to swing to them. And that’s not what usually happens in reality.

Now, I’ll concede in a WTF election like, say 1993, when the political landscape of the country is dramatically changed, uniform swing may not work. I wouldn’t use this model to project the next Alberta election. But uniform swing is simply the best frame to build this model out of, especially since its deficiencies are easy to correct at the simulation stage.


THE BASE VOTE

Projection models generally use the most recent election as their base. The problem with this is that if a party has a good or bad showing in a riding, the data becomes skewed beyond repair. It's the same reason you use more than 1 year worth of data to project hockey or baseball stats - even though Aaron Hill hit 36 home runs last year, it’s foolish to expect him to repeat the feat.

In the political arena, there are prominent examples of this that stand out – the Greens won’t match their 2008 totals in Central Nova next election and, no matter how bad the campaign goes, the Liberals will exceed theirs. The NDP results from last election in Saanich Gulf Islands were obviously hurt by some naked truths that (hopefully) won’t be repeated in the next election.

But even on a more subtle level, parties get good candidates, candidates run bad campaigns, local issues emerge. There needs to be a way to smooth these events out.

To test this, I used a regression model to "predict" the 2008 vote in each riding, based on the 2006 and 2004 riding results and each riding’s predicted vote based on demographics (click here for the low down on this). In all cases, these numbers were adjusted using uniform swing.

And the results were clear – the best model used all three predictors (2004, 2006, and the demographic regression).

So keeping the ratios the same, I’m using the following results to get the “base” vote for each riding:

2008 election: 38.7%
2006 election: 17.4%
2004 election: 7.8%
Demographic regression: 36.2%


OTHER FACTORS

Incumbency exists. It means something. It should be taken into account. I’m not going to argue the point any further, because every study ever run on Canadian politics has come to this conclusion, as has my own research. So, I’m adjusting for incumbency based on the effect it had on the 2004, 2006, and 2008 elections.

By elections are a different beast. They’re unpredictable, and it’s hard to say how good a gauge they are of future results. My research on them has been limited, but the numbers tell me the best prediction model will weight the by election for 44% of the base. And who am I to disagree with the numbers?

The final bit of finessing I’ve used relates to the polling data (which I’ll talk about in a second - patience...). The Green Party has consistently under performed their polling numbers at the provincial and federal level in Canadian elections. As a result, I’ve scaled the Green polling numbers back to 78.55% of their value. Just make sure the angry hate mail is written on recycled paper before you send it.

My spreadsheet is set up so that I can easily remove these correction factors or change their impact. But in each case, I’ve given them the impact the data tells me to.


POLLING DATA

Right now, I’m taking the most recent poll from each polling company and assigning a weight to it based on the sample size and the company’s accuracy in provincial and federal elections over the past 5 years. Under this weighting system, the “best” poll is worth about twice as much as the “worst” poll.

This is the aspect of my projection model most likely to change in the coming months, and I’m open to suggestions. Things to consider are:

-Ekos releases massive amounts of data compared to other companies. They’ll interview 7,000 people a month – but is it “fair” to give that data seven times the weight of an n=1,000 poll from another company?

-Should weight be given based on the freshness of data? Is 3 week old data worth as much as 3 day old data? And if not, what’s the half life of polling data? Does this change during an election campaign?

-Is it fair to judge the accuracy of a polling company on past election results? Right now, pollster accuracy is being based on 8-12 data points. Hardly a large sample.


ADDING VARIANCE TO THE SIMULATION MODEL

Up to this point, I’ve described a very thorough uniform swing model. But a probabilistic model can do so much more. In most models, a seat the Liberals are projected to win by 1% counts just as much in their tally as a downtown Toronto seat they’re projected to win by 40%. Yet in reality, if we project them to win a seat by 1%, it’s basically a coin flip election – it could go either way.

So we need to make the data messy. Unfortunately, I worry my explanation will be messy as well. But here goes.

The first step is to find the regional support for each party in a given election simulation. This is done using the margin of error on the polling data. If I have 1000 interviews from Atlantic Canada, then the Atlantic Canada data carries a margin of error of +/- 3.1%. So the numbers get simulated under a normal distribution accordingly. What that means is if the Liberals are polling at 35% in Atlantic Canada, in some of my sim elections they’ll come in at 37% for the province. In others, 32%. Most of the time, they’ll be close to 35% but we’re talking about 10,000 simulations here, so in some of these “elections”, they may very well get 31% or 40% in the province. That’s just how margins of error and variance work.

After that, we need to add some noise when transferring polling data from the regions down to the ridings. To do this, I looked at how regional shifts have carried through to the riding level in previous elections – for example, if the Liberals drop 8 points in Ontario, they won’t drop 8 points in every riding. They’ll fall by 4 in same and by 12 in others.

So variance is added, keeping that overall regional polling number the same. Based on my research, the variance gets larger when the change gets larger (i.e. if a party goes up by a lot, their gains are a lot less uniform at the riding level) but even if a party’s support is unchanged in a region, their support will still change at the riding level. In English, even if the Liberals are polling at the same level in Quebec now as they got last election, they’ll go up in some ridings and down in others – but it will even out.

I won’t go into the exact mechanics of this, but I’ve test driven this numerous times and the program produces riding variance at the same level it should, based on what’s happened in the 3 previous elections.

But there’s one more level of variance this doesn’t take into account: The polls being wrong. There are elections when pollsters overshoot the margin of the error. We can have 10,000 interviews from 7 polling companies, and miss the bullseye by 3%. That’s not a knock on the pollsters – some people don’t vote, some people lie, and some just change their mind at the last minute. There’s no use pretending otherwise. Think of the 2004 Canadian election when the electorate swung back to the Liberals on the last weekend.

So, I’ve gone back and looked at how much, beyond normal sample size variance, pollsters have missed the mark by (in Canadian provincial and federal elections). And I’ve built this in to the very first step of the model. So even if we have reams of data showing the Tories at 35% nationally, in some of my sims they’ll “actually” be at 32%. In some, they may be at 37% or 38%. Again, the variance is added based on what we’ve observed in recent Canadian elections.


RECAP - HOW IT WORKS

1. Public polling data is grouped together by region.

2. In every simulation, the polling data is adjusted based on sample size variance and on how often polling companies just “miss the mark”.

3. From this, every riding is simulated based on how the numbers tend to transfer from the regional level down to the riding level.

4. The riding simulations take into account past election results, demographics, incumbency, and by elections.

Using this, my laptop simulates 10,000 elections. From this, I can calculate odds of a given party winning the election or a given seat changing hands.

As I said off the top, I’m open to suggestions on changes – I’m sure there are improvements that can be made.

As the call for state sanctioned coercion grows...

Via Aaron Wherry, a non-random sampling of opposition to the government's plan to defile the Census:

To those who oppose the government’s changes to the census you can now add the Statistical Society of Canada, the Federation of Canadian Municipalities, the Inuit Tapiriit Kanatami, the Canadian Marketing Association, the Canadian Federation of Francophone and Acadian Communities, the Executive Council of the Canadian Economics Association, the director of the Prentice Institute at the University of Lethbridge, the senior economist at the Canadian Centre for Policy Alternatives, the Canadian Institute of Planners, the Canadian Association for Business Economics, and the editorial boards of the Saskatoon Star-Phoenix, Calgary Herald, Winnipeg Free Press and Globe and Mail.

They join the co-chairman of the Canada Census Committee, Ancestry.ca, city planners in Calgary and Red Deer, the Canadian Association of University Teachers, the former head of Statistics Canada, and the editorial boards of the Toronto Star, Montreal Gazette, Edmonton Journal and Victoria Times-Colonist.

And hot off the presses, the Association of Municipalities of Ontario has sent Clement an open letter decrying the move. Also, Pundits Guide talks about how the Census is used, and Susan Delacourt finds it a wee bit odd that a political party which probably knows your voting record, ethnicity, and favourite TV show has issues with Stats Canada collecting Census information. Colby Cosh does link to one of the few supporters of the government's decision...and promptly rips him a new one.

Meanwhile, Tony Clement has yet to offer up an explanation over 140 characters for his decision, focusing instead on more important issues.

This Week in Alberta - We Need To Talk

On Wednesday, the Alberta Liberals ran full page newspaper ads in the Calgary Herald and Edmonton Journal urging other parties to co-operate with them to offer a progressive alternative to the Progressive Conservatives.

Even though it's hard to keep track of all the new parties that keep springing up in Alberta, at first glance it would seem this is targeted towards the NDP (who have said they won't work with the ALP) and the Alberta Party (who are only a party in the loosest sense of the word). And, I guess the Greens, who were desanctioned and don't even exist anymore.

But I'm sure the Liberals recognize that formal co-operation on the left isn't going to happen any time soon. After all, you don't negotiate mergers through ads in the Calgary Herald. Rather, this would appear to be targeted towards disillusioned and disenchanted voters looking for a home. Portray Swann as a leader who does politics differently, who puts partisanship aside, who listens.

Is this the start of a larger campaign, or just a one-off to put the "cooperation resolution" behind them? Hard to say, but I think the point has been made and I wouldn't spend much more time or money on this one.

Ladies and Gentlemen, your Next Governor General

University of Waterloo President David Johnston

Lindsay Lohan goes to jail. Another example of state coercion?

OK, OK. The headline has little to do with this post, but I figured no one would read it if they deduced this was the third straight post on the friggin' Census. So indulge me for a moment and don't worry - next week, I'll get back to more important topics, like making fun of politicians in cowboy hats (which reminds me, please e-mail me any pictures you snap of politicians at the Stampede!).

Although Tony Clement was quiet in his initial decision to defile the Census, he has defended the move over Twitter, showing the eloquence and reason we've come to expect from the man who found a way to lose 4 elections within a 2 year span. Via Wherry, here's a selection of Tony's tweets:

TonyClement_MP @wicary Actually the Long Form is still around & will be going to more Cdns–minus the state coercion.
Harbles @TonyClement_MP However the statistical randomness is gone making the data skewed.
TonyClement_MP @Harbles Wrong. Statisticians can ensure validity w larger sample size
stephenfgordon @TonyClement_MP Wrong. Large samples can’t fix sample selection biases.
TonyClement_MP @stephenfgordon Which is why proper weighting will be used, as always the case
stephenfgordon @TonyClement_MP Where will the weights come from? Other voluntary surveys get their weights from the census.
c_9 @TonyClement_MP But weighting is done based on CENSUS DATA. Can’t weight the original data. Answers to these concerns somewhere?
TonyClement_MP @c_9 Folks! There is STILL a mandatory Census!
stephenfgordon @TonyClement_MP How can we reweight for education and income using short form data?

I know if I pointed out that the Minister of Industry shows a glaring lack of understanding of how the Census and statistics in general work, I'd come across sounding like one of those elitists he hates so much. So I'll refrain from pointing that out.

Besides, Clement made this decision despite the objections of the experts at Stats Canada so I don't think he really cares about the science behind it (Tony Clement motto: "Science is for elites"). This isn't too surprising from the guy who thinks he knows more about safe injection sites than Canadian doctors.

No, Tony makes it clear he's making this move to spare Canadians from "state coercion".

Now, to the best of my knowledge, there aren't any documented instances of Census workers using Jack Bauer style interrogation techniques, holding a knife to someone's eye and screaming "Who...do...you...work...for?". So I'll presume Tony is referring to the threat of a $500 fine or 3 months in jail for not completing the Census.

Let's put aside the fact that no one ever actually goes to jail for not filling out the Census and look at Clement's claim. His argument is that threatening to send someone to jail for breaking the law is coercion. This, from the "tough on crime" party.

Equally perplexing is that the short form will remain mandatory. So presumably, Canadians will still be "coerced" into completing their short form. I'd really like an explanation from Tony why short form coercion is acceptable when long form coercion isn't.

In fairness to Clement, he did also raise the issue of "intrusiveness" in another tweet. Personally, I don't think asking 1/5th of Canadians to fill out a form every 5 years is overly intrusive when the information remains 100% confidential. I'm not aware of a single instance where the anonymity of Census data has been compromised but, if this has occured, I invite the Minister of Industry to tweet about it.

Given the backlash we've seen to this move, Clement owes us more than a 140 character explanation.

Grab Your Pitchforks and Sampling Methodology Textbooks

Further to my post over the weekend, people are starting to take note of self-selection-gate. Yeah, I know it's just because Helena Guergis hasn't said anything lately but still, it's nice to see a mind numbingly boring issue that actually matters get some ink.

Below is a non-random sample of what people are saying:


"Abolishing this reliable source of useful information is little more than official vandalism."
-Montreal Gazette


"Before the federal government embarrasses itself further, it should turn back this loopy ruling."
-Edmonton Journal


"The government’s latest move to curtail the census is just another example of ideology trumping common sense."
-Toronto Star


"The outrageous decision by the federal government to eliminate the long form of the census questionnaire must be reversed immediately."
-Calgary Herald


"The findings from the census data influence everything from government spending priorities to political representation, both federally and provincially."
-Victoria Times Colonist


"Sending it out to more people doesn't solve the problem. The problem is that on a voluntary survey, people respond who feel like responding. The most vulnerable groups are the least likely to respond. So if you're interested in data about aboriginal people, if you're interested in data about recently arrived immigrants, if you're interested about the poor, the disadvantaged ... those are the kind of data that will be threatened."
-Ivan Fellegi, former Chief Statistician at Statistics Canada


"I'm just flabbergasted by the fact that they are taking the greatest source of information for the history of the country away from us."
-Gordon Watts, an amateur genealogist and co-chairman of the Canada Census Committee


"As a practising economist, the census is the single most important piece of information we get. It's absolutely crucial from a public policy point of view."
-Craig Alexander, chief economist at Toronto-Dominion Bank and president of the Canadian Association of Business Economists


"If response rates vary with the income and education levels, then you won't have a random sample of income and education levels. There is a rather large amount of evidence in the sampling design literature documenting the fact that people with lower levels of education and income have lower response rates, and so these groups will be systematically under-sampled."
-Stephen Gordon, professor of economics at l'Université Laval


"The long form is the only national source of information on aboriginal educational achievement. Without the census long form there will be no information about whether aboriginal education results are improving and no data with which objectively to assess policy alternatives."
-Social scientist Michael Mendelson


"As the organization that represents Canada's academic research community, we are deeply concerned about the disastrous consequences this will have for the scientific understanding of Canadian society, and for the ability to make informed decisions about social and economic policies."
-Canadian Association of University Teachers executive director James Turk


"Without robust Census data, it is difficult for local governments, health districts and other community service providers to respond effectively to shifting patterns of need or introduce changes – including cuts – that do the least harm or provide the greatest value for money. Indeed, it is the local level that is most hampered by this federal decision. The issue raised by cutting the Census long-form questionnaire is not just about having good information; it’s about having relevant tools for democracy."
-Armine Yalnizyan, Senior Economist Canadian Centre for Policy Alternatives


"Municipalities use the census like a GPS to navigate on-the-ground changes in our communities – to see where we need better bus service, to build affordable housing, or set up support programs for new Canadians. There’s a real concern that these changes are going to make it harder for us to meet the needs of Canadians – we need to know the federal government isn’t going to let that happen."
-Brock Carlton, CEO Federation of Canadian Municipalities


"We're not happy. Nobody on either board is happy." -Paul Jacobson, board member of Toronto and Canadian Associations of Business Economists


Hat tips...and more on this - Wherry, Selley, Tribe

In those days Caesar Augustus issued a decree that a census should be taken of the entire Roman world...among those who felt like it

I know this isn't a big issue. I don't expect facebook groups and protests in the streets over it. But, as someone who worked on the Census in the past, this really grates me:

Former StatsCan head slams census decision by Tories

OTTAWA - The former chief statistician of Canada says he would have quit his job if the Conservative government had tried to axe the long census form, as they're doing now.

Ivan Fellegi, a veteran public servant who spent 51 years at Statistics Canada before retiring in 2008, says he's alarmed by the decision to replace the long census form with a voluntary survey next year.

He joins a chorus of groups, including the Federation of Canadian Municipalities, the Canadian Council on Social Development and others who rely on the detailed data to make major public policy decisions.

"It would have been a heck of a lot better if this long-form census was cancelled because at least we would have saved $100 million — that would have had a rationale," Fellegi said in an interview with The Canadian Press.

"To come out with something (voluntary) that has uncertain quality, and certainly for some groups it will be unpublishable quality, is not something that I can understand."

The mandatory short census, with its basic questions on the ages and sexes of people in a household remains unchanged. But one-third of Canadians, up from one-fifth, will now receive a voluntary "national household survey" with detailed questions on ethnicity, education and income similar to those on the long census.

The cost of the change could reach $30 million, says Statistics Canada: $5 million for the additional mailout, and $25 million in case there is a major problem in getting people to respond.

The problem is that once you make the form voluntary, the sample becomes self-selecting, losing its randomness and, by consequence, its representativeness. The bottom line is that certain types of people will be more likely to fill out their form, skewing the results.

You'll get far more accurate data from 1/5 of all households than you will from the 1/3 of all households that bother to complete their form. It's the same reason why you get better data from a scientific telephone poll of 1,000 Canadians, than you do from the 10,000 people who vote on the Globe and Mail's web poll.

Census data is used for everything from municipal governments deciding where to build schools, to entrepreneurs deciding where to open up new restaurants. Making the long form voluntary will compromise the data used to make thousands of public and private sector decisions.

Yeah, I know data collection isn't sexy. But it's important. This decision needs to be reviewed.