Creative destruction: The fragmentation and consolidation of Canadian political parties (2/2)

Note — After sharing an earlier version of this post with my friend JJ McCullough, he noted that many historical sources on Canadian election data differ considerably: you need to be careful about what sources you pull from.  He has produced a wonderful website on Canada (The Canada Guide), which also covers historical elections.  You should check it out!

How many parties are there?

The previous post got me thinking: it's clear that the number of political parties has increased over the last century but how can we quantify this?  Counting the number of different candidates running is unsatisfactory since that includes all sorts of irrelevant parties (e.g., the Communist Party of Canada).  Instead, we can calculate a quick metric known as the "effective number of parties".

To calculate this, sum the squares of the support that each party achieved and then invert the total.  In economics, this method is a quick-and-dirty (although very dated) way to look at the competition in a given market: the more “effective firms” (or parties), the more competition.  In short, this metric gives an indication for how many political parties people have either voted for or are taking seats in the House of Commons.  

The chart below shows two things: the top panel shows the effective number of parties that Canadians voted for, while the bottom panel show how many political parties took seats in the House of Commons. I've calculated this for every election since 1987 (using data from the Library of Parliament):

This plot shows the effective number of parties for each election. The top panel shows the effective number of parties based off the votes, while the bottom panel shows the effective number of parties that show up in the House of Commons. The average for each era is in the oval.

Eyeballing the data, there appear to be three eras in Canadian politics (also discussed in a previous post).  I've coloured these in black, blue, and orange.  From 1867 until 1917, Canadians voted for ~2 political parties and ~2 political parties showed up in the House of Commons (2.1 and 1.9 respectively, to be precise). 

But from 1920 until 1990 this changed: Canadians voted for roughly 3 political parties and only 2.4 showed up in Ottawa.  After 1990, this became more extreme: Canadians voted for 3.7 parties but only 2.8 showed up at Parliament.  What's going on?

For every incremental effective political party that Canadians voted for, only ~0.5 effective parties show up in Ottawa.  I've shown this in the scatter plot below.

The data shows that, over time, Canadians have voted for more political parties but, although there are now more political parties entering the House of Commons, this is not fully reflected in Parliament's composition.  The reason for this is the convexity inherent in our first past the post system: smaller parties are typically under-represented, while larger ones are over-represented. 

Each dot in this plot represents an election (they are colour-coded by dated). This plot shows that as Canadians vote for more effective parties, the effective number of parties in Parliament increases, but at a slower rate.

A desirable property of FPTP?

I don't think the above trend (alone) is sufficient to warrant worrying about under-representation — in fact, it shows a desirable (and often forgotten property) of the FPTP system: the incentive to merge political factions into larger parties.

Party mergers and breakups are quite common in many political systems. However, in First Past the Post system, the convexity of the system provides an in-built incentive for similar political factions to resolve their differences and present a united front to the electorate. Two similar parties that each receive 15% of the vote will typically have fewer seats than a single party that achieves 30%.  This was the major incentive for the Canadian Alliance and Progressive Conservatives to overcome their differences and form a united Conservative Party in 2003.

This dynamic is often ignored when discussing whether we should adopt a Proportional Representation (PR) system: under PR systems (like in Israel or the Netherlands), factions have little incentive to overcome their differences before an election. Instead, they formally negotiate their differences after an election in order to form government (sometimes this takes a significant amount of time).  Had Canada adopted a PR system in the early 2000's, I doubt the Progressive Conservatives and Canadian Alliance would have merged.  Further, if we adopted PR today, there is no reason to think the Conservatives (or any of the other parties) would necessarily remain united. 

This is not to say that PR is inferior or superior to FPTP; however, the two systems differ in their incentives to create new political parties. PR allows for the existence of small parties, while FPTP incentivizes the creation of “broad churches” from many political factions.

The dynamic process of creating new political parties, destroying old ones, and merging similar ones is akin to creative destruction.  Ignoring party formation in analysis of changes to the electoral system is a bit like analyzing a market under the assumption that there will be no more technological improvements or that the current major players will always be on top.  It might give some insight into what could happen in the very short run but it won't tell you anything about how the new system will behave.

Creative destruction: The fragmentation and consolidation of Canadian political parties (1/2)

Note — After sharing an earlier version of this post with my friend JJ McCullough, he noted that many historical sources on Canadian election data differ considerably: you need to be careful about what sources you pull from.  He has produced a wonderful website on Canada (The Canada Guide), which also covers historical elections.  You should check it out!

Mergers and acquisitions

The creation of new parties and the mergers of old ones is a perfectly normal part of the Parliamentary system and one that is sometimes overlooked:  in discussions on electoral reform, it's tempting  to focus on which party today would benefit from a proposed change (after a quick Google, here is an example).  Although these articles can be fun speculation, there is a problem with this sort of analysis: it assumes today's political parties will remain static even after some fundamental change to the voting system.  

Over the course of two short posts, I want to show you a longer time series of Canadian elections and a couple quick cuts of the data to show how parties rise and fall.  Their emergence (and destruction) is a crucial part of the "first past the post" (FPTP) system — in fact, a crucial part of any democratic political system!  Ideological (or other) movements must iron out their difference and coalesce into a well-organized party so as to present the country with a coherent and unified option for Government.  These "broad churches" are themselves smaller deliberative bodies, where different factions must compromise.

When factions cannot find reasonable compromise within an existing party, they can start their own.  This is what happened when Lucien Bouchard left the PCs to form the Bloc Quebecois or when Preston Manning (and others) founded Reform.  The converse is also true: Stephen Harper and Peter MacKay merged the Canadian Alliance and the Progressive Conservatives, which created the Conservative Party in 2003.

Even over the last decade there has been plenty of speculation on whether the Liberals and NDP should merge (here is but one example).  

Federal votes and election outcomes

I've pulled the historical Federal election outcomes since Confederation from a variety of sources (although mainly from the Library of Parliament).  Below are two plots: the top one shows the distribution of seats from every election since 1867, while the lower one shows the popular vote percentages.

I have ordered the parties (roughly) so that the Conservative / Right-wing parties are near the top of the chart and the Liberals are at the bottom.   Since these are the only two parties that have held government, this should show the back-and-forth between them over time (in particular, whenever one crosses the white 50% line).

This first chart (above) shows the distribution of seats in the House of Commons after every Federal vote coloured by political party. It is easy to see the Liberal and Conservative majorities as they cross the white horizontal 50% line.

This second chart (above) shows the distribution of votes coloured by political party). There are few cases where any political party gets over 50% of the vote (these cases are mostly when there were only two major parties from 1867-1917)

A few quick reflections from these charts:

  • After the 19th century, majority governments have been common but majority popular votes are not: since the rise of >2 major political parties, there has only been 2 elections where a party received the majority of the popular vote

  • The distribution of seats is more volatile than the distribution votes: this shows the convexity that emerges from our voting system - small changes in the vote proportions lead to large changes in the seats (I have another post on this topic.)

  • There appear to be two major "fragmentation moments" in Canadian politics over the last 150 years:

    • ~1917 (onward): The rise of the Progressives, Social Credit and other parties (CCF, New Democracy)

    • ~1990: The fragmentation of the political right and the rise of separatism

  • Political fragmentations take years to resolve: after the break-up of the Right in the 1980s / 90s, the Liberals achieved four mandates (three majority and one minority) before a Conservative party took government

In a future post, I'll show some more detail on how the the “number” of political parties has increased over time.

The magic number: What’s with the fixation on 40% for a majority?

A common trope from the Canadian commentariat is that 40% national support should result in a majority government (see here from Nanos Research).  I’ve always been skeptical whenever a journalist trots out "rules" like this since they often ignore the complexities and subtleties of our system.

But 40% does seem roughly right: the last two majority governments received just above 39% of the vote. 

To test this, I've taken all the elections since 1945 to see the relationship between the popular vote received by each party and the proportion of seats they won in the House of Commons:

Each dot in this plot represents a single party's outcome in an election (e.g., in 2015, the Conservative Party won 32% of the national vote and got 29% of the seats in the HoC).  This plot shows that the relationship between the national vote and seats in the HoC is convex: smaller parties are under-represented, while larger ones are over-represented.  This attribute makes it easier to form majority governments since a party can win 50% of the seats with fewer than 50% of the votes. 

Looking at the data and fitting a quadratic line** shows that when a party gets 40% of the national vote, they should expect to get roughly 50% of the seats in the House of Commons.  The "40% for a majority" rule-of-thumb is pretty good! 

But there is another interesting pattern in the data: the relationship is quite clearly convex. That is, if you get below 30% of the national vote, you should expect to be "under-represented" in the House of Commons.  If you score over 30% of the national vote, then you should expect to get a disproportionate number of the seats in the House of Commons.  (This is shown on the chart as being above or below the 45 degree line - if you are on the 45 degree line, your national vote percent and representation in the HoC are the same)

This demonstrates that larger parties have an advantage in the FPTP system: two smaller parties that each get 20% of the national vote would expect to elect fewer MPs than a united party that gets 40% of the vote.  This provides a strong incentive for parties to unite and create broad coalitions (instead of smaller fringe parties).

Another implication is that relatively small differences in vote can result in large differences in the House of Commons.  Consider the current situation in Canada: the Liberals got just shy of 40% of the vote and received 54% of the seats in the House.  The opposition Conservatives got 32% of the vote and 29% of the seats.  An eight percentage point difference in vote results in a 25 percentage point gap in representation in the House of Commons. 

Put another way, changes in the House of Commons should be much more volatile than changes to the national vote.  This is very clear in the data, but that will be covered in a future post.

** An older version of this analysis on this site used a Loess regression.  I couldn't be bothered running the code again so just did this analysis in Excel using a polynomial regression.  The result is very similar and the conclusion doesn't change.

A week is a long time in politics: 14 years of Canadian polling data

This post’s title is a saying often attributed to Harold Wilson, the British Labour Prime Minister who served in the 1960’s and ‘70s.  Although the origins of the quote are unclear -- Wilson did not recall ever saying it -- the meaning is not: fortunes in politics are subject to rapid rises and falls.  This is, no doubt, very true in Canada.

In this first post, I want to show you a straightforward time series of Canadian political party support over the last 14 years.  To my knowledge, it is the longest single time series of Canadian political polls. 

This period coincides with the creation of a united Conservative party, the secular decline of Quebec separatism, the “coalition crisis” of 2008, the “Orange crush” collapse of the Liberal vote in 2011, and the subsequent Liberal majority in 2015.  It spans four Liberal leaders (Martin, Dion, Ignatieff, and Trudeau), three NDP leaders (Layton, Mulcair, and Singh), and two Conservatives (Harper and Scheer). 

With all that laid out, here are some quick high-level points that emerge from this national picture:

  • Changes in national political support have “fat tails”: party support can appear relatively stable for long periods of time before suddenly shifting.  To put this another way, political support is subject to “Black Swan events” where the magnitude of shifting support can dwarf any smaller variations previously observed.  A clear example of this is the rise of the NDP in the 2011 election: if you look at the data leading up to the 2011 election, it would be nigh impossible to predict the NDP surge, which is well above any historic highs they ever achieved.  In fact, it appears as if the Liberals and NDP quite literally swap 15 points of support in a single week!  Which leads to the second point…
  • Election campaigns matter: major shifts in the political environment coincide with election campaigns.  Whether it’s the Conservatives overtaking the Liberals (2006), the NDP and Liberals swapping (2011), or the Liberals overtaking both the Conservatives and NDP (2016), there are many examples where support for the parties changes radically from the start to end of a campaign.
  • Party leadership conventions move polls: Just eye-balling the chart shows that new leaders often get a bounce when they’re nominated.  Dion saw a ~5ppt bump, Ignatieff was probably closer to 10ppt, Mulcair got ~5ppt, and Trudeau saw a sustained ~10ppt step-up in support.  Although it might be too soon to tell, Scheer and Singh appear to have much smaller bumps in comparison.

With all that being said, anyone familiar with Canadian politics knows that the national support for a party is actually a rather meaningless number.  The actual outcome of the election is determined by the individual electoral districts (like the US electoral college).  For instance, the Bloc Quebecois has little support nationally yet they play a disproportionate role in our politics due to their regional concentration.  Similarly, the Green party is a national irrelevance because their support is so dispersed. 

In short, what’s probably more interesting is what this picture doesn’t show: our political system is highly regional and this can have profound effects on electoral outcomes.   As a continuing series of posts on this blog, I hope to lay out some of my thoughts on how our election system can be modeled.

How to read the chart:

  • Each party is represented by a colour: Liberals are red, Conservatives are blue, the NDP is orange, the Greens are green (surprise!), and the Bloc is purple
  • Each of the lightly coloured dots represents a single poll published by a polling agency
  • The coloured lines and bands represent a smoothed average of these polls over time (using a Loess regression.  I’ll probably explain how this works in a future post.  For now, I want to avoid too much maths)

Sources: Over the years, I have manually tracked publicly reported polls.  In recent years, I have started also relying on Wikipedia to make sure I have not missed any polls.  You can easily access the Wikipedia data here.