SP2016 forecasting post-mortem

Before this week’s Scottish Parliament I said I’d produce a forecast post-mortem, no matter the result.

The tl;dr version of the post-mortem? Uh, not good.

The most obvious point at which to start that post-mortem is the seat forecast. The table below shows the final forecast seat tally from Thursday morning. The total seat error was 17 seats, more than half of which came from under-estimating the Conservative seat tally.

Party Forecast Range Result
SNP 69 (63, 73) 63
Labour 26 (21, 31) 26
Conservative 22 (18, 27) 31
Liberal Democrat 5 (2, 8) 5
Green 7 (3, 10) 6
UKIP 1 (0, 2) 0
Others 0 (0, 0) 0


The final seat tallies were within the 90% forecast interval for all parties save the Scottish Conservatives. The probability of the SNP winning fewer than 65 seats had increased in the final forecast on Thursday morning to eight percent. This was substantially higher than the figure used in a Tuesday morning press release (just 1%). It was also something which I overlooked on Thursday morning.

Error in predicting the list vote

The seat forecasts above depend on the vote forecasts. Under the additional member system modelling the list vote matters more for seat forecasts than modelling the constituency vote. Table 2 therefore shows the forecast list vote for each party, alongside the polling trend on the last day,1 and the actual results.

The forecast list vote share is based on the tendency of polls to exaggerate change since the last election. The general election of last year provided a clamorous example of this tendency: the polls drastically exaggerated the improvement in Labour’s vote share.

Party Polls Forecast Result
SNP 42.4 42.7 41.7
(39.5, 45.2) (39.2, 46.2)
Labour 18.9 20.5 19.1
(16, 22.3) (17.3, 23.7)
Conservative 19.8 18.1 22.9
(16.7, 23.1) (14.9, 21.3)
Liberal Democrat 6.1 5.9 5.2
(4.8, 7.6) (4.2, 7.7)
Green 8.5 7.6 6.6
(6.7, 10.4) (5.6, 9.7)
UKIP 3.1 2.6 2.0
(2.3, 4.1) (1.5, 3.9)
Others 1.4 2.6 2.5
(0.9, 1.9) (1.6, 3.7)


Unfortunately, the effect of applying this forecast model was to discount the improvement in the vote share of the Scottish Conservatives, and the decline of Scottish Labour. Although the forecast was more accurate than the final poll trend in predicting the vote share of the Liberal Democrats, Greens, and UKIP, this is limited consolation. The mean absolute error of the predictions for the top seven parties was larger than the mean absolute error of predictions based purely on the polling trend (1.36 versus 1.29).

The forecast model didn’t just affect the mean prediction: it also affected the width of the confidence interval. The 90% forecast intervals are all wider than the 90% intervals which result from the poll-pooling model. Unfortunately, these 90% forecast intervals failed to encompass the vote share of the Scottish Conservatives.

Error in predicting the distribution of the constituency vote

The forecast model has two principal parts. One part deals with forecasting national vote share — which, as I’ve just shown, didn’t exactly work as planned.

A second part deals with forecasting vote share in individual constituencies. Although I dislike the emphasis on constituency contests, they do provide a useful test of the model’s ability to get the distribution of parties’ vote share right.

A useful benchmark is provided by uniform national swing. I’ve therefore taken the parties’ 2011 performance, and added on the change in their share of the constituency vote between 2011 and 2016.

In doing this, I’m giving UNS sight of the 2016 result. In order to make it a fair contest, I’ll therefore work out on average how much my forecast was short of each party’s share of the constituency vote, and add this on to each constituency forecast.

When I do this, I find that the mean absolute error for UNS was 4.32, whereas the mean absolute error for my forecast model was 4.79. In other words, the extra effort that went in to modelling constituency variation didn’t produce a more accurate picture of what was happening in each constituency than a simple exercise in addition and subtraction.


Not all of the errors described above were within my power to change.  The two mistakes which stand out most clearly concern the over-confidence of the model, and the distribution of parties’ vote shares.

I think the model was over-confident because the reversion-to-last-election model is too simple. It has some nice properties: you can, for example, apply it to all parties symmetrically, so that vote-shares stay within 100 percent. It remains, however, a gussied-up regression model, and is prone to all the problems of over-fitting that that presents.
I’m not sure whether leave-one-out cross-validation would generate much wider forecast intervals, but it’s definitely something I’ll be looking at.

I think the model did a poor job at predicting the geographic distribution of parties’ votes because I just didn’t have the time to add all of the necessary constituency-level predictors. In the 2015 forecasting model, we were able to do a road-test on 2010 BES data, identify useful predictors at constituency level, and use those predictors in a complex model drawing on a relatively large number of constituency-level observations. That wasn’t possible here. I didn’t have any data from 2011, and so I opted to start with a few obvious predictors (GE2015 and SP2011 vote share) and expand as I went.  In the end, the lengthy process of variable testing that I had envisaged just didn’t happen — and with limited media buy-in, I wasn’t inclined to spend much more time on an election that seemed to interest very few.

To use a guid Scots expression, I’m fairly scunnered with election forecasting. It’s doubly fortunate that I’d already sworn off election forecasting (though I might be doing some on-the-night analysis), and that the next major set of elections is only in 2020. Until then, I’ll stick to analysing events that have already happened…

  1. This trend results from a model which pools polls. This model is based on work by Simon Jackman. See his article “Pooling the polls over an election campaign” for an introduction to the model. The pooling model used here differs from Jackman’s in that I used an additive log-ratio transform to ensure that the sum of parties’ vote shares remained 100%; and that I dispensed with house effects. The additive log-ratio transform is necessary to keep the estimates logically possible; the house effects were ditched because (particularly at the start of the year) there were relatively few polls and large gaps between polls, which made it very difficult to estimate house effects.

NSS results for politics, 2015

In past years, I’ve blogged on the results of the National Student Satisfaction Survey, and in particular the results for politics departments.

Here’s a plot of overall satisfaction for all 84 first-degree awarding institutions in the field of politics, together with associated confidence intervals.


Congratulations to Surrey, who have had to fight against the closure of their department this year.

Four things to know about constituency party endorsements

Recently, the nominations of constituency Labour parties have attracted a great deal of attention. That’s probably because Jeremy Corbyn was until recently in second place in the tally of constituency nominations, and "radical firebrand could become Labour party leader" is a good headline.

I thought it was a good idea to repeat a previous exercise I had carried out with MPs’ nominations to see the effect that CLP nominations have on Labour party members’ voting behaviour. Here are four things you should know about constituency party nominations in the 2010 Labour leadership election.

1. Constituency nominations generally followed MPs’ nominations

This table shows MP nominations (down the rows) against CLP nominations (across the columns). I’ve also added the “conversion rate” in the last column.

CLP nominates…
Abbott Balls Burnham Miliband, David Miliband, Ed Conversion (%)
MP nominates… Abbott 3 0 0 3 1 0.43
Balls 0 12 2 3 5 0.55
Burnham 0 0 13 3 1 0.76
Miliband, David 1 0 3 61 12 0.79
Miliband, Ed 1 1 3 9 42 0.75

On average, just over seven in ten CLPs with a sitting Labour MP followed the nomination of that MP.

2. CLPs without sitting Labour MPs were more likely to nominate left-wing candidates

This graph shows the number of nominations picked up by different candidates according to whether or not the CLP had a sitting Labour MP.


The graph shows that Abbott and Ed Miliband did better (both in terms of counts and in terms of proportions) amongst CLPs with no sitting Labour member. Conversely, Ed Balls’ campaign looks as though it’s the ultimate insiders’ campaign.

3. Many local members did not follow their CLPs’ nomination…

Bluntly: lots of MPs and CLPs nominated candidates who had no chance of winning. Again, this table shows CLP nominations down the rows against the candidate that received most first preferences (across the columns). Once again, I’ve put the conversion ratio.

Most first-preferences go to…
Abbott Balls Burnham Miliband, David Miliband, Ed Conversion (%)
CLP nominates… Abbott 0 0 0 18 1 0.00
Balls 0 2 0 13 2 0.12
Burnham 0 0 7 34 2 0.16
Miliband, David 0 0 0 161 3 0.98
Miliband, Ed 0 0 0 116 34 0.23

4. … but nomination still had an effect

Of course, nomination by a CLP might not cause you to receive the most first preference votes, but it may still help you.

This figure shows the average share of first preference votes according to whether or not the candidate was nominated by the CLP.


What effect in 2015?

These figures should suggest that we take information from CLP nominations with a pinch of salt. When all of the CLP nominations are available, they are unlikely to look very different from MPs’ nominations. Those nominations will, in some cases, be ineffective. It’s tempting to interpret Jeremy Corbyn as the Diane Abbott of the 2015 leadership election. I think that would be wrong. But I think it is reasonable to suggest that Corbyn’s eventual share of the vote will be lower than his current share of constituency nominations.

Which are the best performing government departments (2014/5 update)

In 2013 and 2014 I blogged about the best-performing government departments, judged by the traffic light ratings of the Major Projects Authority.

This year’s report is now out. Here’s a table of the average score by department. I’ve collapsed the distinction between capital and non-capital projects in the Department of Health.


Dept 2012 2013 2014 Change
HMT 2.5 3.0 5.0 2.0
NS& I 2.0 5.0 5.0 0.0
DfID 4.0 3.0 4.0 1.0
ONS 2.8 3.4 4.0 0.6
HMRC 3.7 3.8 4.0 0.3
CO 3.2 2.9 3.6 0.7
FCO 3.7 3.4 3.5 0.1
DfT 3.5 3.3 3.4 0.2
MoJ 3.5 3.3 3.3 -0.1
HO 3.4 3.2 3.3 0.1
DECC 5.0 3.5 3.2 -0.3
DCMS 2.8 3.2 3.2 0.0
MOD 3.5 3.2 3.2 0.0
BIS 3.5 3.5 3.1 -0.3
DEFRA 3.5 3.0 3.0 0.0
DfE 3.5 3.5 3.0 -0.5
DCLG 3.3 3.7 3.0 -0.7
DoH 3.7 3.0 3.0 0.0
DWP 3.3 3.4 2.9 -0.5

Congratulations to big improvers HMT and DfID; commiserations to laggards DCLG and DWP.

P.S. At the risk of seeming churlish, the MPA release is kind of annoying. Instead of giving variables along the columns, and projects along the rows, the data is transposed. It’s either a childish plot to prevent people from sorting and filtering, or… Well, if it’s not for that reason I’m at a loss to know what purpose it might serve.

Is Syriza defending pensioners because they turn out to vote?

The Greek government has been extremely reluctant to countenance changes to the pension system in its negotiations with its creditors.

When Ferdi Giugliano noted that defence of pensioners seemed to be politically transversal, it was quickly pointed out to him that there was an obvious reason why governments defend pensions: older people vote, and vote often.

This obvious answer seems wrong to be, at least for Greece and at least for the last election for which I have data.

Here’s a graph showing the smoothed rate of turnout as a function of age, in the May 2012 election.


As you can see, turnout peaks at age 45, before declining. This is a very different picture to most other countries.

The answer, in this case, is most likely a generation effect. After the rule of the colonels, the first democratic elections in Greece were in 1974. Assuming that the voting age has stayed the same since, the first generation to vote would have been born in 1956, and would be 56 at the time of the May 2012 election. Anyone older would have come of age, politically speaking, during a non-democratic regime. We know that voting is habit forming, and so this 55+ cohort would be less likely to vote than the 45-55 cohort.

I am, of course, happy to be proved wrong by those with data on the 2015 election — or indeed those who have a better handle on the CSES data I used (see below).

Technical: I used CSES data to create this graph. You can see the code I used to import the graph at github. Most of the lines of code deal with importing the CSES data, which — inexplicably — is released as a fixed-width file.

Strategy-proofing the Labour nomination process isn’t possible

tl;dr version: you can’t stop people playing silly buggers when nominating candidates

The process of nominating candidates for the leadership of the Labour party seems to have left many people disgruntled — over and above the normal or background level of disgruntlement in the party.

A key source of dissatisfaction is the way in which MPs have used their nominations. In particular, some people are upset that MPs have nominated candidates who are not their most-preferred candidate for the leadership of the party. This is a common criticism of those MPs who have nominated Jeremy Corbyn (who has disavowed the practice).

This kind of voting isn’t necessarily insincere, or strategic, or evidence of bad faith. MPs have preferences about the type of leadership contest as well as the outcome of that contest. MPs may wish to see a contest with at least one female candidate, or at least one ethnic minority candidate. Sometimes these process preferences can trump preferences regarding the outcome. This certainly happened in 2010, when Diane Abbott was nominated by some Miliband (David) supporting MPs precisely so that the leadership contest would not be fought by four white men.

But strategic voting is certainly present. One trivial example from the deputy leadership ballot demonstrates this: Rushanara Ali’s most preferred candidate for deputy leader is Rushanara Ali; but given that she had too few nominations, she strategically nominated Ben Bradshaw.

I find the strategy element of nominations in the leadership contest harder to understand. The most intuitive form of strategy comes in the form of a quid pro quo — if your supporters nominate me, I’ll ask my supporters to give their second preferences to you later.

Yet most work on strategic nomination doesn’t look at quid pro quos, but at what we might call spoiler effects, often involving one side of the spectrum strategically nominating from the other end of the spectrum. In his book Securities against Misrule, Jon Elster gives a useful (but rather scurrilous) example from the French presidential election of 2002:

“It is possible – but unproven – that the 2002 election of Jacques Chirac as President was due to the strategic withdrawal of one candidate (Charles Pasqua) and by strategic action to prevent the withdrawal of another (Christiane Taubira). The election used the “two-past-the-post” system… given the near-certainty that the Far Right candidate (Jean Marie Le Pen) would obtain 15-20 percent of the vote, the system gave the major candidates on the Right (Chirac) and on the Left (Lionel Jospin) an incentive to encourage the proliferation of candidates in the other camp (to reduce the oponent’s chances of beating Le Pen) and reduce the number of candidates in his own (to increase his chances of beating him. It is possible that Chirac prevented Jospin from getting to the second round… by exploiting this possibility [since it was rumoured that Chirac’s party helped to finance Taubira’s campaign]”

Two-round systems are more vulnerable to strategic nomination than the Alternative Vote system used in the leadership election proper. If Corbyn is on the ballot, he’s likely to be eliminated in the first two rounds of voting, and so may not be a very durable spoiler. In any event, this kind of strategic nomination sits uneasily with quid pro quo exchanges. The ideological distances are too far to span.

More generally, it’s unsurprising that the system Labour uses should give rise to strategic voting. It’s a bit of a stretch, but we can think of the system of nominations as an electoral system in itself, similar to the single non-transferable vote system used in Puerto Rico and previously in Japan. Like SNTV, each voter can express one preference (their nomination), and there are multiple “winners” (who go on to become leadership candidates). Unlike SNTV, the number of winners is not fixed in advance: any candidate who gets over the quota can become a candidate.

Just as in SNTV, strategic considerations are widespread. Factions within the party have to ensure that their votes are not spread too thinly. In extremis, a faction within the party might find it has nominated three candidates, each of whom has the support of 20 MPs, all of whom fail to become candidates (similar to the position of a centre-right [Blairite?] grouping in the deputy leadership election before Rushanara Ali’s decision to drop out).

If strategic voting, or systems which encourage it, are A Bad Thing, then surely the answer is to find some electoral system for selecting multiple winners which is immune to strategic considerations of this kind?

Unfortunately, it’s unlikely that such a system exists. The Gibbard-Satterthwaite and Duggan-Schwartz theorems show that single and multiple-winner electoral systems respectively are always susceptible to voters misrepresenting their preferences.

As such, there is no “institutional” fix for removing strategic voting in nomination. The only solution is to remit the problem to the broader party electorate, where tactical manoeuvring may be less visible, harder to organize, and in any case less likely to provide lobby-fodder. This can be done by drastically lowering thresholds for nomination. Comparatively, the thresholds for candidates for the leadership of the Conservative Party and the Liberal Democrats are far lower — and in the case of the Liberal Democrats, the thresholds are now so low that strategic considerations are entirely absent. With a threshold of 0.8 MPs, any leadership candidate need only convince themselves that they are the best person for the job.

Of course, lower thresholds have their own problems. Modulo the distribution of opinion within a party, they’re likely to lead to more candidates with extreme positions. But with nomination systems, as with most things, there’s no such thing as a free lunch.

Does the #Brexit referendum franchise matter?

The government has announced that EU citizens will not be eligible to vote in the referendum on Britain’s membership of the EU. However, because the electoral roll will be the same as that used in the general election, citizens of certain Commonwealth countries will be able to vote.

The question about who ought to be able to vote in a referendum like this is a normative one. From what I can tell, normative theorists, often building on the "all affected interests" principle, tend to support broader franchises, and would typically support extending the franchise to EU citizens. Certainly, I’m not aware of any principled argument why Maltese citizens should be able to vote but Greeks shouldn’t.

There is, however, a separate empirical question: would extension of the franchise make any difference to the outcome?

According to wave 4 of the British Election Study (fieldwork: March 2015), 50.2 percent of British citizens supported Britain staying in the EU.

In that same survey, 81.5 percent of EU citizens resident in the UK supported Britain staying in the EU.

According to the latest ONS Population by Country of Birth and Nationality Report, there are 57678 thousand resident Britons, and 2507 thousand resident EU citizens, or roughly 23 British citizens for every EU citizen.

If we assume that the two groups vote at similar rates,1 the difference between a Britons-only referendum and a Britons-plus-EU-citizens referendum is the difference between:

  • 50.2%
  • (57678)/(57678+2507) * 50.2 + (2507)/(57678+2507) * 81.5, or 51.5%

Therefore the votes of EU citizens, although they will make Brexit less likely, will only matter in a very close referendum. The choice of electoral roll is thus in a very real sense a matter of principle rather than political practice.

  1. This assumption is likely false. On the one hand, a slightly greater proportion of EU citizens resident will be eligible to vote in virtue of being over 18. On the other hand, turnout amongst the eligible is likely to be lower amongst EU citizens. EU citizens were eligible to vote in the Scottish independence referendum — but whilst 90.5% of British respondents said that it was "very likely" that they would vote, only 63% of EU respondents said the same.

Did Labour lose because it was too left-wing?

A number of commentators have suggested that Labour lost the election because it was too left-wing — or, what is equivalent, enough commentators have suggested this as to make its denial necessary for some.

Usually, this is accompanied by an implicit invocation of median voter theorem (“elections are won from the centre”).

There’s just one problem with these arguments: in the early part of the year, Labour was perceived to be closer to the median voter than was the Conservative party.

Wave 3 of the British Election Study asks respondents to position themselves on a left-right scale which runs from 1 to 11.

On this scale, the (weighted) median voter is at 6, equidistant from either end of the spectrum. For what it’s worth, the (weighted) mean is very slightly to the right of this position, at 6.21.

When asked to position the parties, the (weighted) mean position for Labour is 4.18, or 1.82 points to the left of the median voter.

The (weighted) mean position for the Conservative party is 8.67, or 2.67 points to the right of the median voter.

This does not mean that Labour’s positioning was not a contributory factor in the party’s defeat. Perhaps if Labour had been closer still to the median voter, it would have won more votes. But spatial politics and the median voter theorem alone can’t explain the party’s defeat. Other factors – like having a leader who is rated as competent — matter, and probably matter more. After all, if closeness to the median voter was the exclusive determinant of parties’ vote shares, we’d be basking in the bright new dawn of a Liberal Democrat government (weighted mean left-right position: 5.77).

P.S. This conclusion (which is similar to the conclusion that Ed Fieldhouse arrives at, and which would have pre-empted me writing this blogpost had I seen it earlier) seems robust to different survey weights. It’s possible that there are different scaling issues to do with respondents in different party systems, but these would have to be very severe to affect the conclusion. It’s also possible that this finding will change when the campaign wave of the BES is available.

A constituency poll by any other name?

ICM has just released a constituency poll of Sheffield Hallam. Before a spiral-of-silence adjustment, Nick Clegg is on 40%, and his Labour challenger Oliver Coppard is on 36%. These figures are based on responses from 336 individuals, and so have a margin of error of 5.3%.

This compares with a recent Lord Ashcroft poll, which showed Clegg on 36%, and Coppard on 38%, but without naming the candidates. These figures were based on responses from 733 individuals.

Given these numbers and their associated margin of error, it’s not clear to me that we can conclude much about the relative merits of naming candidates versus not naming candidates. Suppose Clegg wins by 38% to Coppard’s 37% — we would be none the wiser.

Of course, there are other reasons for thinking that constituency polls which name candidates should be more accurate than constituency polls which do not name candidates.

The argument for naming is simple: voters in ballot booths are given a list of names; polls estimate what happens in the ballot booth; therefore polls should use the same prompts found in the ballot booth.

The argument against naming is more complicated, and may trade (entirely on in part) on the survey mode.

Suppose you receive a phone call from a pollster. Being a frequent consumer of polls, you oblige. You are given a list of names, but the list is quite long. You can’t quite remember the name of the Labour candidate, but you’ve heard of that Nick Clegg. Unwilling to look foolish in the eyes of the interviewer, you say, “Clegg, he’s the one”.

In other words, naming candidates in phone surveys biases responses to the most well-known candidate.

This name recognition effect probably doesn’t make sense to people with strong party identification. But there are fewer and fewer such people around, and there’s a long history of public opinion research which suggests that public opinion doesn’t really exist outside of a particular measurement context.

Ideally, all parties which conducted named-candidate polls (I’m looking at you LibDems!) will release these polls after the election so that they can be compared to the nearest (in time) Ashcroft poll. Only then will we be able to get some idea of the relative accuracy of named versus unnamed candidate polls across a range of constituencies.

Comments welcome below!

The Rentoul Questions

John Rentoul has put together a very helpful flow-chart intended to help us navigate the thickets of post-electoral coalition formation.

In it, he asks a number of questions about different configurations of parties. I thought I’d try and indicate what the probabilities of these different configurations are, according to the forecasts from electionforecast.co.uk.

Q1: Have the Conservatives plus DUP and UKIP won 323 seats or more?

Very unlikely. The probability of these parties winning 323 seats or more is just 3%.

Q2: Have the Conservatives plus LibDems, DUP and UKIP won 323 seats or more?

Moderately unlikely. The probability of these parties winning 323 seats or more is almost one-third (32%).

Q3: Are the four parties on 321 or 322?

Very unlikely on its own (5%), although obviously this number can be added to the above number.

Q4: Have Labour plus the LibDems and SDLP won 323 seats or more?

Unlikely. The probability of these parties winning 323 seats or more is a little over 12%.

If you follow the most likely outcome at each branching point, these probabilities imply a minority Labour government, with a second election described as “possible”.