Seat Calculators, UNS and the 2010 General election

Summary
Yep - summary up front. This post has grown hugely 'cause I wanted to cover the details of what I'm summarising, which are:

- UNS performed slightly worse (for Tory seat share) than in 2005 and 2001 but better than 1997.

- Regional effects were minimal (about 2 seats difference to the totals)

- "Unwind" occurred to about 50% and was dented by a surprisingly high incumbency boost

- Demographic and ethnic differences between seats had a large effect

- UNS calculators beforehand all predicted heavily hung Parliament, no minority governments feasible, both Con/LD and Lab/LD coalitions possible

- Alternative calculators varied greatly but almost all had Tories over 300 but short of 326 - Tory minority or Con/LD coalitions as the realistic alternatives. Lab/LD total close but overpredicted LD at Lab expense.

- My spreadsheet had more inputs that had to be estimated (a weakness); with the suggested figures I gave just before the poll of 66% unwind, sd's of 4, 4.5, 4, it gave Con 312 (+/-6), Lab 235 (+/-6), LD 70 (+/-4). LD underperformance in hitting possible/probable targets and defending possible/probable holds and strong Labour defences (incumbency) are suggested.

- "Electoral geography" has improved the Tory vs Labour position - the gap between them at level polling has reduced from 89 seats to 54. Labour majority threshold has moved from 0.2% Con lead to 2.5% Labour lead, equal seats from 6.8% Con lead to 3.9% Con lead. The Tory majority threshold hasn't twitched, though, due to a low density of seats between the current point and 2% swing - 10 seats per point against the 20 seats/point reached to here. Accordingly, the hung parliament zone has widened from 10 points to 13 points.

- Without the Clegg surge, the Tories would have achieved more than 320 seats on the 7 point lead they got and the Lib Dems been heavily squeezed.

Review

So, how did the forecast models do? One additional headache has always been the reliability of the input data – if the polls over- or under-represent any party, there’s a problem even if the model is right. So we’ve got to look at what the outcome should have been with perfect knowledge.

Sleazy broken UNS on the slide?

The simplest model is to use additive UNS – add the national swing to every constituency and count which ones change hands.It’s invariably carried out without provisos by the media and has been subject to considerable questioning here on pb. Before this election, we were well aware that the errors could, for the first time since 1992, result in a wrong prediction for the overall result: in 1997, 2001 and 2005, the only error would have been in misjudging the size of the Labour majority. However, virtually every poll in the week or so before the election, put through UNS, resulted in a hugely hung Parliament, with the Tories around the 280 mark and Labour plus the Lib Dems together with a comfortable majority – if they so chose to ally themselves. Mitchell Stirling had an excellent series of posts comparing the UNS predictions to the “alternative” predictions. The final table was here.

Interestingly, the final averaged UNS prediction (based on the polls) had Labour spot on, but the Tories under by 25 seats and the Lib Dems over by a similar amount. Lib Dem kingmakers, who could go almost equally easily with the Tories or Labour.

The final averaged alternative predictions had the Tories over by 8, Labour well under at 40 below and the Lib Dems over by about 25 seats. Either a Tory minority Government, or Lib Dem/Tory Coalition.

Why the surprise, then?

So why were we on pb.com so surprised at the result? The UNS calculators were pointing at a far worse Tory result, the non-UNS ones were pretty close to the mark for the Tory seat total (and were correctly pointing to a hung parliament with potential of Tory minority or Con/LD coalition). We were really knocked off target because we expected both that the UNS calculators would fall short and that the polls would noticeably understate the Conservative lead. To be fair, the Golden Rule had worked for so long that it did appear to be exposing a suspected methodological flaw in British opinion polling. As it turned out, the best triumph of the night (to my mind) was the long-delayed vindication by Nick Sparrow of his method of estimating the "spiral of silence". In 1997, it was inconclusive, in 2001 and 2005 it didn't change the final scores - but this time around it did, and seems to have exposed a very real tendency of voters to be attracted to their last voting choice.

The above table glosses over the variability between the non-standard predictors, ranging from 292 to 363 Tory seats (but with 15 of 18 predicting Tories short and one of the other 3 a razor thin majority prediction of 0). Labour, on the other hand, were clustered in a far lower range, from 199-235 seats. The Lib Dems ranged from 57 seats to 114. To be fair to all methods, if you put in the wrong inputs, you cannot expect to get the right outputs, and the polls failed with respect to the Labour/Lib Dem vote split. In computer programming they call it "GIGO": Garbage In, Garbage Out.

What should the models have predicted?

Many people (including journalists) went straight to Anthony Wells’s excellent UKPollingReport site, which included a weighted poll average and UNS calculation. Now Wells highlighted that UNS is an estimate, not a rule of law, but the final estimate was Con 274, Lab 264, LD 81 with vote shares of Con 35, Lab 28, LD 27 (Con 7 point lead). Almost a dead heat (Conservatives 52 short), with Lib Dem kingmakers (Con/LD majority of 50; Lab/LD majority of 40) – a minority government would be definitely impractical on these figures.

With correct data in (GB shares of Con 36.9%, Lab 29.7%, LD 23.6%, giving Lab-Con swing of 5.1%, Lab-LD of 3.7%, LD-Con of 1.4%) UNS would predict that:

- The Tories would (on the Wells figures) pick up 66 Labour seats (Overtaking Labour in Edinburgh South, but losing to the Lib Dems), 7 Lib Dem seats and maybe the IKHH seat (up by 74 seats from 214).

- The Lib Dems should pick up 6 seats from Labour and lose 7 to the Tories (down one from 63).

- Labour should lose 66 seats to the Tories and 6 to the Lib Dems, down by 72 from 344.

So final scores of Con 288, Lab 272, LD 62.

Conservatives 38 short, Con/LD majority 50 or Lab/LD majority 18.

Lib Dem kingmakers again, with Labour not too far behind the Tories. Con minority government not really feasible. Errors of 19 seats underpredicted for the Conservatives, 14 overpredicted for Labour and 8 overpredicted for the Lib Dems.

Regional UNS wouldn’t do much better – Conservatives would have won 71 seats from Labour, and 5 from the Lib Dems, with the Lib Dems exactly making up their losses from Labour: ending with Con 290, Lab 268, LD 63.So the effects of the regional swing variations pretty much cancelled out - the source of error wasn't the non-uniformity regionally.

How close could that 7% lead have taken them to a majority?

While we’re on UNS and the “11% lead needed for a majority” question, Peter Kellner wrote a piece on the election recently. He pointed out that although Lib Dems were disappointed that the Clegg surge hadn’t fully materialised, their position was far better than it had been before the surge, and estimated that the House of Commons would have been (on 19% Lib Dem score) 321 Con, 264 Lab, 37 Lib Dems.Looking at the YouGov polls of 9^th-15^th April, the Lib Dems are on an average of 19.7%, with Con and Lab on 38.9 and 31.9 respectively – so it seems as though the surge that did occur damaged both parties relatively equally and a 7 point Con lead before the Clegg surge would have put the Tories right on the very edge of an effective majority. Arguably, the Clegg surge did end up putting the Lib Dems into Government, as it turned out.

"A rule of thumb, not a rule of law"

Overall, UNS didn’t do much better or worse than recently. I've had a glance back and it seems that antifrank was the first to put it in these words “It’s a rule of thumb, not a rule of law” in reply to my series of articles on UNS and how it can err (saving about 3,130 words from how I put it ...). It was worse than in 2005 or 2001, but better than 1997. The Labour “seat advantage” (their lead in seats on level pegging in the polls with Lib Dems unchanged) continued to drop. The 2001 peak was 144 seats. In 2005 it was 89 seats (so UNS claimed that a dead heat would see a narrow Labour majority, 89 seats ahead of the Tories). Today it’s 54 seats (so after seeing how things changed during the election, we can say that a dead heat would have been a hung Parliament with the Tories 54 seats behind Labour – not the 89 that UNS from 2005 would have predicted). The Tory damage since 1997 continues to unwind – in 97 they fell behind UNS by 41 seats and in 2001 by a further 15, being 56 seats behind UNS (measured from 1992) at the peak. In 2005 they pulled back by 13 seats and this time by a further 19-20, so pulling back about 32 seats of their disadvantage. Still behind the 1992 relative position, but with the distortion unwinding considerably.

Running it backwards - a 25 seat advantage

It’s also sometimes educational to run an election in reverse – what should the 2005 notional figures have been if created by UNS from the current position? Feed 33.2, 36.2, 22.6 into the (commendably swiftly) updated.additive UNS calculator on Anthony’s site: Con 239, Lab 330, LD 55. So zero swing from the 2005 position would have obtained quite a difference, illustrating how the electoral landscape shifted (again) during the election.

Alternative calculators

On the “what would the answer be with the right question” front, the only alternative calculator I can easily use is my own. This adds another level of challenge with the input parameters – it allows the adjustment of the level of “unwind” of the accumulated tactical voting/marginal boost that progressively damaged the Conservatives from 1992 to 2001, pushing them way below the levels that UNS would expect. Some partial unwind occurred in 2005, but much was still locked up in the electoral landscape.

One way to estimate it would be to try to measure the relative unpopularity (the straight poll figures give us popularity; unpopularity is governed by who the electorate would most like to deny). Through 2009, the second choice of Lib Dem voters was preferentially Conservative. This advantage declined as 2009 slid into 2010 and became about level by February. From then, Labour built up an advantage, although never to the degree of their advantage in 2005. I’d estimate then that the unwind would have been greater than 100% through most of 2009 (giving the Conservatives an advantage to the extent that the political landscape would have possibly looked biased towards them), declining to about 100% around February and lower as the election neared. Had Labour built up their advantage to an equivalent level of 2005, they’d have reduced the unwind to 0%. An even greater advantage would actually produce a negative unwind. As it was, it appears to have been about 50%. On the eve of election, I estimated 66%. I also stated that I reckoned there would be a higher than normal variability (giving higher standard deviations of swing) of 4, 4.5 and 4. These input variables with the correct poll shares gave:

Con 312 (+/-6), Lab 235 (+/-6), LD 70 (+/-4). So Tory minority or Con/LD coalition.

Errors

The Tory share was within the error margins (5 seats off), but the Labour and Lib Dem shares were out by 23 and 16 seats respectively (although the total of them was within a few seats). The Lib Dems were over-forecast with respect to Labour – the anticipated Lib Dem targeting/defensive boost never materialised. Breaking down further, they undershot specifically in England against the forecast – seats that should have turned yellow like the City of Durham, Hampstead & Kilburn, York Outer and Oxford East simply didn’t.

Losses like Newton Abbot, Oxford West and Abingdon, Camborne and Redruth, Richmond, Cornwall South East, Harrogate, Rochdale, and Winchester further damaged the Lib Dem picture. Wins like Redcar and Wells helped, but very few of the “Lib Dem possibles” (like Watford, Guildford, Meon Valley, Ealing Central & Acton, Derby North, Blaydon, Romsey, Birmingham Hall Green and Leicester South were held by or taken by the Lib Dems – they significantly underperformed on the probabilistic front (Lib Dems – what happened? Failure of targeting? Third party squeeze?).

Labour defended well, overperforming slightly in Wales against their poll share – they should have lost three or four more (holding Ynys Mon against the odds and not losing any of Vale of Clwyd, Newport West, Gower, Bridgend, Swansea West, Clwyd South or Delyn. Most of these would have needed above-average swings (in some cases, well above-average), but the high level of variance in swing didn’t materialise in these seats where it would have been very bad for Labour. In England as a whole, they defended very well, getting a noticeable incumbency boost (especially in places like Broxstowe, where the odds of Nick holding on were down around 7% and he very nearly pulled it off, and Birmingham Edgbaston, where the model gave Gisela Stuart only an 11% chance of holding on) as well as an ethnic vote which swung their way. It looks as though there were class based swings as well, with C2s swinging far more to the Tories than ABs (on the whole – with some significant local variations).

I’ve been rather down on my model for the Lab/LD inaccuracies, but I am going to be pleased that the Tory seat share was far closer than most other models and – unlike UNS – the actual result (Choice between Con minority or Con/LD coalition on my model versus no chance of a minority for either and equal feasibility of Con/LD or Lab/LD coalition) was correct. However, with the “GIGO” problem that we were faced with, the fact that there are additional parameters that need to be estimated (primarily the unwind percentage) is a problem – a way of calculating this factor rather than estimating it was needed.

Actually, what may have happened with the Labour seats might be well explained by Martin Baxter – when the Lib Dem vote share during the middle of the last Parliament was plumbing depths and his model started predicting zero Lib Dem seats, he adjusted his “Transition Model” (based upon Proportional Loss) to allow for core and floating voters – the “Strong Transition Model”. I believe that his predictor did very well – with the correct vote shares, he got 299, 255, 65. Maybe the Additive UNS versus Proportional Loss debate needs to be reopened? Certainly his method of adjusting for “strong” and “weak” voters seems to have done very well.

The Future

Interestingly, the regional swings don’t seem to have had a very large effect, changing the result by two or three seats at most. Definitely demographic swing is more important than regional swing. Ethnic factors and incumbency are also necessary.

So – in order to predict reliably, we need:

- Reliable poll data to input

- A way of judging whether or not to adjust for incumbency

- Reliable ethnic breakdowns and a way of highlighting affected constituencies

- Reliable class-based breakdowns and a way of highlighting effective constituencies

Unfortunately, with extra inputs we have the problem of judging what they should be – one significant issue with my model. My estimate of 66% unwind with those particular swing standard deviations was pretty good for Tory seats and the overall outcome, but having to estimate the inputs like that is a weakness. Unfortunately, UNS again had significant problems (the area of hung parliament predicted being rather different to the reality, with Tory minority closed off and Lab/LD coalition very much on the cards). However, as a rule of thumb, it’s a general guideline, but we do need to estimate how far it will be out and in which direction.

And all of this may well be redundant - a change to AV with heavily redrawn constituencies to reduce MP numbers will be a total step change.