Sunday 28 February 2010

WattsUp with YouGov? Haven’t we Been Here Before?

In this article following on from Flockers on the main site last Friday I want to see whether there are parallels between the excessive reliance on black-box computer modelling in the Climate Change debate and the way in which the pollsters use models to systematically reweigh the raw polling data.

Whenever you go and collect data and do an initial analysis on it, there’s always one reason or another why it just doesn’t seem to fit with expectations. Perhaps there have been some gaps in the data? Or perhaps one series of data points seems to be over-represented? So, it’s the most natural thing in the world for the statistician to seek to fill-in or even-out these gaps, perhaps by reference to some sort of historical pattern or even by some sort of fudge-factor until the result seems more sensible and in-line with what you’d expect.

And, of course that’s what Phil Jones at his colleagues at the University of East Anglia just down the road from me did for about 15 years with his climate change models.

With hindsight it’s fairly easy to see that Jones and his colleagues had bought into a fixed notion that temperatures would be expected to rise as the amount of CO2 in the atmosphere increased. And so he designed his models accordingly. And then he got into a series of positive-feedback loops that progressively increased and then exaggerated the minor increases in the temperature record to fit into the peer-reviewed collective wisdom.

It didn’t matter that the fastest rises in CO2 occurred in the mid-1800’s [Krakatoa?] or that the weather during the Battle of Hastings was warmer than today [the Medieval Warm Period], a narrative had taken hold and people then became so engrossed in the computer models and their assumptions that they forgot to look out of the window to see what was really happening outside.

So, are the 2010 political polling reflecting what we’re seeing as we look out of our own windows? Is this poll narrowing really reflecting the conversations we’re having with colleagues at work or in the pub?

Have the Pollsters’ models got out of kilter with what we’re experiencing on the ground? Which is why I want an answer to Richard Tyndall’s Kellner Killer Question from February 20th, 2010 at 10:06 pm.
But I do have doubts about these Yougov daily polls. The difference between the pre and post weighting numbers seems to me to be unjustified and I would like to see this confirmed by a ‘real’ poll before saying whether or not the Tories are getting what they deserve for their recent mishandling of policy.

But before I highlight a worked example, let’s just get some things straight so you don’t characterise me as a Climate-Change or Polling denier. Let’s review some of the events of the last few weeks.

• As Bob Worcester says, it’s the share, not the lead that’s important. You only need a majority of one to win a seat. It’s clear that the Tory’s lead has narrowed over the last few weeks, even if their share has remained quite stable in the 38-40pc range.
• Labour’s share has increased. It’s been wall-to-wall Labour on the telly so Mike’s third rule applies: The more you’re on the box, the better you do in the polls. Their improvement over the last week seems to partly prove the saying that ‘There’s no such thing as bad publicity’.
• The LibDems have dropped and are flat-lining at 17/18pc and their misfortune has been Labour’s gain. The LibDems and Labour seem to be fishing in the same pool. As the LibDem vote falls, Labour’s rises, which has contributed most of all to Labour’s recovery.
• We also know that all-of-a-sudden Labour is doing substantially better in Scotland at the expense of the SNP so in aggregate over the whole of Great Britain, Labour’s percentage is higher. It’s debatable whether the increase is reflected in the English Towns, where the battle will be won or lost.

Oh yes and we also know a few weeks ago YouGov changed their weighting methodology in preparation for their daily polls, noting that Tory voters tended to respond more quickly to invitations-to-survey and older voters often missed-out because they only check their emails every few days.

The goalposts just got moved. Someone at YouGov just pressed the ‘Reset’ button. What’s the effect of this been?

So, let’s have a look at one of the recent polls by having a close look at the YouGov results from 25th February, for which fieldwork was conducted on 23/24th February.



First of all, the sample was reported to be 1473. Of the 1473 sample, the ‘Headline’ result was said to be Con 38%; Lab 32%; LibDem 19%; Others 10%, amounting to 99% with rounding errors. This excludes the don’t knows and the will not vote.

Of the 1473 sample just over 400 respondents were either in the others [10%], will not vote [7%] or undecided/Don’t know [13%] categories. That surprised me and suggests that all the parties still have a lot to play for. With 10% on ‘others’ and 20% of voters undecided, it’s still Game-On for the big parties.

The press is lapping-it-up. The Hung Government narrative sells papers.

Now, the pollsters job is to try and predict the national share. But is estimating the national share the same as predicting the result of the election? Andy Cooke has been postulating that the marginals are behaving differently and Blair Freebairn’s suggested that the key battleground is the METHHs, the medium English Towns and their hinterland. I agree.

So, now I’m going to do something controversial to illustrate a point. Just hear me out while I construct this straw man. There’ll be plenty of people lining-up to demolish it in the comments so I’ll put my tin hat on now!

I’m going to make the simplistic point that in the English METHH marginals, the battle is going to be a three-way fight. The contaminating effect of the nationalist parties in Scotland and Wales [SNP/PC] will be zero. I’m also going to make the intellectual leap that in a ‘change’ election, UKIP and the Greens [except Brighton & Norwich South] will be squeezed too.

So let’s just see what happens in what’s left of the YouGov weighted sample when we discount all except those who say they’ll vote for the main parties. In this scenario, what’s left are 1057 [71% of the whole sample] respondents out of the original 1473. I’m going to suggest that this forms a proxy for the 3-way English-Town fight, where the battle will be won or lost.

In the 1046 Raw Sample, there were 496 [47%] Con; 333 [32%] Lab & 217 [21%] LibDem. In 1057 weighted Headline Result, there were 453 [43%] Con; 375 [35%] Lab & 229 [22%] LibDem. Oooh. That's a big difference between the two!

In the weighted-sample three-main-parties-only figures, the Tories are 8% ahead… and that’s including Labour’s Scottish vote, which we know is disproportionately high north of the border and irrelevant for the METHHs. In the unweighted one, they're 15% clear.

Interestingly these are pretty similar results to the Angus Reid polls that have been studiously ignored by the national press, which have showed an additional swing over-and-above in the marginals.

In the weighted result, the Tories have been scaled-back from 496 to 452 [-43] and Labour improved from 333 to 375 [+42]. The LibDems have been boosted by 12 moving from 217 to 229. That represents a swing away from the Tories to Labour across the whole country of 5.7%. That’s quite a difference in a poll that gave the Tories a 6% lead.

If we accept that the 3-way fight is a proxy for the METHHs, for the Tories, there is still some comfort in the polls. They’re ahead where they need to be but it's still squeaky-bum time. Their campaign has faltered by firing out a series of policies in a scattergun approach without communicating a core narrative since January. It’s has been exposed as a great mistake, which is being punished.


But this weekend allows them to press their own ‘Reset’ button before the voters become engaged in the campaign proper.

I fully accept that the pollsters must re-weight their samples and the way in which they do this is their intellectual property and the value in their business. And it’s good business. I’ve used YouGov myself. And been pleased by the results.

But today's gap apparently narrowing to 2%, betting money at stake, and the future of the country hanging in the balance, we’re putting a lot of trust in the YouGov computer models, which have been recently tweaked. Are the tweaks, in this case resulting in a 5.7% Con-Lab swing over-cooking it? The Tories were 6% ahead overall. The weighting is as much as the lead. It's non-trivial.

We know what’s going on here: Pollsters know that, back in 2005 a certain proportion of people backed Labour. This year the pollsters are calling voters and not as many people are saying that they’re voting Labour as before. We can see that in the raw data.

So, it may well be that either a large number of people reallyaren’t going to vote Labour after all. Or the pollster may infer that, on an historical basis, his sample is just under-weight in Labour supporters and increase the Labour share with technical adjustments over-and-above normal demographic/occupational adjustments to align with the long-term model trend.

And if so, this is the fudge-factor. This is Dr Jones’ of the UEA “Mike’s Nature Trick”. That fudge in this case amounts to 5.7%. It needs to be explained.


Which brings me back to my initial point. Are there parallels here between the muddle with the scientists down the road at the UEA over ClimateGate and what we’re seeing here? Are we seeing too much reliance on black-box systems, whose complexity is now divorcing them from the system that they’re trying to model?

I’m not having a pop at YouGov. I don’t know the answer but you’ll excuse me from posing the question with a seeming 5.7% structural adjustment being applied.

As Political Punters, we now have a judgement call to make safe in the knowledge that if things generally look too good to be true, then they are normally are too good to be true.

• Do the national voting intentions accurately reflect the likely result of the election in 640 seats?
• Is the evidence of our own eyes telling us that only one voter in twenty has changed their mind since 2005?
• Do we trust the black-box polling models when the weightings are so large. Are they arbitary?
• How much of the polling is science and how much of prediction is art. And is this why the betting markets have been relatively stable over the last few weeks.

Peter Kellner’s coming on the site on Tuesday. I hope that this thread can collate the questions that we need to ask him.

8 comments:

Richard Manns said...

I'm no expert, and I don't know much about their methodologies, but I've had a thought:

We know that the pollsters suffered from their 1992 failure, but pollsters presumably avoid adding modifiers "we should add 5 points to the Tories"; instead they went for incumbency and "shy x voters".

Now that this is the 4th election for Labour, in incumbency terms, it's like 1992. So those "shy Tory voter" modifications will, presumably, now apply equally strongly to Labour and give them a boost.

But are Labour voters shy? Speaking as a Tory, I've never known a time where it has been in vogue to be a Tory. Cameron may have "detoxified the brand", but when a new person in my work saw I was a Tory, his response was an incredulous "Why?", as if it were a crime. Meanwhile, watch a recent episode of Argumental, and you'll see a universal derision of right-wingers as "Daily Mail readers" whilst the Guardian is a paragon of civility.

I suggest that there is a fundamental difference between swing voters voting for a Tory and voting for Labour, and that is reflected in the attitudes to the parties.

I might be wrong. We don't know. There might be a "shy Labour voter". But, as some foreign journalist (according to CasinoRoyale) once said, “You British talk Left but always vote Right.”

firstlight40 said...

Bunnco,

an excellent post, with all the questions that I would want to ask Mr. Kellner.

Clearly OGH has cleaned up from the confusion of this poll.

There's a lot riding on the result of the election (including currency / bond speculators betting billions on UK future prospects). If adjustment methodologies appear to be negligent in their design or application I would not want to be in the shoes of the people who implement them

Mark Senior said...

Usually I read your articles with both respect and interest but this one lacks objectivity on a grand scale .
You state that Conservative support is stable in the 38-40% range - False . It has gone down from an average of 41% in early January to an average of 38% now .
You state that it is the LibDems who have lost support and are flatlining at 17/18% - False . LibDem support is a bit lower with Yougov true at 17% but with all other pollsters has crept up this year to 19-21% .
Like many Conservative posters you harp on about the Conservative unweighted lead , This is simply ridiculous an unweighted lead in a heavily pro Conservative sample is meaningless - just as a 1% lead in Mori's unweighted figures is meaningless because their sample will always have a pro Labour bias .
As with every poll you need to look at the base data and the weightings used and then come to a reasoned judgement of what the correct figures may be .

Anonymous said...

(AnneJGP)

Very nteresting, Bunnco.

"...a few weeks ago YouGov changed their weighting methodology in preparation for their daily polls, noting that Tory voters tended to respond more quickly to invitations-to-survey and older voters often missed-out because they only check their emails every few days"

"Older voters often missed out" clearly implies that they responded too late to be included in the survey.

But, as long as non-Tory voters responded in time to be included in the survey, why does "Tory voters responding more quickly" matter?

Does their model somehow give added weight to a quick response? If so, why?

Or do they really mean that non-Tory voters, like older voters, responded too late to be included in the survey? If so, what does that mean for their sample?

Richard Manns said...

@ Mark Senior

I agree that the Tories have been falling slightly, etc. But historically, samples have had to be weighted to avoid a massive Labour lead, not vice versa.

Why has this changed? Bunnco comments on the "black box" of methodology, so it's hard to tell. But some large factor must have kicked in to switch the weighting against Labour; even YouGov's apparent success at reducing the pro-Labour bias through internet polling never converted to a pro-Tory bias, until now (it seems).

This is the question. Why? I seem to recall that YouGov froze its sample after the BNP proposed entryism to distort their results. Is this still true? In which case, why the shift in weighting? YouGov says they've changed things to avoid the loss of "older voters", but don't they historically vote Tory? If so, you'd expect any adjustment to be pro-Tory!

I propose that it's a pro-incumbency weighting to deal with shy voters last seen in the Tories' 4th victory, as this would be Labour's 4th.

Mark Senior said...

Richard , If you look at the detailed data tables last page , you will see that Yougov have been oversampling by a large extent males over 55 and respondents with a Conservative Party ID , I would suggest that these 2 facts are linked .
You are correct that the telephone pollsters have invariably to weight Labour downwards because telephone pollsters contact a disproportionate number of Labour voters . Yougov are an internet pollster and they have the opposite pronlem as do AR but seemingly to a smaller extent .

bunnco said...

Mark Senior. Naturally I apologise for putting the Tory range at 38-40 instead of 38-41. And ditto for the LibDem ranges.

The point I'm trying to make here is that we know that different pollsters have different weighting criteria.

But lately there appear to be some funnies and, in the case of YouGov, combined with a tweak in their methodologies.

It's reasonable to wonder why this might be, what effect it might have on the results and whether there are parallels to be drawn with other black-box systems, which [re climategate] we know have been less robust than we first imagined.

Richard Manns said...

Mark,

I didn't know about those sampling factors.

It is a concern that they have more male over-55s, but again, they're the ones that are likely to vote. As for party ID, this is fluid, although not as fluid as day-to-day preference.

So let me propose this thought experiment: if the Tory vote has "hardened", i.e. more and more people positively associate themselves with the Tories although the overall percentage of current Tory supporters remained stable, then this would paradoxically reduce the weighted Tory percentage.

You state that internet pollsters have the opposite problem of pro-Tory sampling. Do they? As Flockers illustrated, their downweighting of the Tories has shifted from 0.02% mid-last year to 6.3% now.

This is a huge shift, and as Flockers illustrated, the unweighted samples have given a 14-16% lead to the Tories for almost a year.

What has changed? I can't find the tables for May 2009, I wanted to compare the group sizes myself, especially over-55s.