Tag Archives: polling

The Corbyn Effect, or ‘you’re nobody til everybody hates you’

There were a couple of by-elections in the UK this week, both in traditionally-safe-ish Labour seats.

Normally in the sixth year of Conservative government, this would be a boring event that nobody cared about: a medium-strength opposition wins government seats at by-elections (even if, as with Labour under Neil Kinnock pre-1992 and under Ed Miliband pre-2015, it will go on to narrowly lose the general), but even a weak opposition holds its own seats.

However, the times right now aren’t normal, and so Labour was seen as under threat – from the shambolically dumb and awful neo-fascists of UKIP in depressed former pottery town Stoke on Trent Central, and from the Tories in part-ex-industrial-towns, part-nuclear-power-plant, part-grumpy-farmers Copeland. The threats were real. As the fascists imploded, Labour beat them by a not-exactly-resounding-but-better-than-feared majority of 2,620. Meanwhile, the Tories won Copeland with a majority of 2,147.

This fits the post-Brexit course of UK politics: Labour’s performance in 2010 and 2015 was artificially propped up because the kind of traditional Tories who don’t-want-a-person-of-colour-for-a-neighbour voted UKIP. Now that the Tory party is led by a petty authoritarian who hates everything from after 1953, rather than a toff who loves money and doesn’t really care very much about anything else, these people have gone back to the Tories.

Labour’s hilariously awful leadership and infighting over the last 18 months hasn’t helped, and certainly hasn’t provided an alternative narrative, but at worst Jeremy Corbyn and his backstabbier rivals are drilling new holes in the bottom of a ship that was already leaking and on course for the rocks anyway.

What does any of this have to do with popularity?

After the election, Momentum true believers – both within the Labour party organisation and outside – displayed an Iraqi Information Minister-ish commitment to presenting the results as a Great Victory. I was particularly struck by the quote in this tweet:

On the face of it, Lavery’s claim is ridiculous. But there are some completely reasonable definitions of ‘popularity’ under which he has a point.

Like most of the people reading this piece, I would score net favourability of zero in an opinion poll, because only a niche selection of transport and politics wonks have the slighest idea who I am. In one sense, that makes me 40 points more popular than Jeremy Corbyn. But in another sense it’s silly to say that I’m vastly more popular than Jeremy Corbyn, because if I organised a weird cult rally in my name then the turnout would be nobody; probably not even my boyfriend, and certainly not 50,000 Momentum pod people.

The difference here is gross and net popularity. Both are important in political leaders, depending on the kind of organisation they lead and the kind of electoral system within which they operate, but we tend to dwell on the net numbers. So, I’ve put together a chart aggregating poll results in order to show gross popularity and unpopularity of various UK political leaders, which reflects the fact that most people haven’t heard of most of them:

This is based on two slightly different YouGov polls – one from 2-3 Feb 2017 of famous politicians and one from 13-21 Feb 2017 of Labour politicians (PDF). YouGov calculated the numbers for non-Labour politicians; I calculated the numbers for Labour politicians.

So the claim that Jeremy Corbyn is one of Britain’s most popular politicians is defensible. On gross popularity, he’s the sixth-most popular current Westminster politician, of the ones for whom I could find recent data. At the same time, he’s the most unpopular current Westminster politician by a fairly wide margin.

The problem here comes when we think about when the two different sorts of popularity are relevant.

If you want to dress up as an evil clown and sell albums to dumb flyover state poor people, then being extremely unpopular on net but with high gross popularity is a route to immense success. If you want to push horrific far-right ideas into mainstream UK discourse and then fuck off to America to do speaking tours to dumb flyover state rich people, then likewise.

On the other hand, if you want to become President of France, then the runoff electoral system means that you need decent net popularity. The two candidates with the highest gross popularity will get into the run-off, but then the candidate who the voters of France hate the least will be the winner. This setup kept Jean-Marie Le Pen out of the Élysée in 2002, and hopefully will keep his daughter out this year.

The UK Labour Party, traditionally, competed in the President of France space: a centre-left party that people voted for under the UK’s antiquated first-past-the-post electoral system because they thought it was less awful than whatever the Tory Party was up to at the time. That requires leadership with decent net popularity, and high gross popularity isn’t very important.

However, Corbyn’s popularity is the opposite: someone who has a strong enough following of fervent weirdos that their success condition is “shifting the acceptable window of mainstream opinion”. This isn’t a bad thing in its own right; there’s a reasonable argument that we need a Farage-of-the-left to counter the global rise of far-right ideology. The probem is that at the same time, it isn’t a particularly great idea for either side to chain such a movement to a moderate centre-left party seeking to defend its position in hundreds of FPTP seats.

Header image by The People Speak / CC-BY 2.0.

There was no late swing, and there were no shy Tories

One of the most interesting questions after the 2015 UK General Election is, how could all of the electoral polling possibly have gone so incredibly wrong?

Labour and the Conservatives were predicted to be neck-and-neck and both short of forming a government on their own, with Labour losing about 50 seats in Scotland to the Scottish Nationalist Party.

election_eve_poll
UK 2015 seat prediction on election eve, according to the Guardian’s poll-of-polls

Instead, the Conservatives won a small majority of total seats. Netted out, Labour gained only four English seats from the Conservatives despite its low 2010 base, and lost two seats in its heartland of Wales to the Conservatives.

Labour’s remaining gains in England came from the brutal destruction of the Liberal Democrats, which the polls dramatically understated. This was cold comfort, as the Conservatives took far more former Lib Dem seats, including almost all of the ones that the polls had predicted would stay orange.

Actual UK election results
UK 2015 actual UK election results

So what the hell happened?

Two popularly floated explanations in the media have been a late swing, and the ‘shy Tories’ problem. Both are almost certainly wrong.

Don’t mean a thing if it ain’t got late swing

One thing we can reasonably rule out is the concept of a late swing to the Conservatives: a scenario where the polls accurately captured voting intentions, but where people changed their mind at the last minute.

We know this, because online pollster YouGov made an innovative attempt at a kind of internet-based exit poll (this is not how YouGov described it, but it’ll do). After the vote, it contacted members of its panel and asked them how they voted. The results of this poll were almost identical to those recorded in opinion polls leading up to the election.

Meanwhile, the UK’s major TV networks carried out a traditional exit poll, in which voters at polling stations effectively repeated their real vote. This poll (which covered a balanced range of constituencies, but whose results weren’t adjusted as they are for small-sample opinion polls) found results that were utterly different from all published opinion polls, and came far closer to the final result.

Putting the two together, the likeliest outcome is that people were relatively honest to YouGov about how they voted, and that they voted in the same way they told all the pollsters that they were going to vote. This isn’t a late swing problem.

No True Shy Tories

If we ignore Scotland (where the polls were pretty much correct), this is a similar outcome to the 1992 General Election: opinion polling predicted a majority for Labour, but the Conservatives instead won a majority and another five years of power.

A common narrative for poll failure after the 1992 election was one of ‘shy Tories’ [1]. In this story, because Tories are seen as baby-eating monsters, folk who support them are reluctant to confess anything so vile in polite society, and therefore tell pollsters that they’re going to vote for the Green Party, the Lib Dems, or possibly Hitler.

From 1992 onwards, polls were weighted much more carefully to account for this perceived problem, with actual previous election results and vote flows also being used to adjust raw data into something that can reasonably be expected. This happened in 2015, as it has for every election in between [2].

We know that the internet provides the illusion of anonymity [3]. People who’d be unlikely in real life to yell at a footballer or a children’s novelist that they were a scumsucking whorebag are quite happy to do so over Twitter. Foul-minded depravities that only the boldest souls would request at a specialist bookstore are regularly obtained by the mildest-mannered by an HTTP request.

In this environment, if ‘shy Tories’ and poor adjustment for them were the major problems, you would expect internet-based polls to have come closer to the real result than phone-based polls. But they did the opposite:

The current 10-day average among telephone polls has the Tories on 35.5% [and] Labour on 33.5%… The average of online polls has the Conservatives (32%) trailing Labour (34%)

So what is the explanation then? This goes a bit beyond the scope of a quick blog post. But having ruled out late swing and unusually shy Tories in particular, what we have left, more broadly, is the nature of the weighting applied. Is groupthink among pollers so great that weighting is used to ensure that you match everyone else’s numbers and don’t look uniquely silly? Are there problems with the underlying data used for adjustment?

Personally, I suspect this may be a significant part of it:

According to the British Election Study (BES), nearly 60 per cent of young people, aged 18-24, turned out to vote. YouGov had, however, previously suggested that nearly 69% of under-25s were “absolutely certain” to vote on 7 May.

Age is one of the most important features driving voting choice, and older voters are both far more conservative and far more Conservative than younger voters [4]. If turnout among younger voters in 2015 was significantly lower than opinion pollsters were expecting, this seems like a good starting point for a post-mortem.

Update: YouGov’s Anthony Wells comments on the YouGov not-quite-an-exit-poll:

[1] Some polling experts think the actual failure in 1992 had more to do with weighting based on outdated demographic information, but opinion is divided on the matter.

[2] Several polls in 2015 that showed 33-35% Labour vote shares were weighted down from raw data that showed Labour with closer to 40%.

[3] An illusion that diminishes the closer one comes to working in IT security.

[4] There are papers suggesting that this is to do with cohorts rather than ageing, and that It’s All More Complicated Than That, but anyone denying the basic proposition above is a contrarian, a charlatan or both.