X Factor 2012 Review Part 6: The Impressive Accuracy of YouGov

On US election day, we published a light-hearted piece suggesting ten ways in which the presidential campaign could be compared to the X Factor. Now we can add an eleventh: the result vindicated the profession of opinion polling.

As Daniel and I watched the comments come in under our final preview article on the final Saturday, we joked that Sofabet had become like fivethirtyeight.com above the line and unskewedpolls.com below it. We’d happily trusted YouGov’s forecast of a win for James, but only one commenter (take a bow, Ben Cook) concurred; of the six commenters who put up their own 1-2-3 predictions, four went for Christopher and two for Jahmene. All put James in third.

Joe Twyman, YouGov’s head of polling and an avid X Factor fan, had kindly agreed to be interviewed by Sofabet after last year’s series to explore the firm’s polling methodology (read it here). We got in touch with Joe again after this year’s series, and began by asking for his take on why the polls – in both the US election and UK X Factor – had inspired such strong scepticism.

“Even very smart people can always find reasons to dismiss evidence that doesn’t fit with what they’re emotionally invested in believing,” Joe says. “I met a Republican strategist just before the US election who assured me that ‘mainstream’ polling could be ignored because the more reliable indicators were the enthusiasm of crowds at rallies, and the number of lawn signs, and…”.

As Joe says, with both politics and X Factor, the principle of polling is the same: you have to “ask the right people the right questions in the right way”. Much of the pollster’s craft comes in sampling and weighting by demographics to translate between the composition of people polled and the actual electorate. That was the source of scepticism in the US – Republicans believed that pollsters were overestimating the likely turnout of the young and minorities, demographics likely to favour Obama.

It’s also why you can’t compare a poll from a professional organisation such as YouGov to the kind of poll that’s more commonly conducted on reality TV – one which involves a website sticking up a “vote” button. There’s no reason to think that the people who vote in such polls are at all representative of the kind of people who vote on the show.

Joe continues: “With political polling, too, you wouldn’t believe the number of people we get phoning the office insisting that our poll must be wrong because ‘everyone I know thinks that…’. Fine, but is the group of people you know carefully weighted by demographics to be representative of the population at large?”

As Joe explained in our interview with him last year, polling on the X Factor is never going to approach political polling in accuracy, in part because on political election days it’s rare that anything happens to change voters’ minds, whereas obviously a lot can happen to change minds during an X Factor final. Nonetheless, the story of YouGov’s three X Factor final polls is one of increasingly impressive accuracy. Their poll ahead of the 2010 final showed:

Matt Cardle 52%
Rebecca Ferguson 32%
One Direction 9%
Cher Lloyd 8%

This turned out to be too kind to Matt and Rebecca and too harsh on the acts aimed at the younger demographic, One Direction and Cher Lloyd. Still, it’s worth bearing in mind how underwhelming Matt and Rebecca’s celebrity duets were (they were also first and second in the running order), and Matt was below par in general with a sore throat.

Furthermore the order was bang on, and by no means obvious – true, Matt was favourite at a shade of odds-on, but the markets gave Rebecca (who traded at around 9/4) and especially One Direction (10/3) much more of a chance than the YouGov poll did. The result, up to the first voting freeze:

Matt Cardle 39.9%
Rebecca Ferguson 25.7%
One Direction 18.5%
Cher Lloyd 15.8%

The final poll from YouGov in 2011 was much tighter – indeed, Joe was at pains to point out beforehand that the firm was calling it as “too close to call” rather than predicting Marcus as the winner. (We at Sofabet, alas, were not so cautious). The figures were:

Marcus Collins 39%
Little Mix 35%
Amelia Lily 26%

The result:

Marcus Collins 34.5%
Little Mix 39%
Amelia Lily 26.5%

On the face of it, this might look like a less good result than in 2010 because the order was wrong. But actually, it was much better as each of the three acts in 2011 were closer to the predicted figure than any of the four acts in 2010. Then, Matt was 12.1% off, One Direction 9.5% off, Cher 7.8% off, and Rebecca 6.3% off. In 2011, Marcus was 4.5% off, Little Mix 4% off, and Amelia 0.5% off.

Pertinently, the poll was off in the direction in which producers had, by our reading, tried to steer the final – boosting Little Mix, and nobbling Marcus. In contrast, 2012’s poll showed a lead for the act which we were fairly sure producers most wanted to win. YouGov’s figures from the 2012 final poll:

James Arthur 50%
Jahmene Douglas 35%
Christopher Maloney 15%

Both James and Jahmene would have had to be off by an average of 7.5% for Jahmene to have won; both James and Christopher would have had to be off by an average of 17.5% for Christopher to have won. The actual figures turned out to be much, much more accurate:

James Arthur 51.7%
Jahmene Douglas 31.5%
Christopher Maloney 16.8%.

Interestingly, the crosstabs of the YouGov poll show that one of the questions they asked (see page 5 of the pdf) was whether respondents voted in the semi-final, and, if so, for whom? The poll findings:

James Arthur 39%
Jahmene Douglas 25%
Christopher Maloney 18%
Union J 10%
Can’t remember 8%

The actual result of the semi, once the statistics were revealed:

James Arthur 41.2%
Jahmene Douglas 22%
Christopher Maloney 18.7%
Union J 18.1%

Again, remarkably close – and interesting, as an aside, that the most underestimated act was the one aimed at the teen market, Union J (although maybe the 8% who said “can’t remember” were all Union J voters too embarrassed to admit it).

Daniel had noted in his preview article that in the last couple of years it was the teen acts – One Direction, Cher and Little Mix – who had been lowballed; conversely, just as granny’s choice Marcus underperformed last year, this year the act marginally overestimated in both semi and final was Jahmene (the crosstabs show it was he, and not Christopher, who was winning the older demographics).

When we talked to Joe afterwards, he was obviously delighted to have come so close to the result. But, hand on heart, how confident had he been beforehand?

“Honestly, before the final shows I wouldn’t have been completely surprised if Jahmene had won. He has an extremely powerful backstory, and the X Factor is increasingly about backstory. You never can tell how that’s going to be packaged and play out in a final. But, based on our figures, I was certain that Christopher would be finishing third.”

As Joe told us in our interview with him last year, YouGov don’t poll just the final – they poll X Factor throughout the series, but keep the data private. Joe says: “We could see that Christopher was out in front during the early weeks, and also how very close it was at the bottom around week 7, so the bottom two of Ella and James wasn’t such a huge shock. And then in week 8, our polling picked up that huge bounce for James”.

It’s an especially impressive feat given that viewing figures for the show have gone down so much, making it more difficult to poll. Joe says: “Because of the lower viewing figures, we did our polling slightly differently this year. Instead of identifying people who were voting in the early weeks, and then going back to poll them ahead of the final, this year we identified people who were watching, and then went back to them ahead of the final. Because of that, we had to bump the sample size up.

“Our usual rule of thumb is that a show has to have at least 10 million viewers consistently for us to be happy to poll it. In other words, if we hadn’t polled the X Factor before, we wouldn’t have done it this year – we went ahead with it only for historical reasons. It’s also why we’ve never polled Strictly, and why we weren’t tempted to poll The Voice.

“Over the years, there have been only three reality shows that have dominated the national conversation sufficiently for us to feel confident polling – Pop Idol, Big Brother and X Factor. Something new will have to come along at some point, because sadly it seems to be increasingly clear that the X Factor, in its current form at least, has jumped the shark. And as we saw with Big Brother, when the decline sets in, producers tend to start tinkering with the format.

“We were extremely relieved this year that there weren’t any mid-show surprises, like with the return of Amelia Lily in 2011. That kind of thing makes our job much harder.

“But who knows what will happen next year?”

We’ll certainly be keeping fingers crossed that we get a YouGov poll on next year’s show. And on a more parochial note, please don’t be shy about getting involved in the Sofabet comments – we want it to be a friendly and welcoming place for a plurality of opinions.

This has been the penultimate article in our X Factor 2012 review series. Tomorrow, Dug is back to round things off with a look at what James’s win means for the franchise, then in the coming week Daniel turns Sofabet’s attention to Eurovision. A reminder that you can keep updated of Sofabet posts via email, RSS and following @sofabet on Twitter (and while you’re at it, do follow YouGov’s @JoeTwyman, too).