The Big Reveal goes awry

This morning was the big reveal. The experts would announce the results of the surveys and tell us who was going to head the ticket and who was going to take the second spot, and then we would have a great photo op with the two candidates raising their hands and promising to lead the joint ticket to victory.

It didn’t quite work out that way.

The rules of the game were not spelled out clearly in the agreement three days ago, so as you might expect, the two sides did not agree on what to look at, much less what the results were. The KMT said that there were nine surveys, of which Hou won 8, and the TPP said that there were only 6 valid surveys and both sides won three each. That is, the KMT thought that Hou was clearly victorious, while the TPP argued that the situation is still unresolved. At present, both sides say that they are still working towards cooperation, but all signs point to a deadlock and potential breakdown.

Before we get into that, let’s look at the results.

There were nine polls proposed for consideration. Each poll asked two questions: “Do you support a Hou-Ko ticket or Lai-Hsiao ticket?” and “Do you support a Ko-Hou ticket or a Lai-Hsiao ticket? Each survey had a different margin of error based on the sample size. Everyone agrees on this set of facts. They don’t agree on much else.

	pollster	Q1 Hou Ko	Q1 Lai Hsiao	Q2 Ko Hou	Q2 Lai Hsiao	Margin of error
1	UDN	42.00	36.00	41.00	36.00	±2.90
2	KMT	38.20	30.60	38.80	29.30	±2.55
3	東森	39.93	34.53	39.30	34.80	±2.94
4	匯流	46.10	41.60	48.30	39.20	±2.17
5	ETToday	41.60	37.10	39.60	36.20	±2.83
6	鏡電視	46.50	34.90	46.60	33.10	±2.94
7	世新	40.82	35.86	46.01	32.22	±2.94
8	好好聽	44.60	39.50	44.00	37.20	±2.97
9	TPP	39.70	33.00	44.00	32.00	±2.98

The first argument was about which polls should be used. The TPP representative* argued that three of them should not be considered. #5 was conducted using text messages rather than by calling telephones. Respondents had to actively opt in rather than passively be sampled. #3 and #8 were conducted purely by landlines; no cell phones were called. The TPP argued that this meant they could not represent the entire population. The KMT rejected these arguments, arguing that all nine surveys should be considered as valid. The two sides agreed to set this question aside and look first at the other six surveys. However, neither side yielded on this question.

(* The TPP did not appoint Ko’s high school classmate as its expert, as he had suggested he planned to do. Instead, they appointed the head of a polling company. Perhaps one of his staff members persuaded him they needed a professional for this job.)

Personally, I think this was an entirely predictable argument. The TPP should have insisted on only using surveys that included cell phones during the original negotiations. They made a mistake when they didn’t specify what kinds of surveys would be acceptable to them.

A second, and the most intractable, argument was over how to interpret the margin of error. I’m afraid we need to do a very short lesson in statistics to understand each side’s position. So bear with me.

When you do a survey, you always get a point estimate. That is, you get a number such as “35.2% of people support Ko.” This number is almost always wrong. The actual percentage in the full population is almost always a little bit different. The margin of error helps us to understand how much different the actual number might be from our point estimate.

Imagine flipping a coin. The actual probability of getting heads is 50%. If you flip that coin twice, there’s a good chance you won’t get heads exactly one time. Fortunately, large numbers are our friends. If you flip the coin 1000 times, you probably won’t get exactly 500 heads, but you will get pretty close to that. In fact, we know the margin of error for that estimate with 95% confidence is 1/√n, where n is sample size. Since 1/√1000=0.032, we can be 95% sure that we will get between 468 and 532 heads. To put it another way, if we flip the coin 1000 times and get exactly 487 heads, we can be 95% confident that the actual probability is somewhere in the interval of 487 plus or minus 32. Lo and behold, 500 is inside the 95% confidence interval of 455 to 519. If you repeat this exercise an infinite number of times, 95% of the confidence intervals around your point estimates will contain 500.

So let’s imagine a survey with sample size 1068 that shows Ko at 34% and Hou at 30%. Is 34% higher than 30%? Should Ko win this point? 1/√1068=0.030, so Ko’s confidence interval is from 31% to 37%. Ko is statistically significantly higher than 30%, so he wins, right? Not so fast. There is one problem with that, and this is the critical point. Hou’s 30% support is also taken from this survey, so it is also a point estimate and it also has a margin of error. Hou’s 95% confidence interval is between 27% and 33%. These two confidence intervals overlap a little, so we cannot be 95% confident that Ko’s support is, in fact, higher than Hou’s support. Even though Ko leads by 4% and the margin of error is only 3%, 34% and 30% are actually not statistically significantly different.

(This is a point we often ignore in common discourse, where we usually just look at a single number and the margin of error, but if you’ve ever tried to publish a paper in a serious academic journal, you will know that overlapping confidence intervals are the kiss of death, no matter how small the overlap is. Ko, as a distinguished academic, should be well aware of this.)

So, let’s look at how the KMT wants to interpret the results.

	pollster	Hou- Ko	MoE	Who is higher?	Who gets a point?
1	UDN	1.00	±2.90	neither	Hou
2	KMT	-0.60	±2.55	neither	Hou
3*	東森	0.63	±2.94	neither	Hou
4	匯流	-2.20	±2.17	neither	Hou
5*	ETToday	2.00	±2.83	neither	Hou
6	鏡電視	-0.10	±2.94	neither	Hou
7	世新	-5.19	±2.94	Ko	Ko
8*	好好聽	0.60	±2.97	neither	Hou
9	TPP	-4.30	±2.98	neither	Hou

Only one of these results is statistically significantly different, so the score is 8 to 1. Even if you throw away the three surveys that the TPP disputes, the score is still 5 to 1 in favor of Hou.

(Actually, it isn’t clear why Ko is deemed to be significantly higher in survey #7. It looks like that should also be within the margin of error. Just after Mrs. Garlic pointed that out to me, a reporter asked Eric Chu about this at the news conference. He did not give a clear answer. We’ll come back to this in a minute.)

(Let’s step back and look at the point estimates and ignore the margin of error for a minute. Ko actually does better in five of the nine surveys. And if you throw away the three disputed surveys, Ko wins 5 out of 6. Remember, Ko was the one who introduced margin of error into this discussion in the first place. Oops.)

OK, now let’s look at how the TPP wants to interpret the results.

	pollster	Ko-Lai-(Hou-Lai)	MoE	Who is higher?	Who gets a point?
1	UDN	0.00	±2.90	neither	Hou
2	KMT	1.90	±2.55	neither	Hou
3*	東森
4	匯流	4.60	±2.17	Ko	Ko
5*	ETToday
6	鏡電視	1.90	±2.94	neither	Hou
7	世新	8.83	±2.94	Ko	Ko
8*	好好聽
9	TPP	5.30	±2.98	Ko	Ko

The TPP insists that there are only 6 valid poles, and they think that Ko was significantly higher in three of the polls and no one was significantly higher in the other three. Therefore, each side gets three points.

You can see what they’re doing here. They are using margin of error in the intuitive way, treating it as the standard to differentiate between two point estimates. If the two point estimates differ by more than the margin of error, they consider that to be a statistically significantly different period.

That’s not good statistics, but it is an intuitive argument that they can make to the public. This is why they have yelled loudly that they were comfortable yielding 3%, but the KMT is unfairly insisting on a 6% advantage.

Once again, I have to conclude that Ko did a lousy job of negotiating the terms of this contest. He didn’t stipulate he would yield 3%. He said the result would have to be outside the margin of error. I suppose he might have thought he was yielding 3%; after all, what kind of idiot would yield 6%?

You might have noticed that the point estimates in the two tables are different. This is the third obvious point of contention. In the KMT tables, the numbers are support for Hou minus support for Ko. Lai’s results are ignored. In the TPP tables, they compare Ko’s advantage over Lai to Hou’s advantage over Lai. This doesn’t make a whole lot of difference, but it’s just one more thing that they should have clarified in advance. This whole process was extremely sloppy.

I think these different definitions of the dependent variable might be behind the KMT’s judgment that Ko was the winner of survey #7. The numbers in the KMT table suggest that Ko was not, in fact, significantly higher than Hou on survey #7. (5.19 < 2*2.94) However, in the TPP table, even using the KMT definition of margin of error, Ko was in fact higher than Hou. (8.83 > 2*2.94) I’m guessing the KMT did not want to press the issue and insist on its definition of the point estimate. Since it was not absolutely clear that there was no significant difference, I’m guessing they decided to yield this point. After all, they don’t want to humiliate Ko. They want his support eventually, or at least his supporters’ votes. There’s no harm in giving him a little face.

If this was, in fact, an olive branch, my instincts tell me it’s not nearly enough. The fact that Ko has decided to dispute these results suggests to me that he is still planning on running for the presidency. Everyone is still talking about finding a way to cooperate, but the KMT thinks it has won while the TPP’s thinks it’s still negotiating. Ko suggested this morning that there were still five days before the registration deadline, and that’s enough time to holding traditional polling primary by doing a new survey. (Well, why didn’t they think about that three days ago when there were still eight days left!?) I can’t imagine the KMT agreeing to this. They have to think that the process is already over.

There are three plausible outcomes to this controversy. First, Ko could completely surrender and agree to run as Hou’s vice presidential candidate. I think this is looking less and less likely all the time. Second, he could decide not to run, but also not to serve as Hou’s running mate. He would just drop out and focus on the legislative candidates. The KMT would be on its own, and the TPP would go back to being an opposition party that distrusts both major parties. Third, he could declare that this process was a flat-out fiasco that, if anything, showed him to be the stronger candidate. Then he would righteously announce he was running for the presidency. If I had to guess right now, this is what I would say is most likely.

In either of the first two scenarios, I suspect the two candidates have done themselves no favors. They both want to get votes from each other ‘s supporters, but gluing together in an uneasy coalition will be more difficult now because of the higher level of acrimony between the two sides. Likewise in the third scenario, strategic voting will be more difficult because both sides will be more likely to stick with their first choice no matter what and less likely to think of the other side as their clear second favorite. I suspect both sides would have been better off if they had never started this attempted cooperation, which is quickly turning into a calamitous debacle.

update: As several of the commenters pointed out, the margin of error is a little more complicated than I made it out to be. Sorry, I’m not a great methodologist. The precise formula is quite complicated and you need to know the covariance, which I don’t know how to calculate. They were discussing this in my Line group, and someone gave an example of n=1000 and covariance is 3% so the two variables would have a margin of error of 4.243%. In retrospect this is probably why survey #7 was deemed to show Ko leading, and survey #9 was deemed to show no significant difference.

So I’m wrong about the exact calculation, but the larger point I was trying to make is that the margin of error that you see reported in the media is not actually the standard for determining whether two variables are statistically significantly different.