reporting polls in the UDN

What if you are a partisan media outlet, and you have to report news that looks bad for your preferred party? What if you have a lot of flexibility in how you report the story? If you want to maintain a basic level of media ethics, you have to accurately report the basic facts. However, you have a lot of leeway in how you frame those facts, emphasizing some and downplaying others.

In this post, I’m going to look at how the United Daily News has reported its own recent survey results. The UDN has a strong partisan preference for the KMT. (I don’t think even my good friends working for the UDN would dispute the newspaper’s institutional political bias.) However, their own surveys show Han Kuo-yu trailing far, far behind. This isn’t the kind of news that will boost the morale of KMT sympathizers or inspire consumers to purchase lots of UDN copies at 7-11. So how should the UDN report those results?

 

[Aside: This post feels extremely old-fashioned. It’s about print layouts of newspapers. Printed on actual paper. Is this 1993? Yes, yes. Humor me. I know I’m old and out of date. The thing is, even if very few people still read hard copies of newspapers, those few remaining readers are very influential. Newspapers still drive the rest of the media discourse. Newspapers may not break the news any more, but they still provide an irreplaceable blend of depth, authority, and speed. Newspapers are the foundation that everything else is built on top of (including this blog).]

 

How do newspapers ordinarily report their own survey results? Let’s look at a few recent results. Here is a recent survey from the Liberty Times. The survey is in the middle of the first page, and the headline simply reports the horse race numbers. The Liberty Times is a green-leaning newspaper, so they and their readers are probably happy to see those numbers.

Apple Daily is politically more neutral. Their first priority is to sell papers, not to promote either KMT or DPP interests. This is from Dec 3, and their entire front page is dedicated to their poll.

As with the Liberty Times, the main headline is the horse race result. Apple also has consistently fantastic graphics that illustrate something about the race. Here, they are making the point that Han’s attempt to disrupt polling isn’t making it any easier for him to get to the top. [Apple is making a push to encourage readers to purchase digital subscriptions instead of hard copies. Unfortunately, the digital readers don’t get the fun graphics. Sometimes print is better!]

Here’s Apple’s poll from earlier this week (Dec 17). Again, the headline is all about the horse race (in which Han is losing by 29 points).

So that is what normal newspaper stories about the newspaper’s own polls usually look like. What about recent UDN reporting?

Here is the Dec 10 issue of UDN. The headline is not about the horse race. In fact, that is not even mentioned on the front page. The front page story and graphics are all about electoral culture. The headline screams, 45% think green camp online soldiers are ruining electoral culture.

If you read the fine print, that headline seems a bit exaggerated. In this section, they asked four questions. [Note that UDN doesn’t provide their exact question wording, so I – along with all their other readers – am relying on their graphics and reporting.] First, how serious is the problem of purchasing online soldiers? 35% said very serious, 18% said somewhat serious, and 15% said not serious at all. Second, how serious is the problem of untrue mudslinging? 33% said very serious, 20% said somewhat serious, and 23% said not serious at all. Note that these question are set up to produce an impression of a bad electoral culture. The normal, more neutral way to ask would be, “Some people say that purchasing social media influencers or other online influence is a serious problem. Do you think that it is very serious, somewhat serious, not too serious, or not serious at all?” Instead of leading the respondent to assume the problem is serious, the idea is presented as someone’s opinion (which may be totally wrong). There are also two positive answers and two negative answers. The third question asks whether purchasing online soldiers or untrue mudslinging is a more serious problem. They don’t report results from this question, but they use those answers to ask the fourth question, which party is most responsible for this problem? For respondents who thought that buying online soldiers was more serious, 45% blamed the DPP, 19% blamed the KMT, and 1% blamed other parties. This question was the inspiration for the headline. Note that these questions fit neatly into KMT campaign themes. The KMT has been screaming for months that DPP attacks on Han about his real estate dealings, entering the PRC Liason Office in Hong Kong, drinking, etc. are just baseless mudslinging, and they have been complaining that everyone online supports the green camp since at least the Sunflower Movement. Blue supporters know exactly how to answer these questions; green camp sympathizers don’t think about these events in quite the same terminology. If you wanted to give green camp sympathizers an equivalent prompt, you should probably include the PRC as a response category.

Enough of the first page. UDN also dedicated most of page 2 to this survey.

Page 2 covers the horse race, though the horse race (Tsai 48, Han 20, Soong 9) is shown in a somewhat complicated chart that you have to stare at for a while to understand. However, the horse race is not the main theme of page 2. The main theme is that you should not believe those results. The headline reads, “Only 29% believe the gap between Tsai and Han is large.”

[Begin rant. The inset story reinforces this headline by quoting people who feel that the gap shouldn’t be that big. Who cares what the numbers actually say? I feel that Han is doing better. Well, if you are just going to substitute your feelings for data, why bother collecting data? Even more, one scholar explains that people feel the race should be closer because they think Tsai is doing such a bad job. However, as we will see below, voters are currently relatively satisfied with Tsai’s performance. Never mind, someone feels she’s doing a bad job and she isn’t leading. End rant.]

Again, the headline is stretching the truth. The question was (apparently), “Concerning surveys showing the Han-Chang ticket trailing by a large margin, what is your opinion?” 29% thought that he was really losing by a large margin, 24% thought that he was trailing but not by as much as media reports said, 22% thought that surveys are inaccurate and he was not trailing, and 25% did not know. What exactly does that 24% mean? If they have just seen a survey that says Tsai is leading by 35% and they think Han’s attempt to undermine polling has had a small but real effect so that maybe Tsai is “only” leading by 28%, they could be in that middle category. That is, they could easily believe that Tsai is leading Han by a lot. The percentage who think that the gap between Tsai and Han is “large” could be anywhere between 29% and 53%. Again, this question was set up to produce a headline palatable to their editors and readers rather than to faithfully report public opinion.

 

UDN produced another poll this week. Here are the first two pages from Dec 16.

By now, UDN has wholeheartedly swallowed the assumption that polls of the presidential race are meaningless. They don’t even bother to write a story or produce a graphic dedicated to the horse race. The result (Tsai 48, Han 22, Soong 9) is buried in the middle of the story halfway down page 2. It’s there, but you have to look really hard to find it. In essence, they have simply adopted the Han camp position that polls are no longer informative.

A good friend and frequent blue-leaning media pundit recently complained on his Facebook page that it was irresponsible to interpret a poll without taking Han’s ploy to undermine surveys. He is correct. However, it is just as irresponsible to go too far in the opposite direction and dismiss polls altogether. Han has muddied the waters, but polls still convey useful information. Other pollsters are trying to figure out how much Han’s ploy matters. Look back at the Dec 17 issue of Apple Daily. On the bottom right corner, they produce an adjustment for latent Han supporters. You might argue that their adjustment is wrong, but at least they are trying.

 

Public intellectuals complain all the time that the media focuses too much on the horse race and not enough on substantive questions. So what substantive concerns are the UDN emphasizing? The headline screams that fewer than 30% of people think that Taiwan is better off under DPP government. Wow, Tsai must be doing a terrible job.

Or maybe not. Let’s look at their evidence. They asked something like, “The DPP has had complete control of government for over three years. During that period, have ordinary people’s lives gotten better, gotten worse, or stayed about the same?” “In the same period, has Taiwan’s overall development gotten better, gotten worse, or stayed about the same?” For the first, 13% said better, 29% worse, and 49% same; for the second question, 23% said better, 30% worse, and 38% said the same.

These are good, neutral questions. The modal result is that things are about the same, but more people think things are worse than better. The headline is not unfactual, but it misleads readers by giving the impression of a binary question. If fewer than 30% think that things are getting better, does that mean that more than 70% think that things are getting worse? No, it does not.

Things getting better or worse is a meaningful question, but it is not the same as asking whether the DPP government is doing a good job. In the 2012 election, the DPP tried to argue that President Ma had failed miserably to fulfill his 633 promise (6% growth, 3% unemployment, USD30,000 per capita income), so he must have done a bad job as president. Of course, Ma had an easy answer: he had done a good job, but the external circumstances – the Global Financial Crisis – had made it impossible to meet the 633 promise. By re-electing Ma, voters effectively agreed with the proposition that, even though things had gotten worse, Ma had done a reasonably good job.

Voters seem to be making a similar judgment about Tsai’s performance. Even if most people either think things are about the same or worse, many of those people think she has done a good job. The line chart on page 1 shows changes in Tsai’s approval rating over her term, and she is now at 48% satisfied. Holy Crap!! UDN has (intentionally) buried the lede!! She was 43 points underwater just after last year’s election, but she is 10 points above water now. Maybe she’s not doing such a terrible job after all.

The text tries to minimize Tsai’s positive approval rating by saying that of the people who approve of Tsai’s performance, 43% couldn’t name a specific policy that they are satisfied with. Of people who did give an answer, 15% said pension reform, 10% said maintaining Taiwan’s sovereignty, and 7% cited the legalization of same-sex marriage. (These response categories were unprompted, which is one reason many people did not name anything.) I’m not surprised by sovereignty, but I am a bit surprised that pensions and marriage equality were so high on the list.

 

On page 2, UDN continues to try to paint a cynical picture of Taiwan’s democracy. The headline reads, “Most voters don’t believe Tsai and Han will fulfill their campaign promises.” That headline is a bit problematic, but we’ll get to that later.

Before that, we have to talk about the graphic. This is what finally broke me. The top part of the graphic says that it is about using policies to buy votes and recklessly making campaign promises. The first line reads, “the government’s increases of stipends and subsidies,” while the second line reads, “making reckless campaign promises.” For the former, 25% say it is very serious, 21% say somewhat serious, and 29% say not serious. For the latter, 42% say very serious, 31% say somewhat serious, and 11% say not serious. There are several problems. First, the response categories are unbalanced. If you have three categories, you need one positive, one negative, and one neutral. If you provide two negative response categories, you need to balance them with two positive response categories. This is basic survey methodology, and everyone at UDN from the person in charge of the survey unit all the way up to the editor-in-chief knows this. Second, the use of the word “serious” presumes that something is wrong. What if you think that subsidies have been increased a lot and that this is fantastic? Third, the question asks about increases in subsidies and stipends, but the headlines and text have transformed this into “policy vote buying.” Survey respondents were never asked about “policy vote buying.”  Fourth, the way the graphic is put together, the first question is about something the government is doing, which makes it look like the second question is also about something the government is doing. However, the text clarifies that respondents were asked whether candidates had made reckless campaign promises. That is, it refers to ANY candidates, not necessarily the candidate representing the government or even most candidates. A response of “serious” is not necessarily an indictment of the government. In fact, if you keep reading the fine print, it isn’t at all. For people who responded that the problem was serious, they then asked which candidate was the worst offender. 51% named Han, 26% said Soong, and 1% replied Tsai. (Only 1%!!!) The UDN produced these questions and this graphic to give the impression that Tsai is spending and promising recklessly in order to get re-elected. In fact, a closer look at the data suggests that voters mostly think she is acting responsibly. Han is the one making crazy promises.

The main headline is taken from the bottom half of the graphic. They asked, if Tsai is re-elected, do you believe she will fulfill her campaign promises. They then asked the same question for Han. For Tsai, 43% said they believed she would, 40% said the did not believe she would, and 17% didn’t know. For Han, 23% believed, 59% did not believe, and 18% didn’t know. Recall that the headline proclaimed that most people do not believe the candidates will fulfill their promises. For Han, this is accurate. For Tsai, however, this requires manipulating the words a bit. To get to “most,” you have to lump together all the categories other than “believe” and call them “not believe.” However, there was already a category explicitly labeled “not believe,” and this category was far below 50%. Given the pattern of reporting, I’m not willing to give them the benefit of the doubt; I have to assume this was an intentional slight of hand to make Tsai look just as bad as Han. In fact, more people believed than disbelieved Tsai would fulfill her promises, while over twice as many disbelieved Han as believed him. They are not equally bad; Han is the one with a credibility problem.

 

I understand that UDN has a partisan line and its readers want things presented in a partisan way. However, UDN has a fundamental responsibility to uphold basic media ethics. The construction of these survey questions and the reporting of the results goes right up to the edge of that ethical line and then slips over it a few times. The headlines, in particular, are consistently designed to mislead. I’m also quite offended by UDN’s failure to use balanced response categories. UDN likes to say that it is the paper of record in Taiwan, with a stature equivalent to the New York Times in the USA. If it aspires to such a lofty standard, it must do a better job prioritizing its duty to report facts neutrally over its partisan preferences.