Since June 1 there have been 10 new polls released. As luck would have it, five were taken before and five after the killing of Iraqi al-Quada leader Zarqawi. The data continue to support an upturn in approval of President Bush (see the blue line in the top left figure). But what have the news reports had to say about these polls? How do the statistical results get described and interpreted by the press and by those who conduct the polls?
I was prompted to look at this by what seemed to be a cacophony of conflicting headlines. You can read about 1500 of them by following this link to search results from Google News. Depending on the headline you find that either approval is up or down or unchanged. Thanks.
The fact is that there is some apparent conflict among the polls taken in the first two weeks of June. Two showed slight declines from previous polls in presidential approval (CBS, Zogby), five showed gains of 1 or 2 percentage points (Cook, NBC/WSJ, CNN, USAToday/Gallup, and AP) and three found gains of 3-5 percentage points (USAToday/Gallup, Harris and Fox). This has provided fodder for those who wish to argue for "no real change, just a blip" and for those who argue "Bush is back big time."
Yet of the 10 polls, only two actually found a statistically significant change from the previous poll by the same survey organization. Fox's 5 point gain was highly significant (p=.028), and Harris' 4 point gain just barely failed to reach significance (p=.053) but I'll give it credit by rounding down. The p-values for all the rest were not close to statistical significance: .64, .64, .16, .35, .45, .34, .63 and .65. So IF ONE ONLY COMPARES adjacent polls by the same pollster, eight of these ten organizations should have reported "no change". (See my post here on the perils of this approach to measuring change.)
The IF here is important. In all but one of the initial accounts of these polls, the news organization or pollster refers ONLY to previous polls by the same organization. In characterizing change, therefore, they act as if there were no other evidence for either a rise, fall or stable approval level. If one takes that seriously as a statistical standard, then the tests above are the appropriate measure of whether the changes are easily explained as random variation or if there is enough evidence of change to reject random noise as the explanation. In 8 of the 10 cases, the conclusion should have been "no change."
Here are the ways in which the organizations characterized these polls, sorted in descending order of change. These are brief but direct quotes, not paraphrases. Fuller quotes and links to entire stories follows at the end of this post.
+5 Fox: "out of the thirties", "up 5 percentage points to 40"
+4 Harris: "Rises Slightly", "risen from 29 percent in May to 33"
+3 USAToday/Gallup (6/1-4): "For the second straight time ... edged higher", "now five percentage points above its low"
+2 USAToday/Gallup (6/9-11): "Poll sees a boost for Bush", "going up to 38% from 36%"
+2 AP: "35 percent, essentially unchanged from his rating of 33 percent last month"
+1 CNN: "hasn't budged", "virtually the same", "However ... has risen since late April"
+1 NBC/WSJ: "slightly improved", "remains below 40", "baby bounce", "changed next to nothing", "slight tick up"
+1 Cook: (No direct characterization of approval trend.)
-1 Zogby: "we see him at 31%" ... "lowest we have ever had for him", "coalition of supporters ... is crumbling."
-2 CBS: "done little to ... boost President Bush's approval ratings", "Zarqawi's killing hasn't helped" "remains just 33 percent-- down slightly from 35 percent"
There are several interesting comparisons here. First, the tone of the characterization is generally cautious: "slightly" and "edges" for gains of 4 or 3 points. Fox, with the strongest statistical evidence, actually avoided adjectives of magnitude in favor of "out of the thirties" and "up 5 percentage points", both descriptively accurate.
USAToday/Gallup and AP seem to differ in how big a 2 point gain is. USAToday/Gallup: "Poll sees boost", AP: "essentially unchanged". Neither is close to statistically significant, but the implication of the descriptions are quite different.
CNN and NBC/WSJ characterize their 1 point increases in minimalist terms: "hasn't budged", "virtually the same", "slightly improved", "changed next to nothing". In contrast Zogby reads a one point decline as "lowest we have ever had him", which is literally correct, if not statistically significantly different from the previous Zogby reading.
CBS, with a 2 point decline, minimizes the magnitude of change with "down slightly".
In the full articles, some of these use comparisons to earlier polling to point out that approval has risen more compared to earlier in the spring, again making comparisons only with their own polling. That context is important. However it isn't enough.
And so, if we read the stories, we should come away with an impression of only a "slight rise" or an "edging up", AT MOST. We would also be justified in concluding that there has been only minimal increase or a loss if we read the words of five of the poll descriptions. Four would make us believe there is some upward movement (though two of these four are not statistically significant changes.) In short, ambiguous evidence of at most small gains.
And yet, the evidence, when pooled across all polls throughout the year, is quite unequivocal: approval started moving up after May 15. The blue estimated approval trend in the top left plot marks a very sharp turnaround after that date, and estimates a gain of about 2.3 points in the past month. The strength of this analysis is that it draws on ALL the available polls, rather than make pair-wise comparisons with previous polls of the same company. This also recognizes the substantial amount of random variation around the trend, reminding us that individual polls are subject to quite a bit of variation that has nothing to do with the "true" level of presidential approval. Noise happens, but when we estimate the trend across all the polls we get better, more precise estimates of approval.
In part, the news organizations are handicapped by their sponsorship of polling. CNN wants to quote only CNN polls, and CBS acts as if NBC didn't do polling. There are obvious business reasons for doing so, but it limits the writing reporters can do. This is especially true because the reporters are, of course, quite aware of what other polls are showing. Some are even aware of what my blue trend line is estimating approval to be. Yet they are constrained to write "what the new poll shows" without the benefit of putting it into the context of what other polling shows. The result is that reporters either literally report the poll's results without the more helpful broader perspective, or they color their choice of adjectives in light of other evidence, which remains uncited. (In very rare cases, they actually do point to other trends, or more commonly interview pollsters who can be quoted providing the context.)
Yet on the whole, I think the characterizations we find in these 10 polls largely misses the big story of mid-May to mid-June, 2006. After three and a half months of continuous decline, President Bush has constructed a sudden and clear upturn in public support. That should be the story here. And, in fact, political reporters NOT constrained to write about the polls, have been saying that loudly for the last week or so. (And see the Charlie Cook quote below, which neatly captures the story.) But not so much with the poll stories. Ironically the best evidence for the upturn is the aggregation of those poll results, even though individually the poll stories only weakly suggest an improvement.
I suppose I should be grateful for this commercial limitation on news organizations' ability to write about other companies' polls. It is exactly the niche that PoliticalArithmetik fills, as does the polling collected by RealClearPolitics and PollingReport. But having said that, news organizations need to find ways to provide better perspective and analysis of trends that are clearly visible, if only when aggregated across organizations.
Here are the fuller quotes describing each poll, and links to the full stories. I've tried to locate the organization's OWN description of their results, rather than news stories written by others about the results. Again, I follow descending order of changes from previous polls.
Fox (6/13-14/06 poll): Published 6/15/06
06/15/06 FOX News Poll: Victories in Iraq, Bush Approval Up
... President George W. Bush’s job approval rating is out of the thirties for the first time since February and currently stands at 40 percent. ...
President Bush’s approval rating is up 5 percentage points to 40 percent this month, up from 35 percent approval in mid-May. Just over half of Americans disapprove (52 percent).
Much of the increase in support comes from the president’s party. Fully 82 percent of Republicans say they approve of Bush’s job performance, up from 71 percent last month and a low of 66 percent in April. Overall, the president hit a record low of 33 percent approval in April 18-19.
Harris (6/2-5/06 poll): Published 6/9/06
President Bush’s Approval Rating Rises Slightly from All-Time Low
Just over one-quarter of U.S. adults believe the country is going in the right direction
After reaching an all-time low last month, President Bush’s job approval rating has risen from 29 percent in May to 33 percent this month. Two-thirds of U.S. adults have a negative view of the President’s job performance.
USAToday/Gallup (6/1-4 poll): Published 6/6/06
Note: This was published in the "On Deadline" blog. Google did not find a full news story based on this poll.
Bush's approval rating notches second rise
For the second straight time, President Bush's approval rating has edged higher in the USA TODAY/Gallup poll. It is now five percentage points above its low.
The latest reading: 36% of the 1,002 adults polled over the weekend said they approve of the way he is "handling his job as president." That's up from 33% who said that in a May 8-11 poll and 31% - Bush's low in the USA TODAY/Gallup poll - in a May 5-7 survey.
The surveys each have margins of error of +/- three percentage points. At 36%, Bush's approval rating has now edged higher than its possible "range" of 28%-34% in the May 5-7 survey. (Of course, there is a chance - given the margin of error - that his current approval rating is 33%. But Gallup's pollsters do not believe that is likely.)
USAToday/Gallup (6/9-11 poll): Published 6/13/06
Poll sees a boost for Bush, Iraq war
The survey, taken Friday to Sunday and released Monday, also showed Bush's approval rating going up to 38% from 36% earlier this month and an all-time low of 31% in May.
AP (6/5-7/06 poll): Published 6/9/06
Poll: Many Support Troops, but Not the War
The survey also found that the war continues to take a toll on the public's opinion of Bush. Approval of the president was at 35 percent, essentially unchanged from his rating of 33 percent last month based on the poll's margin of error of 3 percentage points.
CNN (6/14-15/06 poll): Published 6/16/06
Poll: Bush's unpopularity could hurt GOP candidates
President Bush's continued unpopularity could present a challenge for Republican candidates in this fall's mid-term elections, according to a new CNN poll released Friday.
The poll results weren't all bad news for Bush. After his surprise visit to Baghdad, Americans were slightly less critical of his handling of the war in Iraq, but his overall job approval rating hasn't budged, the poll shows.
...
In the poll, conducted on Wednesday and Thursday by Opinion Research Corp., the president's overall job approval rating was at 37 percent. In May, before his trip to Baghdad, it was virtually the same: 36 percent.
However, Bush's approval number has risen since late April, when it was just 32 percent.
NBC/WSJ (6/9-12/06 poll): Published 6/15/06
Bush’s standing rises slightly
After a week of positive news for the White House-- including the death of terrorist leader Abu Musab al-Zarqawi and the formation of a new government in Iraq-- the latest NBC News/Wall Street Journal poll finds that President Bush's standing has slightly improved, especially when it
comes to the situation in Iraq.
...
Democratic pollster Peter D. Hart, who conducted this survey with Republican pollster Bill McInturff, calls the results a "baby bounce" for Bush. "The death of Zarqawi may have improved attitudes about ... the war," he says. "But I think it changed next to nothing toward the overall attitudes about the president ... and the upcoming elections."
McInturff agrees. "You can see a slight tick up. But you don't see any influence on Bush as of yet."
According to the poll, 37 percent approve of Bush's job performance-- an increase of one point since the last survey in April. This is the seventh straight NBC/Journal poll that has had Bush's job approval below 40 percent.
Cook Political Report/RT Strategies (6/1-4/06 poll): Published 6/16/06
(Note: Cook did not directly write about presidential approval based on the Cook/RT poll other than to report the topline results. A subsequent power point presentation of the results showed the trend but commented on the overall negativity rather than the 2% rise in approval. Cook's characterization of the state of approval is not based solely on the Cook/RT Strategies poll, which is probably why it is such a good analysis.)
Pride Before The Fall?
Perhaps the irony in all of this Democratic victory clucking is that it is happening at a time when the president's freefall in the polls has stopped and he has even regained a couple of points.
It was just a few weeks ago that most polls showed job-approval ratings in the low 30s; now most are in the 35 to 38 percent range. While these numbers are still terrible, they are no longer dropping.
Zogby (6/2-6/06 poll): Published 6/9/06
Poll: "Had enough?" Not Enough to Lift Congressional Dems to Power
"Clearly a weakness that the Republicans have is President Bush. In this poll, when we look at President Bush, not only do we see him at 31% positive overall job performance rating, which is lowest we have ever had for him, but we also see the coalition of supporters who supported his election and reelection is crumbling. He wins just a 48% positive job approval among born-again Christians, but we also see lows among conservatives, Republicans, veterans, WalMart shoppers, and among NASCAR fans. Right now, the President is a burden for the Republican Party heading into the midterm elections."
CBS (6/10-11/06 poll): Published 6/12/06
Poll: Bush's Ratings Remain Low
The death of Abu Musab al-Zarqawi has done little to improve views of how things are going for the U.S. in Iraq or boost President Bush's approval ratings, a CBS News poll finds.
Zarqawi's killing hasn't helped President Bush with the public, either. His overall job approval rating remains just 33 percent-- down slightly from 35 percent last month-- while 60 percent disapprove.
------------------------------------------
Click here to go to Table of Contents
