Bush Approval: 3 New Polls, Trend at 32.6%

There have been three new polls since my last update of approval of President Bush. The USA Today/Gallup poll done 8/13-16/07 found approval at 32%, disapproval at 63%. A Pew Research poll taken 8/1-18/07 got approval at 31%, disapproval at 59%, and the Fox News poll done 8/21-22/07 put approval at 33%, disapproval at 56%.

With these three polls, the trend estimate is 32.6%, a small continued gain over the previous batch of polling.

The plot below shows the six most recent polls in relation to the trend estimate.

























None of the recent polls qualify as outliers, with all comfortably within the 95% confidence interval.

The Effect of ARG Polling on Iowa Trends

American Research Group (ARG) does a large amount of state primary polling and is therefore potentially influential in estimating candidate support because they contribute more polls than most other organizations. This week we saw conflicting results from ARG and Time/SRBI polls of Iowa. (See Mark Blumenthal's analysis here.) The discrepancy of ARG polls from others in Iowa has been an issue here before, as has been the question of how much any single poll influences our trend estimates. Today we take another step towards systematically answering that question.

In the Democratic race, ARG has consistently found support for Clinton well above that of other polling organizations. In the chart above, ARG polls are in purple, the blue line is the trend estimated with all polls, including ARG, while the red line is the trend estimate without ARG. The light blue points are all non-ARG polls, while the purple points are the ARG polls.

This lets us compare three things: ARG polls to other polls, ARG polls to the trend, and the trend with ARG to the trend without ARG.

In the case of Clinton, ARG polls are consistently far above the results of other polls. This has been widely remarked upon already. And in the Clinton case, the ARG polls have shown some decline in support in Iowa, while other polls have shown an increase in her support. This is also the case in which ARG exerts a significant influence on the trend estimator. The blue trend line (with ARG included) is well above the red trend estimate which excludes ARG. This was especially true early in 2007 when there were few polls and several from ARG, giving them an extra influence due to lack of non-ARG data. As polling frequency has increased the two trend estimates have converged, but the non-ARG estimate remains a couple of points below the overall trend.

Blumenthal has talked about possible reasons for this, and I encourage you to see his post here.
I'm more concerned with the magnitude of difference and their effects here, so will leave it to Mark to explain the "why".

It is clear that ARG's estimates for Clinton have consistently been out of line with others, and that this has had an effect on my trend estimates, making Clinton appear more competitive in the first half of 2007.

But let's also look at the other candidates. ARG is less consistent in over- or under-estimating Edwards' support. Some ARG polls have put Edwards below trend, but others have him above trend. While ARG has disagreed with other pollsters in individual polls, the effect of ARG on the trend estimate for Edwards is negligible.

On the other hand, ARG has consistently had Obama below the support found in other polls, and well below the trend estimate. Despite this, the effect of ARG on the trend estimates has been small for Obama, with the blue and red trend estimates consistently quite close to one another.

Finally, Richardson has been a bit underestimated by ARG, but again with little influence on the trend estimates.

Bottom line: ARG has had a substantial effect on the Clinton trend estimate until recently. Still, the substantive effect is not trivial. Estimates including ARG put the trend at 26.2% for Clinton, 24.2% for Edwards, a Clinton lead of 2.0 points. But excluding ARG from the trends we get Clinton at 24.6% and Edwards at 25.9%, a 1.3 point Edwards lead. Of course both estimates say the race is close in Iowa, and perhaps we should stop there. But the consistent ARG overestimate of Clinton has influenced perceptions and estimates for this race.

If we switch to the Republican side, there is a consistent ARG overestimate of McCain support until very recently. ARG is also a bit high on Giuliani and a bit low on Romney. The Thompson numbers are relatively few and jump around.
























Unlike the case of Clinton, the trend estimates are not much affected by the ARG data. The blue and red trend estimates lie very close to one another for all four Republican candidates, despite the high ARG readings for McCain.

There are two bottom lines here. Any pollster can experience consistent house effects that lead to over- or under-estimating support for some candidate. These may be due to sampling methods, filtering for likely voters, question wording or order, weighting methods, or perhaps to mysterious gremlins. ARG is an example of house effects, at least for Clinton and McCain and probably Obama. House effects are important because they give us a way of estimating what a poll would be if we adjust for those house effects. That gives better perspective than the raw numbers might. But house effects also allow us to say which polls are more in line and which more out of line with others. A house effect is not in and of itself evidence for bad polling methodology. There may be good reasons for choices that lead to significant house effects-- for example deciding to interview likely voters rather than adults or a decision not to push undecided voters or to push them for a preference. So we should be careful here in how we interpret the results. That said, it is crucial to know which organizations are consistently high or low for candidates (or any other variable.) The ARG lines in the figures above give a clear reading of that for the Iowa polling.

In the next few days we'll be rolling out a series of posts that look at house effects for all polling organizations across state and national polling. We'll have a systematic look at this, with estimates of the effects for each organization. I hope that will help clarify things.

The second bottom line point is that the trend estimates are pretty resistant to the effect of a single polling organization when there are plenty of other polls taken around the sample period, but that, as in the case of Clinton and ARG, this effect can be quite a bit larger when polling is sparse and a single organization contributes a substantial share of the polls while at the same time exhibiting a significant house effect. In one sense this problem goes away as we approach elections because the density of polling increases as does the heterogeneity of polling organizations. But as Iowa illustrates (and we'll see again in other primary states with limited polling) it is not always possible to be sure which polls are misleading us when the evidence is limited.

Stay tuned next week for the next step in examining the house effects in primary polling.

Bush Approval: 5 New Polls, Trend at 32.4%

I've been off-line the past week, recovering from vacation. But the polls haven't stopped, and I'm now back to work as well.

We saw five new polls last week on approval of President Bush. A CNN/ORC poll taken 8/6-8/07 found approval at 36%, disapproval at 61%. The Associated Press/Ipsos poll also on 8/6-8/07 has approval at 35%, disapproval at 62%. The Reuters/Zogby poll done 8/9-11/07 got approval at 32%, disapproval at 67%. The CBS News poll done 8/8-12/07 put approval at 29%, disapproval at 65% and the Quinnipiac University survey of 8/7-13/07 finds approval at 29% and disapproval at 64%.

With these new polls the trend estimate of approval now stands at 32.4%. At the moment none of the last 10 polls qualify as outliers, with some above and some below the trend estimate but none further away than we would expect given the variability in polling. It remains to be seen whether the current upturn in approval that began after June 28 is continuing up. The Quinnipiac and CBS results at 29% are more than 3 points below trend, but not abnormally so and both polls typically run a bit below trend, so they don't give much evidence of a new downturn. Let's get some more data to answer that question.

Clinton vs. Obama: Dueling Memos and Some Data

The Clinton and Obama campaigns released dueling memos yesterday, each touting their standing in some polls and dismissing evidence to the contrary. The Clinton memo is here and the Obama memo here thanks to our friends at MSNBC. Both memos, not surprisingly, selectively interpret the polls to their advantage. But let's look at the data and see who has the better evidence at this point.

The national polls have shown a long period of stability. Obama enjoyed a nice rise in the polls after he became a candidate following the 2006 election. He had a rapid rise to 23% by April 1 and it was nearly impossible to read a news article about him during this period without encountering the phrase "rock star". But perhaps he was a "one-hit-wonder" because since April 1 there has been no further upward movement in his national support. If anything there has been a negligible decline to a current estimated support of 22.6%. That has put him solidly in second place since November, but he has failed to close the gap with Clinton since April.

The Clinton campaign also experienced a long period in the doldrums. After entering 2006 at about 37% support, Clinton declined slightly to 35% just after the November elections. And there she sat until May. There were hints of tiny increases and tiny declines, but she remained well within a point of 35% during the spring. This stability could be interpreted as evidence of strength because her support was sustained throughout the period of Obama's rise. I think this reflects the power of Clinton's hold on her core supporters. At the same time, during this period of ramping up of the campaign there was no detectable improvement in Clinton's standing with Democratic primary voters.

That began to change by early June and has accelerated a bit since. My best estimate of Clinton's current support is 38.8%, a rise of nearly 4 points since the end of April. That four point rise won't sound like much to those accustomed to the noisy variation from poll to poll, but the trend estimator I use has the advantage of aggregating across many polls and hence has a much smaller range of random variability. A move of this much is certainly not negligible.

The blue line in the figure is my standard trend estimator. The red line is more sensitive to short term trends, but also more easily fooled by random noise. I provide both so you can judge for yourself any differences between the more solid blue estimator and the more rapid change of the red estimate. In this case, the red estimator is in complete agreement with the blue for Obama, but suggests a slightly greater surge for Clinton recently.

Of the 90 national polls included in my data, Clinton has led Obama in 89. But the more important point is that gap has not closed since April 1, and since May the gap has widened a bit with Clinton's move up and Obama's stagnant polling.

The Obama memo characterizes the " irrelevant and wildly inconsistent national polls" as meaningless. As the figure above makes clear, there is indeed considerable variation across national polls, but the story they tell is not inconsistent except when cherry picking results. The polling varies in about the random pattern around the trend that we would expect from surveys based on probability samples of the electorate.

On the other hand, the Obama memo is quite correct that nomination races are about performance in individual states, not national polls. The early primaries have carried great weight in the modern period since the reform of the primary system in 1972. There is great debate this year about whether this will continue or if the massive February 5th primary day will fundamentally alter the traditional dynamics. But everyone agrees that it is better to do well in the pre-February 5 primaries than to do badly in them. So let's see where the Democratic campaign stands in the first five caucus and primary states.











The available state polling substantially agrees with the national polling in putting Clinton ahead of Obama in all five of the first states, based on my trend estimate that incorporates all polls. Moreover, Clinton's support is stable or rising in all five states, while Obama has risen substantially in only South Carolina, and perhaps a bit in New Hampshire. In Iowa Obama's support has clearly fallen off, while Nevada and Florida appear essentially flat.

The important caveat here is that state polling is more limited than national, and that the numerous polling organizations vary in their results with some sometimes substantial "house effects". Either campaign can (and both do) select individual polls in the states to make the case for their improving standing. But this is pure cherry picking of data. With the variability clear in the plot, it is easy enough to pick the poll that puts your candidate at the highest mark while finding one that minimizes your opponent's standing. That is the reason we use all the data here and let the trend estimates fall where they may.

So where does this leave the race? A clear Clinton advantage and strong evidence of some recent improvement in both national and state polls. For Obama, there is no comparable upturn nationwide and the picture in the states is mixed at best. The Clinton upturn is of interest because some argued that her support was solid but had little or no upside. "Everyone" has already decided about Clinton, this line of argument goes, and so while her base is rock solid she is vulnerable to a coalition of "anybody but Clinton" voters. That vulnerability remains a real liability, but the recent upturn suggests the upside for her is not as limited as some analysis suggested.

The bright side for Obama is that he still has a considerable upside in public awareness and in favorability, an area where Clinton does indeed seem in some peril among the general electorate. Obama also has very impressive fundraising success which indicates support among more engaged partisans.

The critical question is what happens to the roughly 60% of Democratic voters who currently do not support Clinton. Can they be won over or can Obama (or someone else) become the focus of an "anybody but Clinton" coalition? Until that dynamic is sorted out, and until some candidate other than Clinton starts to move up (none, so far) the advantage has to go to Clinton.

Bush Approval: Gallup/USAT 34%, Trend 31.4%

A new Gallup/USAToday poll taken 8/3-5/07 finds approval of President Bush at 34%, disapproval at 62%. This is in line with recent improvement in the President's approval rating across a number of polls. The current trend estimate now stands at 31.4%.

For a full discussion of the current upturn in approval see here, here, here and here.

The more sensitive red line estimator and my standard more reliable blue line estimator are in good agreement now about when the low point of approval occurred, though not yet agreed on how low the low point of approval was. The current estimates of approval are reasonably close, as is the agreement on the current upward trend.

Bush Approval: Newsweek 29%, Trend 30.7%

It has been a busy week for presidential approval polls. Newsweek is the latest to report. Their survey taken 8/1/07 finds approval at 29% and disapproval at 63%. This is Newsweek's second consecutive poll at 29%, but up from the two previous Newsweek polls both at 26%.

With the new data the approval trend estimate stands at 30.7%.

None of the diagnostics change significantly since the previous approval updates this week here and here, which offer more discussion of recent trends and the timing of the recent upturn in approval.

Bush Approval: DemoCorps 36-Pew 29-Trend 30.8

A new Greenberg Quinlan Rosner/Democracy Corps poll, taken 7/25-29/07 finds approval of President Bush at 36%, with disapproval at 60%. The new Pew Center poll, also taken 7/25-29/07 puts approval at 29%, disapproval at 61%.

With these two new polls the trend estimate of approval now stands at 30.8%.

The GQR/Democracy Corps poll is a statistical outlier, falling above the 95% confidence interval around the approval trend. At the moment we have two outliers-- ARG's 25% is below the confidence interval.

GQR/Democracy Corps polls are of likely voters, rather than adults as with most polls here. As a result, they consistently estimate approval levels higher than for the adult population and hence above the trend line as well. This is clear from the plot below showing "Greenberg" surveys tracking high. This is a "design decision" to survey likely voters, and should not be considered a defect in the poll. Inferences are to a different population from that of the general population.
























The "house effect" estimate for GQR/Democracy Corps surveys is just below 3 percentage points, with a confidence interval from just under 2 to just under 4 points. Despite this substantial house effect, GQR surveys have only rarely exceeded the confidence interval for presidential approval.

As the number of recent polls has increased, the standard "blue line" estimator of support has increased it's "bend" and is coming closer to the more sensitive "red line" estimator, further evidence that approval trends changed direction around June 28, when the red estimator put approval at 28.7%. Currently the blue estimator agrees with that date as the turning point, though the estimate will continue to change until enough data are available for a stable estimate of the turning point.

While the change in support is now pretty clear, the reasons are less so. The timing coincides with a series of Supreme Court decisions, all carried by the majority created with Bush's appointments of Roberts and Alito. I don't think the Court is salient enough with the general population to be a strong driver of public opinion, especially of presidential approval. However, it is possible that the most politically involved conservatives were both aware of the decisions and gave Bush credit for his appointments, helping arrest his decline in the polls.

The other event about this time was the commutation of Scooter Libby's prison sentence on July 2. Initial reaction among the general population and even many self-described conservatives was disapproving of this decision. But here too the effect may have been positive among conservative Republicans and served to shore up support among that constituency.

But that said, unlike previous sharp turning points that have corresponded to major presidential speeches, this one is harder to account for with "obvious" actions of the President (or of Congress, for that matter.) I invite your speculation on this.

Bush Approval: Two New polls, Trend at 30.2%

New polls from Hotline and NBC/WSJ continue to raise the trend estimate for approval of President Bush, taking the estimate above 30% for the first time in a while.

Hotline, taken 7/19-22/07 has approval at 33%, disapproval at 63%. The NBC/Wall Street Journal poll done 7/27-30/07 has approval at 31 and disapproval at 63%.

With these new data, the approval trend now stands at 30.2%. This is in keeping with the upturn in approval first noted here, and confirmed by recent polls. Of course, 30.2% is nothing to brag about, but it is a tad better than being in the 20s.

The exact timing of the change remains unclear from my standard estimator, but the more sensitive "red" estimator thinks the turn was sharp and the rise has been substantial. Remember, "red" is often fooled by a few polls, so retain some skepticism about the red estimator, but we've got enough recent polls to make me pretty confident "red" has the story right, if not the exact timing or extent of the rise.

With the new trend estimate, the ARG poll at 25% has now moved clearly into outlier territory. This is a good example of how determining what is or is not an outlier can shift as more complete data arrive.

Does One Bad Pollster Spoil the Trend Estimate?

Yesterday, in response to this post at Pollster.com, readers raised a number of excellent questions about the effects of individual polls in our trend estimates of candidate support (and just about everything else here as well, including presidential and congressional approval, support for the war and more.) Much of the discussion was about how to detect and exclude "bad" polls, which is a topic that covers a huge range of issues including "house effects" (the tendency of polling organizations to poll consistently high or low on some questions), outliers (single polls that are far from the rest) and more. The discussion will provide fodder for a number of posts to come later this month as I review our methods and try to clarify these and other issues. So there is a lot to do. Consider this a down-payment on the rest.

To paraphrase one question: "Why not exclude a polling organization if it consistently produces results out of line with everyone else?"

We could approach this in several ways. For example, suppose a pollster was consistently 4 percentage points high but their polls moved in synch with the trend in all the other polls. Movement in those polls would tell you a lot about dynamics of opinion even if the pollster were "biased" by 4 points. If the bias were consistent, then we could just subtract 4 points and have an excellent estimate of the trend. A simple shift of the average poll result above or below the overall trend is not in and of itself a clinching argument for excluding a pollster. I'll come back to this issue in much more detail later in this series of posts.

A simpler and more direct way to approach the question is to ask what difference does it make if we do include all polls, rather than exclude supposedly "bad" ones? Of course we'd have a major problem if the trend estimator were quite sensitive to individual polls or all the polls by a particular pollster. Happily, this is an empirical question, so we can answer it. And we don't have to know which pollster is "bad" to begin with.

The plots above show the trend estimate, using our standard estimator, as the black line. This uses all the polls we have available for the national nomination contests in both parties. The light blue lines are trend estimates that result when I drop each of the 19 different polling organizations, one at a time. Though the lines are indistinguishable, there are 19 different blue ones for each candidate in the figures. If the impact of individual organizations on the trend estimate were large, some of these blue lines would diverge sharply from the black overall trend line and we'd be seriously concerned about those polls that were responsible for the divergent results.

But that isn't what actually happens. The blue lines all fall within +/- 1 percentage point of the overall trend estimate and the vast majority are within less than +/- 0.5 points. There is no evidence that excluding any single organization has more than a trivial effect on the estimated trend. This alone is strong evidence that whatever problems specific pollsters or individual polls may have, they do not seriously disturb the trend estimates we use here at Political Arithmetik and Pollster.com.

It is interesting that the variation around the top candidates in both parties, Clinton and Giuliani, is larger than it is among the third place candidates, Edwards and Romney, while variation for the middle candidates falls in between. This is a possible clue to one aspect of "house effects". One well known source of house effects is due to how hard the interviewer pushes for an answer. Some organizations now routinely find 20% or more unable or unwilling to pick a candidate. Other organizations have less than 5% failing to choose a candidate. Now imagine yourself asked to pick, but lacking an actual preference. When pushed, who do you most likely "settle" for in order to placate the interviewer? I'd bet on the best known names. If that were the case, we'd see the greater variation around Clinton and Giuliani substantially explained by differences in how hard pollsters push for answers on the vote preference question. This is one more topic for another day.

The fact that we find little effect on the trend estimate due to excluding each pollster could mean one or both of two things: either no pollster is biased or discrepant enough to actually raise a problem in the first place, or the trend estimator we are using is statistically robust enough that it resists the influence of unusual pollsters or polls. The second possibility is true by design. I've chosen an estimation method and designed the approach we take so that the trend estimator should be resistant to bias due to a single organization or a single poll. While it can be fooled under the right circumstances, those should be both rare and short lived, rather than common and long term.

We are not in a position today to reach a conclusion about the first possibility, that none of our pollsters are consistently out of line with the others. That could be, but it could also be that one or more pollsters are in fact out of step but that the estimator successfully resists their influence. To address this more interesting question, will require more work and a separate post (or series of posts). It does seem to me that there are clearly systematic differences across polling organizations. I've done a good many posts in the past on "house effects" and on individual outliers, and will do more of that in the coming weeks. But it also disturbs me that many complaints are hurled at specific polling organizations with little or no effort to support the claims empirically and systematically. That is a job we'll begin to undertake here as a route to clarifying what the actual evidence is for which polls are less "reliable" than others, and what exactly that means. Stay tuned.