Local regression fit and residuals for presidential approval. The horizontal lines mark the 95% confidence interval for the residuals. Points in red are the Diageo/Hotline polls, points in blue are ABC/Washington Post.
New polls from ABC News/Washington Post and Gallup/CNN/USAToday again present us with puzzling differences in the approval of President Bush. The 47/52% approval/disapproval rating the the ABC/WP poll is a 8% gain since their last poll, 10/30-11/2, (39/60%) which was before the White House launched its aggressive response to critics on November 11.
In contrast, the new Gallup poll finds a 41/56% approve/disapprove rate, which happens to be the same as the 41/56% results in their 10/28-30, the Gallup poll closest to the dates of the earlier ABC/WP poll. The most recent Gallup poll, completed 12/9-11 had a 42/55% approval/disapproval reading.
This pair of polls resemble the contrast between last week's Diageo/Hotline poll (50/47%, 12/12-13, a +11% increase in approval) and the Fox News poll of 12/13-14 which found support stable at 42/51%. I discussed those results here, and showed that the Diageo/Hotline result was one of the most extreme outliers in my data since 2002. Now with the addition of another pair of polls, one with a substantial approval gain and the other with no gain at all, we again have to consider what is going on here.
Since November 11, my model predicts a gain in approval of +5.8% from the low point of approval following the Scooter Libby indictment and Harriet Miers nomination failure. With that gain, the model estimates approval at 42.5%-43.9% as of 12/18. Depending on how you estimate the model, the ABC/WP poll is either within the 95% confidence interval of this estimate (as in the graph above) or just above the upper margin of the confidence interval (as in the graph below).
The first point is to not focus on the change since the previous poll by ABC/WP, because that poll is over a month old. President Bush has been gaining support at a rate of +.15% per day since November 11, so we should certainly expect gains from polls taken before November 11. Rather the comparison should be between our best estimate of current approval and the individual poll result.
I've estimated the current approval model in two ways. One is with a local regression which flexibly fits the local trends in the approval data. That local fit, graphed above, produces a current estimate, as of 12/18, of 43.9%. That model produces a 95% margin of error of approximately +/-5%. By that standard, a poll as high as 43.9+5=48.9 would fall just on the upper limit of the confidence interval. At 47% approval, the ABC/WP poll falls easily within this margin of error. This leads to the conclusion (based on this model) that the new ABC/WP poll is not inconsistent with the estimate that approval is currently 43.9%. We should expect to see polls this high if approval really is 43.9% and need not conclude that approval has surged to 47%. Note that the same does not hold for the Diageo/Hotline poll, at 50% approval, which falls easily outside the 95% confidence interval. (But note also that with the addition of the new ABC/WP data, the Hotline poll is less of an outlier than it was on Friday because the ABC/WP result helps pull up the estimated approval at the time of the Hotline poll.)
The conclusion here, then, is that ABC/WP is not an outlier, and not inconsistent with the model estimate of 43.9% approval.
There are other ways we could estimate current approval. One of those is to use a linear model, rather than a local regression. The linear model is far less flexible, but also less likely to respond to a short term "bump" that doesn't really represent meaningful change. So for comparison, I estimate a linear model here. The data are fit only to the approval data since January 1, 2005. The model incorporates a linear decline in approval from January 1 until November 10. Step-function shocks for Katrina and Libby are also included. Starting November 11, I estimate a linear growth in approval as well as a positive step function to capture the new White House approach. This model has previously been found to fit the data reasonably well, and it continues to do so with the addition of the post-November 11 data.
The results of this model are plotted in the figure below. There has been a clear and statistically significant increase in approval since November 11, and especially the upward trend is positive and significantly so. Based on this linear model, approval on 12/18 is estimated at 42.5%. This model also produces a somewhat tighter margin of error, of approximately +/- 4.3%. By this standard, the ABC/WP data at 47% falls just above the upper limit of the confidence interval, 46.8, as plotted in the figure. Combined with the results from the local regression model, this makes me view the ABC/WP results with some serious doubt. But the ABC/WP is not nearly as extreme an outlier as the Diageo/Hotline result, which falls clearly outside the confidence interval in both figures.
An important aspect of these graphs is the plotting of previous polls by both of these organizations. The Diageo/Hotline polls are in red, the ABC/WP polls in blue. In both figures, these red and blue points track the trend lines in the top part of the figures quite well. In the lower residual plots, all the previous polls by both organizations fall within the 95% confidence interval. This is important because it puts these latest polls in perspective. There is absolutely no evidence from this analysis that either of the polls is systematically out of line with other polling organizations. That these last polls by both are relatively large outliers in no way suggests that the polls are seriously flawed in any way.
What it does raise is two points: first, even excellent polls can produce an individual outlier on an individual question. Partisans are quick to seize on a single example to impugn the entire polling enterprise (and quick to reverse themselves if the next poll by that organization is more to their liking.) Second, we need not be helpless in the face of unusual data. If you believe in "just averaging" the polls, then you will include extreme outliers and as a result bias your estimate of approval upwards, in this case. If you look at the polls in the context of the margin of error you can (and I would) discount the value of a single outlier. My estimate here of 42.5% approval is based on a model that discounts the weight given to large outliers.
One other important issue is that any model (mine included) can be misled in times of highly dynamic change. It is conceivable that we are currently in a period of very rapid change in presidential approval. The upward trend since November 11 is 5 times as rapid as the downward trend in the first 10 months of the year. So we may yet see new polling that looks closer to the results of the Hotline and ABC/WP polls. But if we do, we will have a new problem: how to explain the larger number of polls that have missed that large upturn. I'm still putting my money on the significant upturn to about 42-43%, and holding off on bets in the high 40s. For now.
Related Posts:
Spacing of polls confounds the lede.
How out of line is the new Hotline Poll?
Approval of President Bush in 2005
Bush Approval, 2001-present
Approval in historical perspective
