Kadima support stable after first Sharon stroke UPDATED 1/5/06

This was posted before Prime Minister Sharon suffered his second and far more serious stroke on January 4. Clearly the political landscape will be dramatically changed because of this second event. I've considered taking this post down out of consideration for the emotional impact of Sharon's illness. I've decided not to, since my purpose here at PoliticalArithmetik is to document the dynamics of political events, and what comes next will certainly be of great interest to political analysts. As it stands, the original post below documents clearly the prospects of Sharon's Kadima party just before his second stroke. Subsequent events and polling will further illuminate the dynamics of Israeli politics in the wake of this stunning turn.

Support for Prime Minister Ariel Sharon's new Kadima party appears to have remained stable in the wake of the Prime Minister's December 18th stroke and brief hospitalization. In seven polls taken since Sharon's illness, Kadima continues to receive support sufficient for 39-40 seats in the Knesset. (Likud currently holds 40 seats.) As a new party, rooted primarily in Sharon's personal appeal, it is interesting that support has not wavered in the past 10 days. The "health issue" is likely to be raised explicitly in the coming elections, set for March 28. For an example of the possibilities, see this column from the Jerusalem Post.

Support for Kadima has risen since Sharon broke from Likud and established the new party on November 21. From initial support of 30-32 seats, Kadima grew to near 40 seats by the second week of December. Since that time there has been no evidence of a systematic trend in support, which has remained stable at about 40 seats.

The same cannot be said for Labor, behind new leader Amir Peretz, which has seen a substantial decline in support from some 25-26 seats to 19-20.

Meanwhile, Likud has seen a small upturn in support which has continued since the December 19 selection of former Prime Minister Benjamin Netanyahu as party leader. Immediately following Sharon's withdrawal, Likud polled around 13 seats but has since risen to 16 or 17. While a dramatic decline from the 40 seats held by Likud in the current Knesset, this modest
gain is the first hopeful news for Likud in the past six weeks.

The center Shinui party appears to have virtually collapsed. From a total of 15 seats and third largest party in the current Knesset, Shinui is currently polling as low as 4-5 seats.

If Kadima, Labor and Likud match their poll results, they should win 75-77 seats. Currently Likud, Labor and Shinui hold 76 seats. With Shinui at only 4-5 additional seats, this division of the vote suggests that there has been considerable shuffling among the these three parties plus the new Kadima party, but the other parties have retained most of their aggregate strength.

The Sephardic Shas party, currently the 4th largest bock in the Knesset with 11 seats, is polling near it's current level with 9-10 seats in most polls.

The remaining smaller parties are polling between 3-8 seats each, a total of some 30-35 seats divided among them. There have been no strong trends among this group. The latest results are

Arab parties, 8-9 seats
Yachad, 4-5 seats
National Union, 3-8 seats
Yisrael Beiteinu, 6-7 seats
Yahadut Hatorah, 5-6 seats
NRP, 3-4 seats

Spacing of polls confounds the lede

The above graph shows the expected change between polls in presidential approval given the current trend of +.15% per day and the average spacing between polls in 2005 by various polling organizations. The more frequent the polls, the less change on average between consecutive polls. The less frequent the polls, the more likely successive polls will "find" large changes in approval. This makes some polls appear more stable and others less even though the actual daily rate of change in approval is exactly the same for all polls. (What's a "lede"? Click here!)



Polling organizations vary dramatically in the average number of days between polls. When approval is trending either up or down, the amount of change between polls depends on how often the poll is conducted. This difference poses problems for how reporters frame the results of polls, as reporting this week has dramatically demonstrated.

On Tuesday, December 20, readers were confronted with two seemingly incompatible stories about President Bush's job approval rating.

The Washington Post had it this way:

Bush's Support Jumps After a Long Decline: More Americans Upbeat on Iraq, Economy

By Dan Balz and Richard Morin

President Bush's approval rating has surged in recent weeks, reversing what had been an extended period of decline, with Americans now expressing renewed optimism about the future of democracy in Iraq, the campaign against terrorism and the U.S. economy, according to the latest Washington Post-ABC News Poll.

CNN had a rather different lede:

Poll: Iraq speeches, election don't help Bush

CNN -- President Bush's approval ratings do not appear to have changed significantly, despite a number of recent speeches he's given to shore up public support for the war in Iraq and its historic elections on Thursday.

(Sadly, PoliticalArithmetik used essentially the same headline here.)

Such dramatically different interpretations of polling data contribute to public distrust of polls and the suspicion (seemingly universal in both left- and right-leaning blogs) that polls are widely manipulated to suit biased media agendas. Oddly, at the same time, poll results that support the blogger's own preferences are immediately embraced as "proof" that Bush is either surging dramatically or reports of surge are a myth, again depending on the blogger's preferences.

I've written extensively here over the past week about variability in poll results and when results are really out of line with other polling and when it falls within the limits of normal variation. But there is a different and systematic problem with the interpretation of poll results that deserves attention by those who report on polls.

Many, perhaps most, poll stories take their lede as in the above two examples by pointing out the change from the previous to the current poll. To say approval of President Bush is either up or down requires a comparison to some previous poll. Because news organizations sponsor their own polls, the point of comparison is almost always to that organizations own previous poll. But there is the rub. News organizations vary dramatically in the average time between their polls. In 2005 here is how many days elapsed between polls, on average:

Gallup 8.6
Fox 20.7
AP 25.1
CBS/NYT 27.1
Pew 27.1
Zogby 27.1
ABC/WP 32.0
NBC/WSJ 39.1
Hotline 44.0
Newsweek 58.7
Time 58.7

Gallup polls are taken more than twice as often as the next most frequent poll, that of Fox News. Gallup is 3 times as frequent as CBS News and the New York Times, and 3.7 times as frequent as the ABC News/Washington Post poll.

So what happens when reporters compare results between editions of the Gallup and ABC/WP polls? If approval is trending up, as it is right now, then on average, the ABC/WP change between polls will be 3.7 times larger than the change between Gallup polls, even when both are changing at exactly the same rate per day!

Since November 11, approval of President Bush has been trending up by +0.1533% per day. Given the average spacing between polls, this means that consecutive Gallup polls would be expected to register gains of 1.3% while the average spacing for an ABC/WP poll means the average gain between their polls would be 4.9%. And that has absolutely nothing to do with the real rate of approval gain, but only the spacing between the polls.

Now the average sample size of national polls implies a margin of error of about +/-3.2%. So what's your headline and lede for a story based on the Gallup poll? "President Bush's approval ratings do not appear to have changed significantly" But if you write the lede for ABC/WP: "President Bush's approval rating has surged in recent weeks, reversing what had been an extended period of decline"

Let's take a look at the year-long trends in Gallup and ABC/WP. The figure below shows the two poll's path over the year and the gray line is the linear trend based on all polls taken this year, taking account of the shocks of Katrina, Libby and the post-November 11 aggressive White House response to its critics.


























Both Gallup and ABC/WP fall a little above the linear trend of all polls, a well documented house effect that need not concern us here, but they track each other quite well. What is also obvious is that there are more blue dots for Gallup polls and fewer red dots for ABC/WP polls. But when the two polls are taken close together in time, they are usually pretty close to each other in results, certainly within reasonable margins for error.

So what do these two polls tell us has happened since late-October/early-November? Both show a substantial upturn in approval of President Bush. Gallup's three latest polls raise the question of whether approval is continuing to climb, but compared to Gallup's low points in early November, approval is clearly up. For ABC/WP, there are only two polls-- early November and mid-December. The upturn is more dramatic than Gallup's (though it's low point is also a bit higher so the gain in ABC/WP is +8% while Gallup's is +4%) but the conclusion is obvious from the figure: both polls see an end to the year-long decline and a significant upturn since early November.

The differences in these two stories' ledes is primarily a problem of journalistic interpretation of polling trends. Paradoxically, those who poll the most often, are least likely to find significant changes between pairs of polls, though they are more likely to capture short-term dynamics that other polls miss altogether for lack of data.

Because news organizations have a proprietary interest in the "branding" of their own polls, news stories lede with comparisons to their own past polls. But this practice is dramatically out of touch with the reality that the public is exposed to a large number of other polls. So when CNN writes that the current Gallup poll shows no change, they are implicitly forgetting the substantial trend over the past six weeks. When ABC/WP writes of a ratings surge, they are ignoring the fact that we've been reading 25 (twenty five!) other polls since ABC/WP did their last poll. Instead their lede is written as if there were no information about opinion in the intervening month and a half.

While no news organization will want to lose it's brand name poll, the practice of writing about polls without a context longer than "our last poll" and without a context "in relation to other polls" will continue to produce ledes that are an embarrassment when read side-by-side and which mislead the marginally attentive reader who needs the story to put the results in a meaningful context, both over time and compared to other results. One way to accomplish the first goal is for each story to come with a graphic showing that poll's history over the past year. This would also play to the strength of frequent polling by some organizations-- their graphs would look more informative and authoritative.

The second goal could be partially met if reporters wrote their ledes with the implicit acknowledgment that they had kept up with PollingReport.Com (or better, PoliticalArithmetik.) In that case, the ABC/WP poll could be cast as the confirmation of trends seen in other polls. Note the air of authority! Others suggest the trend, but ABC/WP "confirms it as fact". That's not counter to branding concerns but is less misleading and seemingly unaware of 25 other polls.


P.S. A reader asked "What's a lede? It isn't in my dictionary." Thanks to them, I found this very nice discussion of the journalist's term (a sign of my misspent youth) for the first sentence or paragraph of a news story.

ABC/WP Approval Way Up, Gallup Steady

Local regression fit and residuals for presidential approval. The horizontal lines mark the 95% confidence interval for the residuals. Points in red are the Diageo/Hotline polls, points in blue are ABC/Washington Post.

New polls from ABC News/Washington Post and Gallup/CNN/USAToday again present us with puzzling differences in the approval of President Bush. The 47/52% approval/disapproval rating the the ABC/WP poll is a 8% gain since their last poll, 10/30-11/2, (39/60%) which was before the White House launched its aggressive response to critics on November 11.

In contrast, the new Gallup poll finds a 41/56% approve/disapprove rate, which happens to be the same as the 41/56% results in their 10/28-30, the Gallup poll closest to the dates of the earlier ABC/WP poll. The most recent Gallup poll, completed 12/9-11 had a 42/55% approval/disapproval reading.

This pair of polls resemble the contrast between last week's Diageo/Hotline poll (50/47%, 12/12-13, a +11% increase in approval) and the Fox News poll of 12/13-14 which found support stable at 42/51%. I discussed those results here, and showed that the Diageo/Hotline result was one of the most extreme outliers in my data since 2002. Now with the addition of another pair of polls, one with a substantial approval gain and the other with no gain at all, we again have to consider what is going on here.

Since November 11, my model predicts a gain in approval of +5.8% from the low point of approval following the Scooter Libby indictment and Harriet Miers nomination failure. With that gain, the model estimates approval at 42.5%-43.9% as of 12/18. Depending on how you estimate the model, the ABC/WP poll is either within the 95% confidence interval of this estimate (as in the graph above) or just above the upper margin of the confidence interval (as in the graph below).

The first point is to not focus on the change since the previous poll by ABC/WP, because that poll is over a month old. President Bush has been gaining support at a rate of +.15% per day since November 11, so we should certainly expect gains from polls taken before November 11. Rather the comparison should be between our best estimate of current approval and the individual poll result.

I've estimated the current approval model in two ways. One is with a local regression which flexibly fits the local trends in the approval data. That local fit, graphed above, produces a current estimate, as of 12/18, of 43.9%. That model produces a 95% margin of error of approximately +/-5%. By that standard, a poll as high as 43.9+5=48.9 would fall just on the upper limit of the confidence interval. At 47% approval, the ABC/WP poll falls easily within this margin of error. This leads to the conclusion (based on this model) that the new ABC/WP poll is not inconsistent with the estimate that approval is currently 43.9%. We should expect to see polls this high if approval really is 43.9% and need not conclude that approval has surged to 47%. Note that the same does not hold for the Diageo/Hotline poll, at 50% approval, which falls easily outside the 95% confidence interval. (But note also that with the addition of the new ABC/WP data, the Hotline poll is less of an outlier than it was on Friday because the ABC/WP result helps pull up the estimated approval at the time of the Hotline poll.)

The conclusion here, then, is that ABC/WP is not an outlier, and not inconsistent with the model estimate of 43.9% approval.

There are other ways we could estimate current approval. One of those is to use a linear model, rather than a local regression. The linear model is far less flexible, but also less likely to respond to a short term "bump" that doesn't really represent meaningful change. So for comparison, I estimate a linear model here. The data are fit only to the approval data since January 1, 2005. The model incorporates a linear decline in approval from January 1 until November 10. Step-function shocks for Katrina and Libby are also included. Starting November 11, I estimate a linear growth in approval as well as a positive step function to capture the new White House approach. This model has previously been found to fit the data reasonably well, and it continues to do so with the addition of the post-November 11 data.

The results of this model are plotted in the figure below. There has been a clear and statistically significant increase in approval since November 11, and especially the upward trend is positive and significantly so. Based on this linear model, approval on 12/18 is estimated at 42.5%. This model also produces a somewhat tighter margin of error, of approximately +/- 4.3%. By this standard, the ABC/WP data at 47% falls just above the upper limit of the confidence interval, 46.8, as plotted in the figure. Combined with the results from the local regression model, this makes me view the ABC/WP results with some serious doubt. But the ABC/WP is not nearly as extreme an outlier as the Diageo/Hotline result, which falls clearly outside the confidence interval in both figures.

























An important aspect of these graphs is the plotting of previous polls by both of these organizations. The Diageo/Hotline polls are in red, the ABC/WP polls in blue. In both figures, these red and blue points track the trend lines in the top part of the figures quite well. In the lower residual plots, all the previous polls by both organizations fall within the 95% confidence interval. This is important because it puts these latest polls in perspective. There is absolutely no evidence from this analysis that either of the polls is systematically out of line with other polling organizations. That these last polls by both are relatively large outliers in no way suggests that the polls are seriously flawed in any way.

What it does raise is two points: first, even excellent polls can produce an individual outlier on an individual question. Partisans are quick to seize on a single example to impugn the entire polling enterprise (and quick to reverse themselves if the next poll by that organization is more to their liking.) Second, we need not be helpless in the face of unusual data. If you believe in "just averaging" the polls, then you will include extreme outliers and as a result bias your estimate of approval upwards, in this case. If you look at the polls in the context of the margin of error you can (and I would) discount the value of a single outlier. My estimate here of 42.5% approval is based on a model that discounts the weight given to large outliers.

One other important issue is that any model (mine included) can be misled in times of highly dynamic change. It is conceivable that we are currently in a period of very rapid change in presidential approval. The upward trend since November 11 is 5 times as rapid as the downward trend in the first 10 months of the year. So we may yet see new polling that looks closer to the results of the Hotline and ABC/WP polls. But if we do, we will have a new problem: how to explain the larger number of polls that have missed that large upturn. I'm still putting my money on the significant upturn to about 42-43%, and holding off on bets in the high 40s. For now.


Related Posts:

Spacing of polls confounds the lede.

How out of line is the new Hotline Poll?

Approval of President Bush in 2005

Bush Approval, 2001-present

Approval in historical perspective

How out of line is the new Hotline poll?

Approval trends and residuals, January 2002-December 14, 2005. The solid red points are the most extreme 1% of all residuals. The middle band marked by horizontal lines includes 95% of all polls.

Yesterday the new Diageo/Hotline/Financial Dynamics poll was released showing approval of President Bush at 50%. I posted here about that result yesterday. An email conversation with MysteryPollster Mark Blumenthal has provoked me to follow up on this. Blumenthal suggested looking at other polls that are unusual and using that to put the Diageo/Hotline result in perspective. Excellent idea! And since Typepad is down, Blumenthal will have trouble posting his own thoughts about this to MysteryPollster.com. Credit to Mark for the idea. Blame to me if the execution falls short.

All polls are subject to sampling errors, so two polls taken under identical conditions will vary by a highly predictable amount. That variation depends on the sample size (and sample design) and the variability in the population. Polls also vary due to non-sampling errors. These come from question wording, order of questions in the survey, "house" effects due to different procedures across polling organizations, response errors and "everything else" that can lead to error in a poll. The combination of sampling and non-sampling error is often referred to as "total survey error". The problem is that we have well established formulas for calculating sampling error, but the modeling of non-sampling errors is much harder and lacks the formal properties that sampling errors possess. Sampling error reflects the variation in polls we'd find if there were no non-sampling errors. Total survey error reflects the actual variability we see. My approach here reflects the total survey error approach.

There are 858 national polls measuring approval of President Bush since January 1, 2002 in my database. (I'm omitting 2001 here because the shock of 9/11 creates unique polling problems that are not present in the 2002-2005 data I use.) The median sample size of those polls is 1004 (I have missing data in the sample size variable, so take this as a reasonable estimate but not necessarily exact.) If sampling error alone were the source of polling variability, then the margin of error we would expect for samples of 1004 cases would be +/- 3.1%. So that gives us a baseline for errors due to sampling alone.

To estimate the total survey variability, I fit a local regression to the series of 858 polls. That is plotted in the top half of the figure above. In this case I deliberately use a fit that allows lots of local variation. In other posts, I normally "smooth" this trend more to emphasize systematic trends over time, rather than short term "bumps" that are probably not meaningful. Here however, I want to give the polls the maximum benefit of the doubt. This means that the trend line will respond to what is in fact just noise, but will treat that noise as if it were a politically meaningful signal. This is a conservative procedure because it will tend to UNDERESTIMATE the size of the polling variability.

From this fitted trend line, I calculate the "residuals", the observed poll value of approval minus that predicted by the trend line. These are plotted in the bottom half of the figure. The upper and lower horizontal lines include 95% of the polls in the dataset. In theory, these would be symmetric about zero, but in these actual data there are a few more low values than high ones, so the lower bound is -5.55% while the upper is +5.07%.

Compare this result with what sampling error alone would produce, a range of +/- 3.1%, or a range of 6.2%. The actual range for 95% of the cases is 10.62, or 75% more variation than would be predicted by sampling alone. This is quite typical of comparisons of total survey error with sampling error alone in the sort of pubic opinion data we are working with here. (A footnote here: I'm using the average sample size to compute the sampling error. Technically I should compute that for each of the ACTUAL sample sizes and then combine that to get the implied sampling error. That number would be bigger than the +/- 3.1% I'm using here, so my approach is a little unfair because it underestimates the true sampling error. That makes the proportion of non-sample error a bit larger than it actually is. I doubt the technical correction would matter a lot, and it doesn't matter for my main point, so I'm using the simpler approach.)

So any poll is subject to this kind of variation, and the total survey error is a good estimate of the amount of variation we should expect in practice. It gives us a benchmark to use in assessing the deviations from the trend that we see in any particular poll.

We can now turn to the Diageo/Hotline poll. The residual for that poll is +9.12% above the trend line estimated based on all the polls. That is considerably beyond the upper bound for a 95% confidence interval. In fact, it is among the most extreme 1% of all the polls taken since January 2002. The 10 most extreme residuals in that time are

Zogby -10.96 4/9/2003
IBD/CSM/TIPP -10.17 9/8/2002
Zogby -9.13 9/5/2003
Zogby -8.88 3/24/2003
IBD/CSM/TIPP -7.94 6/9/2002
Gallup/CNN/USAT 6.95 12/16/2003
Gallup/CNN/USAT 7.31 2/6/2005
ABC/WP 7.44 2/9/2003
Hotline/FinancialDynamics 9.12 12/13/2005
Annenberg 10.89 1/25/2004

This represents about 1.1% of all polls in my database. So by that standard, the current Diageo/Hotline poll is exceptionally far from the trend line.

But every dataset has to have a most-extreme 1% of cases, so what can we make of this result? One conclusion is that we should thank the Diageo/Hotline people for reporting their results despite the large difference from other polls. It is not unheard of for pollsters to bury their results that look too different, or to "fiddle" with the weighting or other things to bring the results a bit closer to other polling. Everyone wants to make news but no one wants to look too extreme. It is, ironically, a sign of the credibility of the Diageo/Hotline people that they were willing to put this result out when it was certain to be remarked upon as out of line with other polling (exactly as I am doing here!).

But the fact remains that this poll is far beyond the bounds we would normally expect, even when taking total survey error into account, and not just sampling error. By this standard, we can say that we would very much doubt that the Diageo/Hotline result is simply a random outcome from the same process that has generated all the other polling. But the source of that exceptional variation remains a mystery. Our conclusion should be that this result should be substantially discounted in estimating approval of President Bush.

Every polling organization can produce results that are outliers. What is important is spotting them and putting them in proper perspective. That is far more desirable than suppressing the results or pointing to them as examples of "bias". What matters is performance over the long term, not in any single sample.

Fox Steady, Hotline Out of Line

Fox/Opinion Dynamics add to the impression that President Bush's approval ratings rise has slowed. The Fox poll taken 12/13-14 finds approval 42%, disapproval 51%. The previous Fox poll taken 11/29-30 had it 42-48. Both are better than the 36-53 on 11/8-9, just before the White House launched its aggressive response to critics. The latest numbers however show no gain in approval and a +3% increase in disapproval.

The Diageo/Hotline poll by Financial Dynamics 12/12-13 shows a much higher approval rating than any other poll since February, with 50% approval and 47% disapproval. That's a stunning result, as the graph makes clear. The Diageo/Hotline result is far removed from other results of the post-Veterans Day polling. The previous Diageo/Hotline poll had approval-disapproval at 39%-59%, which was in line with other results at that time, 11/11-15. As of 3:20 pm, 12/15, the full topline results were not available on the poll's website here so it was not possible to look for a possible reason (partisanship distribution, for example?) for the discrepancy.

Even including the Diageo/Hotline results in the estimation above, the blue lowess line continues to suggest a flattening of the rise in approval. The rising green line smooths the data more and continues to show a rising approval. Until convinced by other polling, I'll think of the Diageo/Hotline result as an anomaly and stick with the conclusion that approval has at least temporarily flattened out.

(See the previous post here on Pew, Zogby and NBC/WSJ for a full explanation of the lowess fits above and my reason for concluding that the rise in approval has slowed.)

UPDATE: 3:40 EST. Diageo/Hotline now has the topline numbers here. Their party identification numbers are 35% Rep, 37% Dem, 20% Ind. The party numbers include leaners. That's in line with other estimates of party. It isn't clear if Diageo/Hotline weights by party id, which would explain this, or if their sample just happens to be this close to what the exit polls would show. In any case, the high Bush approval is not due to a poll heavily weighted towards Republicans.

The one difference in the Diageo/Hotline and most other polls is that Diageo/Hotline puts the Presidential job approval question after a battery of favourable/unfavorable ratings of political figures. It isn't obvious why that should prime higher job approval ratings, however. In fact, several of the rated figures (DeLay, Duke Cunningham and Jack Abramoff) might be expected to hurt the Presidents rating, if anything.

The one other interesting anomaly is the approval rating within party id category. This poll finds approval of 86%, 44% and 17% for Reps, Inds and Dems. Those are quite a bit higher than other polls have found recently, but are up across the board in this sample. Gallup's most recent results were 81%, 38% and 10% approval by Reps, Inds, and Dems.

So, either it is a fluke or a harbinger. What it isn't, is in line with other current polling. We'll see.

Pew, Zogby, NBC/WSJ: Approval Upturn Stalls (?)

New polling by Pew, Zogby and NBC/Wall Street Journal suggest that the upturn in President Bush's approval may have stalled. The three polls reported approval levels of 38%, 38% and 39% respectively. (See PollingReport.Com here for the full details.) Disapproval levels in the three polls were 54, 62 and 55 respectively. In terms of changes from the previous polls by the same organizations, the changes in approval were +2, -1, +1. Previous polls have seen gains in approval of as much as +5% over previous surveys.

Also new here is a second Gallup poll, taken 12/9-11 that found approval at 42 with disapproval at 55. That was a statistically insignificant drop from 43-52 in Gallup's 12/5-8 poll. While that modest decline isn't convincing by itself, coupled with the three more negative polls, there is reason to wonder if the significant rise we've seen since Veterans day has now reached a plateau, at least for the moment.

In the graph, the green lowess fit is the same fit as previous graphs have shown. By that measure the upturn appears to continue. The blue lowess fit reduces the smoothing slightly (from 1/2 to 1/3 for the lowess geeks out there.) With a little less smoothing, the flattening is apparent in the blue line, that is absent from the green line. Local fits of this sort, especially at the endpoints as we are considering, can be quite sensitive to the amount of smoothing, and there is no simple test for the "right" amount of smoothing. My "eyeballing" of the data leads me to see a flattening of the upward trend when these new polls are added, which is why I fit the blue lowess fit. You may take your pick, or wait for more data.

In either case approval continues to be significantly higher than the low-points before Veterans Day, but there is at least a reasonable doubt that it is continuing up at the rate it has for the past 3-4 weeks.

Gallup, Cook continue Bush Upturn

New polls by Gallup and Cook/RT Strategies add further evidence that President Bush's approval ratings have improved significantly since Veterans Day. The new Gallup poll, taken 12/5-8 finds 43% approval and 52% disapproval. That compares to 38% and 57% in the previous Gallup poll taken 11/17-20. The Cook Political Report/RT Strategies poll, taken 12/8-11 shows a gain from 41% to 42% in approval, but also a gain in disapproval from 52% to 55% compared to the previous Cook/RT poll taken 11/17-20.


This is the first sustained upturn in the President's approval since January 2005. Based on a statistical model that incorporates the year long trend and "house effects" of polling organizations, the current estimated increase since Veterans Day is 2.66%. However, it is worth noting that the increase now looks stronger than the current model will allow. An updated model is in the works. Stay tuned. In the graph above the local regression line, which does not depend on that model, estimates that the current resurgence in support now appears close to the pre-Katrina levels of approval in late August. The upturn has taken place in less than half the time of the decline since Katrina.

There has been a great deal of speculation about "how low can he go". There has also been discussion of "free fall" in approval. The current evidence clearly shows that the Bush administration is still capable of rallying support. The questions now are "how long will the upturn continue" and "is it possible for the administration to restore stable and favorable approval levels"? There are few historical examples of second term presidents who have recovered from approval levels in the high 30% range. Thus any appeal to history rests on the weakest of evidence.

The substantive driving forces remain mixed. The economy is doing well by historical standards, and consumer sentiment has recently improved. Gas prices are down. But the war in Iraq continues to cost lives and money. The domestic agenda lacks a clear positive thrust. What next? Being an empiricist, I'll wait and see.

AP/Ipsos Poll Continues Bush Upturn

The Associated Press/Ipsos Poll conducted 12/5-7 adds further support for the upturn in approval of President Bush's job performance. In the poll, released today (12/9) the AP/Ipsos poll found 42% approval and 57% disapproval. That is up from 37% approval in the last AP/Ipsos poll of 11/7-9. The previous poll measured 61% disapproval.

My model now estimates a statistically significant gain in approval of +2.82% since the Veterans Day launch of a more aggressive response to White House critics. That gain has now more than made up for the post-Libby indictment drop of -1.91% and is approaching parity with pre-Katrina levels of approval. The Katrina effect is estimated at -1.33%, so the net loss over these events is now -1.33-1.91+2.82=-0.42%. There has been an additional decline due to the long term trend in approval, which has been reversed for the first time since January with the 9 most recent surveys.

While the White House cannot be pleased with 41% approval, the trend is clearly up for the first time since January, a clear win for the President.

Bush Approval Upturn Reaches Statistical Significance

The addition of the CBS/New York Times poll, taken 12/2-6 has pushed the improvement in approval of President Bush into statistically significant territory, even accounting for variation in polling organizations. The CBS/NYT poll is the 8th taken since the more aggressive effort by the White House to defend it's Iraq policy and decision making.

The CBS/NYT poll registered an increase of 5% since 10/30-11/1, from 35% to 40% approval. Disapproval also dropped 4% from 57% to 53%.

With the new poll added to the data, the estimated Post-Veterans Day increase in approval of the president is now estimated at +2.07%, up from the previous estimate of +1.26%. Such a large change based on a single additional poll should provide some caution that these estimates are not yet stable. Nonetheless, the balance of the evidence now favors an interpretation that the Bush administration strategy and rhetoric is paying off in improved public approval.

Bush Approval Post-Veterans Day

Approval of President Bush may have taken an upturn in the wake of his Veteran's Day defense of the Iraq war and renewed White House emphasis on responding to critics of the war. However, the upturn is not yet statistically significant, leaving some doubt based on the seven polls currently available since Veterans Day.

In the above graph, the general decline in presidential approval since Hurricane Katrina is evident, with evidence that there was increased decline following the week of the Lewis "Scooter" Libby indictment, the Harriet Miers withdrawal and the 2000th U.S. military death in Iraq.

On November 11th, President Bush launched a new series of speeches, aided by Vice-President Cheney and other White House spokespeople, that vigorously defended the Iraq war, the President's use of intelligence and strongly criticized the arguments for a quick withdrawal of troops from Iraq. Coupled with good economic news, one might predict an improvement in the President's approval as he gives his supporters a renewed argument in favor of his handling of the presidency.

The upturn in the polls is evident from the graph. What is still in question if whether the data are strong enough to give us confidence the increase is real, and not a random fluctuation in the polls. If we estimate a model of approval for the year that includes effects of Katrina, Libby and Veterans day, as well as the long-term downward trend seen all year, then the post-Veterans Day effect is a robust +2.85% improvement, and is statistically significant. However, these polls are themselves subject to persistent "house effects", the tendency of polling organizations to produce results that are consistently above or below the average of other polls. When we take these "house effects" into account in the model, the post-Veterans Day effect is a still positive +1.26%, but it is no longer statistically significant.

A way to get a "feel" for this is to note in the graph that three of the seven post-Veterans Day polls are well within the range of variation of the previous trend. The other four polls all are either above that range or right at the top of the range. So the evidence is certainly suggestive, but not yet convincing beyond a reasonable doubt.

Another way to see this is to plot the data as boxplots, so we can see the median and the spread of the data at various times of the year. In the boxplots below, the horizontal black line inside the rectangle is the median, and the rectangle covers the 25th-75th percentiles of approval. The lines that extend out from the rectangle show the range of the data outside the 25th and 75th percentiles. There is one unusually high approval rating in the Post-Katrina box, which is plotted as a point. (The area of the boxes is proportional to the number of polls in each period.)
























While the Post-Katrina data are well below the box for Pre-Katrina, and Post-Libby is well below that for Post-Katrina, the current resurgence in Post-Veterans Day still significantly overlaps with the Post-Libby data. If the improvement in approval is real, we should see this box separate from the Post-Libby box. More polling, and a continued upward trend, could demonstrate that. The Thanksgiving holiday depressed the rate of polling, unfortunately, just when some new polls would be welcome. Hopefully the next few days will provide more data and more confidence in the trends.

Data: The data include all national polls through December 4. This includes a Time poll taken over the weekend, and a Quinnipiac poll ending December 4. See The PollingReport.Com for the data.

Note the comparison with my plots of Bush Approval over the year and over the term here and here. The year-long plot shows a small upturn at the end. The full-administration plot does not. That reflects the relative influce of these seven polls at the end of the data. The lack of statistical significance keeps the upturn looking small or non-existant in the more comprehensive graphs. As more data arrive, the late-trends will become more apparent there as well as here where I have "zoomed" in on recent developments, giving more influence to the last 7 polls in a smaller pool.

Bush approval updated 12/6

President Bush's approval rating graphs have been updated. Not a whole lot of new polling since the last update, but some potentially interesting changes, not yet statistically significant. See the links to the right. Updated again with latest Quinnipiac Poll through 12/4.

Polling and Parties in Israel

The early elections in Israel are providing an interesting example of party system change. Prime Minister Ariel Sharon's decision to quit the Likud party to form a new centrist party, Kadima, has not only split Likud, but also drawn some support from other parties as well.

Likud has been dramatically hurt by the split, falling in polls to support worth only 12.8 seats on average across polls. Likud currently holds 40 seats. Likud has also been continuing to decline in recent polls (see below). The reduction of Likud from the dominate party in Israel to a modest sized third or even fourth largest party is a remarkable shift. Even with nearly four months of campaign left, and allowing for short term support for the new party to subside, this still looks like one of the most remarkable collapses in modern party history.

Like a business worth more when broken up, Likud and Kadima are worth more seats than Likud alone was before the split. Prior to Sharon's decision, Likud was estimated to win 37 seats based on five polls taken shortly before the split. Since then Kadima and Likud together have averaged 46 seats combined. The increase in total support has come at the expense of the centrist Shinui party, which currently holds 15 seats in the Knesset. Since the Likud split, Shinui support has averaged only 5.3 seats in polls. Shinui averaged 8.6 seats in polls before the formation of the new Kadima party, suggesting that not all of the loss from 15 current seats was due to competition from Sharon's new party. Three other parties in the figure above show visible loss of support. Labor has seen poll support decline from 27.2 to 25.2 seats. Shas has lost one seat, from 9.6 to 8.6 seats. National Union (Ikhud) has dropped from 6.4 to 4.8 seats.
























The campaign has a long while to run, but the first 10 days of polling has been kind to Kadima and harsh to Likud, with Labor holding steady or slightly gaining over the last 10 days. In the most recent round of polls, Kadima has support ranging from 31 to 39 seats. Labor is estimated to win 25-27 seats while Likud wins only 9-14 seats. For Likud this is a drop from the 13-18 seats they held a week earlier, immediately after Sharon left the party.

Likud's prospects are currently also clouded by the choice of a new party leader. Finance Minister Benjamin Netanyahu commands the most support in current polls of Likud members, but among the entire Israeli electorate, Likud polls worse under his leadership than that of one of his competitors, Defense Minister Shaul Mofaz. In a poll conducted by Shvakim Panorama for Israel Radio on November 30, Likud wins 11 seats with Netanyahu and 14 with Mofaz. Earlier polling found Likud won four more seats with Mofaz than with Netanyahu (18 vs 14, November 23, Maagar Mochot for Makor Rishon.) Likud will choose its new leader on December 19. Clearly that decision could affect the dynamics we have seen so far.

For those of us who study electoral and party systems, this is an exciting opportunity to observe a party system in change. For everyone, it is a powerful political story. Stay tuned. I'll update the polling on and off until the March 28 election.

Data: The data are taken from two primary sources, both of which are excellent. Angus Reid Consultants in Canada has a nice collection of polls from all over the world. Their summary of polls from Israel usually include helpful commentary as well. Independent Media Review Analysis in Israel does an excellent job of reporting current polling, usually the same day polls appear, or the next.

And one Note: Because the Israeli electoral system is extremely proportional and governing outcomes highly dependent on coalitions of Knesset parties, polls are routinely reported in terms of estimated seats won, rather than percentage of voter support. Raw percentage figures are in fact quite hard to come by. In particular, I am unaware of how undecided voters are handled in these calculations. That is a topic for another day.

Weekly Gas Price Update (11/21/05)

Judging from the number of Google keyword hits on this site, the gas price graphs I posted in September (scroll down to the 2nd entry) remain of considerable interest. So here is an update of weekly price of regular gas in the U.S., in real (that is, adjusted for inflation) dollars (based on October 2005 CPI excluding energy). The weekly series begins 8/20/1990 and includes 11/21/2005.

As you can see, retail gasoline is now back to pre-Katrina prices.

While some have argued that gas prices have driven President Bush's approval ratings, the recent drop in price has not corresponded to a rise in approval, at least so far.

Data: The weekly price series is available from the Department of Energy's Weekly Petroleum Status Report here. The link is to the Excel file.

Prices were adjusted using the CPI, excluding energy prices, available from the St. Louis Federal Reserve here.

Cheney vs Bush Approval Ratings

Vice-President Cheney's approval ratings, while consistently lower than those of President Bush, have converged with those of the President, especially in the last year. The two ratings are very highly correlated (above .90) but the Vice-President's approval generally trailed that of the President by double digits in the first three years of the Bush-Cheney administration. The gap narrowed to under 10 points in 2004 and narrowed again to about 5 points in 2005.


























(Note on the figure: The rectangles represents the 25th-75th percentiles of all polls in each year. The horizontal line inside the rectangle is the median of annual polls. The dashed line extends to indicate the extreme values in each year.)


The 2004 campaign and subsequent second inaugural appear to have marked the convergence of the two approval ratings. Since January 2005 both approval levels have fallen from around 50% to the low-to-mid 30% range. But both have dropped at about the same rate.

Earlier in the administration, the gap between approval ratings was much larger. Whether this is due to the President's extraordinary high standing in the wake of 9/11, the Vice-President's relative obscurity during this time, the common practice of giving Vice-Presidents the more partisan role, or the frequent criticism of the Vice-President in connection with Halliburton is uncertain.

In each of the major events which drove the President's approval rating (9/11, start of Iraq war, capture of Saddam Hussein) the Vice-President's ratings rose, but less so than those of the President. This is exactly what we would expect given the two men's roles.

I suspect the convergence is also due to the collapse of support from Democrats and the substantial fall in approval among Independents. With both Presidential and Vice-Presidential approval now resting primarily with the Republican base, there is less variation in support. When Independents and even a fair number of Democrats approved of the President, there was more room for these groups to provide lower levels of support for Cheney, thus creating a greater gap between Presidential and Vice-Presidential appoval. With Democratic approval now at or below 10% and Independent approval in the low 30% range, Republican identifiers are more likely to approve of both and hence produce the convergence we see.

While some commentators seem to believe that the Vice-President faces much lower approval than the President, the fact is that those days are substantially over. The linked fate of the two is now tighter than ever.


P.S. A reader commented that they would like to see the comparison for Gore and Clinton. Here it is for the modest amount of data available. I could not find Gore job approval prior to 1997. Harris did some polling then, but not enough for a very clear graph, so I also had to switch to Fox, which has a very good series starting in 1998. Fox uses the more common "Approve/Disapprove" question format.

























P.P.S. Saddlebags12 and Robert both raise the important question of the don't know rate, which can affect this sort of an issue. In the case of Cheney and Bush, there are some effects, but not enough to reverse my point which is that Cheney is not dramatically less approved of than Bush. The figure below shows three things (responding the the comments):

1) The raw disapproval rate for Cheney was a bit higher than for Bush in the first term, not the other way around. In the second term the disapproval rates are nearly identical.

2) Despite 1, Cheney's don't know rate was considerably higher than Bush's in the first term. This dropped to near-parity during the 2004 campaign. In 2005 the Cheney don't know rate has grown a bit, but is still 1/3 what it was in most of the first term.

3) If we take out the Don't Knows, by recalculating approval based only on the (Approve+Disapprove) denominator, then we find that Cheney was below Bush in approval through the first term, with the gap narrowing considerably during the 2004 campaign and in 2005. Qualitatively this is the same story as I originally told above, though the numerical differences are a bit shifted. (The Disapprove figures are just the complement of the approve when the DKs are removed, so I just show the approve figure.)

4) Different polls might be more sensitive to this. For example, the Fox data I used for the Gore-Clinton comparison has a higher DK rate for Gore than the Harris data do for Cheney-Bush.

5) Don't Knows can bite you. They didn't change my argument here, but they can't be ignored without peril.


























Data: The Cheney-Bush data are from Harris Polls. Harris is the only organization to publish a substantial number of both Presidential and Vice-Presidential approval results. By keeping to a single polling organization, we also avoid making comparisons across different question wording and organizational practices that contribute to differences between surveys. The Harris approval question is a four-alternative version, unlike the most commonly used approval questions. Harris' wording is:

"How would you rate the job Vice President Dick Cheney is doing: excellent, pretty good, only fair, or poor?"

Excellent and pretty good are coded as approval, while only fair and poor are coded as disapproval. These differences in wording appear to have modest effects in approval levels compared to those of other wordings and organizations. The data are available at PollingReport.Com here for the Vice-President. Presidential approval is also available at PollingReport.Com here.

Approval of Handling of War, Johnson to Bush

Approval of the job four presidents did in handling wars. The horizontal axis is scaled so the axes are comparable across presidents, which allows accurate visual comparison of the trends. Because of this, the scale may begin before a president takes office. Data are plotted only during a president's term.



President George W. Bush's approval ratings on handling the war in Iraq deserve to be put in historical perspective. It is likely that any president presiding over a long term war will suffer declining approval ratings. This would be especially so when battlefield victories of the World War II type are impossible due to the nature of the conflict. A challenge of presidential leadership is to maintain support for U.S. military and foreign policy in such circumstances.

In comparison with three predecessors, President Bush's approval ratings more closely resemble those of Lyndon Johnson than those of Richard Nixon or the President's father, George H.W. Bush. While initial support for the Iraq war was a good deal higher than for Johnson's conduct of the Vietnam war, the subsequent decline in support is similar. Presidents Bush and Johnson both had approval ratings in the 60% range initially. In Johnson's case, the Tonkin Gulf Resolution and subsequent buildup of troops in Vietnam was not accompanied by a significant “rally effect” upturn in approval. President Bush, in contrast, enjoyed a large upturn in support in the early weeks of the Iraq war, with approval reaching 76% in Gallup's poll of 4/16/2003, a month into the conflict.

Subsequently approval for both presidents' handling of the wars declined to the 40% range. In Johnson's case, this approval remained relatively stable in the 40% range through the last 9 months of 1966 before dropping significantly in January 1967. Through 1967, approval of Johnson's handling of the war was mostly in the 30's, with a few polls as high as the low 40s and an all time low of 27% in August 1967. Approval appeared to improve a bit in early 1968 with Johnson's withdrawal from the presidential race and announcement of a bombing halt and a renewed effort at negotiations with North Vietnam, though the final reading was 41% in May of 1968. (No polling is available after that for Johnson's handling of the war.)

By comparison, approval of President Bush's handling of the Iraq war remained above 50% until September of 2003. During the fall of 2003, approval bounced around in the 45-50% range across various polls. This changed in December 2003 with the capture of Saddam Hussein, when approval rallied again to the 60% range. This again fell, and most of 2004 found approval varying in the 40-50% range, with some improvement to the high 40% range in most polls as the November 2004 election approached. Approval of President Bush's handling of the war has fallen steadily throughout 2005, to current levels in the low 30% range. These approval levels are now approaching the all time lows of the Johnson administration.

In contrast with Presidents Johnson and George W. Bush, President Nixon enjoyed rather strong support for his handling of Vietnam. Despite some general decline in support during his first term, fully three-quarters of the Nixon polls are above 46% approval, and only two polls throughout his term fell below 40% approval. Half of Nixon's polls find approval levels over 52%. Approval improved from a median of 52% in 1969 to 53% in 1970, then fell to a median of 46% in 1971. The all time low reading of approval for Nixon's handling of the war came in April 1971, at 32%. However, as is clear from the graph, that was an unusually low reading (ironically, as that poll was one conducted for President Nixon by Opinion Research Corporation). Average approval was still about 40% at that time. Nixon's approval ratings rebounded strongly throughout 1972 to a median of 54% as the Paris Peace Talks appeared to make progress, and topped out at a median of 65.5% in 1973 with the conclusion of the peace accords.

President George H. W. Bush enjoyed much stronger support for the brief Gulf War of 1991. In the immediate aftermath of the Iraqi invasion of Kuwait approval of his handling of “the situation with Iraq and Kuwait” (or words similar to that-- there was some variation across polls.) was very high, peaking at 83% in August 1990 shortly after Iraq's invasion of Kuwait. Support for Bush's handling of the situation then declined sharply until November 1990, bottoming out at 50% in a November 15 CBS poll. Subsequently, support increased as the U.S. diplomatic and military preparations for the Gulf War took on renewed urgency. Approval rose to the mid-60% range on the eve of the war, then surged to a high of 88% in a LA Times taken January 18, 1991. Approval remained in the mid-80% range though the end of the war. The last reading, taken April 9, 1991, also by the LA Times, was 80%. (Subsequent polling focused on post-war conditions and retrospective questions of whether military operations should have continued into Iraq and deposed Hussein. I've not mixed those questions in here with approval of handling of the war proper.)

If we can generalize at all from these results, I think it is that Nixon's “Vietnam-ization” strategy, combining negotiations, shifting fighting to Vietnamese troops, reducing U.S. troop commitments, even while maintaining heavy air support and bombing activity, won substantial approval from the public. (As an “approaching draft-age” high school student at the time, I was frankly surprised by these results. I remember a much less popular war which I had assumed had dogged Nixon's approval ratings. As the data show, one person's recollections aren't a very good measure of what the polls actually showed.)

If we assume the current administration is committed to continued troop presence in Iraq for the foreseeable future, the prospects for a recovery of President Bush's approval look bleak, barring a dramatic improvement in the Iraq insurgency situation. A shift to a clear “Iraqi-ization” of the war, in deeds as well as words, might possibly produce approval similar to President Nixon's. Or perhaps that has to wait for a new administration in 2009 to adopt a clear change in policy. The one important difference is that Johnson had only 16 months left in office when his approval ratings reached bottom. President Bush still has over 38 months remaining.

Data: I used all polls from any polling organization that asked questions that directly measured approval of a president's handling of the respective wars. Data are from the Roper Center's iPoll (subscription only) database.

Low Points:

Johnson, 27%, Gallup, 8/29/1967
Nixon, 32%, Opinion Research Corporation, 4/1/1971
GHW Bush, 50%, CBS/NYT, 11/15/1990
HW Bush, 30%, Newsweek/PSRA, 11/11/2005

Virginia Politics is (Mostly) Local

The Virginia gubernatorial election appears to reflect local forces more than a referendum on President Bush or demobilization of Republican voters. Republican margins in some of Virginia's larger counties and independent cities shrank substantially in the 2005 governor's race compared to the 2004 presidential election. But these margins shrank considerably less in the Attorney General's race, suggesting that rather than a uniform loss of Republican votes, the Virginia results were driven more by candidate and campaign specific forces.

Seven large Virginia counties and independent cities showed especially dramatic reductions in the Republican margin, as shown in the figure. Five larger counties/cities shrank from substantial Republican margins in 2004 to pro-Democratic margins in 2005. (Virginia Beach and Chesapeake cities and Henrico, Loudoun and Prince William counties.) Four of these had provided President Bush with margins over 10,000 votes in 2004, but shifted to 1,000 or more Democratic margins in 2005. Even where the Democrat, Tim Kaine, failed to win, the loss of Republican margin was damaging to Republican Jerry Kilgore. The Bush margin of over 34,000 votes in Chesterfield county was reduced to a Republican margin under 8,000 in the gubernatorial election. And in the Democratic bastion of Fairfax county a Democratic lead of 34,000 in the Presidential election was nearly doubled to a 60,000 vote advantage in the Governor's race.

In the figure, the purple line is the local regression fit between presidential and gubernatorial vote. The relationship flattens dramatically in counties/cities won by Bush. The Republican gubernatorial margin only modestly rises with Bush margin. In places won by Kerry, the relationship is much closer to the 45-degree line, representing equal margins in both elections. This shows that the Republican vote was much less strongly related to the Bush vote in those counties Bush won. On its face, this is consistent with a claim that Republican's failed to mobilize their voters, possibly because of disaffection with the President. That could be due to a weak President. It could equally be due to a weak gubernatorial candidate. Which is it?

Here the Attorney General's race (currently with a margin of some 410 votes) provides a crucial point of reference. If Virginia Republicans stayed home due to unhappiness with President Bush, then that effect should be clear across races. If on the other hand "all politics is local", then the Virginia vote is more a function of the appeal of the candidates and the short term campaign forces, and this should vary from race to race.

The Republican vote for Attorney General is much more strongly related to the 2004 Presidential Republican vote than is the case for the gubernatorial vote. The green line in the figure plots the local regression fit of Republican margin in the Attorney General race with the presidential vote. Among counties/cities won by Bush in 2004, there is a much stronger relationship between the AG vote margin and the presidential margin than was the case for the governor's race.

If Republicans had responded primarily to national forces, we'd expect the green and purple lines to run roughly parallel to each other. It is clear from the figure that this doesn't occur. The AG race margin is close to (but not quite) linear across all presidential margins. This contrasts with the strongly non-linear relationship of the governor's race margin. As a practical matter, the AG Republican candidate, Bob McDonnell, ran well ahead of gubernatorial Republican Kilgore's margins in the most Republican counties. McDonnell also did relatively much better in Fairfax county, losing by only about as much as Bush had in 2004, rather than twice as badly as did Kilgore. (I've not included the data points for the AG race in the figure because it makes an already complicated figure even harder to read. The AG margins cluster quite closely to the green fitted line however. In particular, the six highlighted counties/cities are considerably closer to the AG fit than they are to the governor's fit.)

There is still some modest evidence that Republicans suffered a decline in their voting base relative to 2004. Even in the AG race, the slope of the local fit is a bit smaller in Republican localities than it is among Democratic majority areas. The difference is not large, but could reflect some demobilization of Republican constituents, possibly due to President Bush's low approval ratings. However, the much larger declines in gubernatorial votes cannot be accounted for by this general decline. Most of that loss in Republican margin seems due to the candidates and the campaign. That effect could be due to a failure by Kilgore to mobilize his partisans or the effective stratgey of Democrat Tim Kaine to appeal to religious (and Republican leaning) voters. Thanks to the lack of exit polling we must be left to speculate about the reasons voters had for the votes they cast. But the county returns do cast serious doubt on the claim that Virginia's gubernatorial vote was primarily a repudiation of President Bush.

Data: The data are taken from the Virginia State Board of Elections website here.

Thanks to the "econometrician who must not be named" for helpful conversations, even if he/she didn't want to be associated with my analysis.

Approval of President Bush in 2005 and 2006

Updated through polls completed 10/22/2006

2005-2006 estimate of approval of President Bush.

The blue line is the estimated support for the President at each point in time. The gray dots are the individual polls which vary randomly around the estimated support. Each poll will differ from the common trend due to sampling variation, question wording or order among other ideosyncratic factors. The estimated support is based on a local regression which can flexibly fit the trend in the polls.

Approval of President Bush, 2001-present

Approval of President Bush. Gray points are polls, blue line is my estimate of approval trend, using a local regression. The spread of gray around the line indicates how much variation there is across polls taken at the same time.

The vertical grid marks quarters, with January 1 in a darker shade. Tic marks on the horizontal axis are months.

This graph is at 768 x 768 resolution. Some readers may need to click or double click on the graph to see it at full resolution.

Presidential Approval in Historical Perspective

Updated: 11/8/07























Presidential Approval since Roosevelt. The blue vertical lines indicate midterm and presidential election dates.

This graph shows all presidential approval readings by the Gallup organization since Roosevelt. Gallup's long series and consistent question wording provide a useful historical comparison across Presidents. The data are freely available at the Roper Center web site at the University of Connecticut here.

The data are graphed here in exactly the same perspective across all presidents. This means that trends will look the same regardless of whether a president served two terms, one term or only part of a term. Visual comparisons of trends will be valid because of this constant perspective.

For example, compare the decline prior to the 1992 elections in President George H. W. Bush's approval with those of President George W. Bush. The elder Bush is a clear example of "free fall", the sharpest and largest approval drop since President Nixon's in 1973-74. President George W. Bush's decline more closely resembles the long-term decline of Jimmy Carter's approval than it does the free fall of either the elder President Bush or President Nixon.

Because the vertical scales are also equal, it is easy to compare the level of presidential approval across Presidents.

A more complete explanation of how the almost universal failure to keep perspective constant distorts presentation of polling results can be found here.

Polling error distribution in CA propositions

Distribution of deviations between polls and final vote on seven California referendums, November 2005.

The several California referendums this week provide a good illustration of polling variability as well as a look at the average accuracy of the polls. Earlier I illustrated the distribution of the pre-election polls here. Now that we know the results, we can compare the outcomes with the estimates from the polls.

MysteryPollster has a comparison of the polls average accuracy across the referendums here. He focuses on the mean prediction error, while I focus on the “spread” of the distribution of errors. My concern is with the uncertainty we should attach to the polls as predictions of vote. The more spread there is, the greater our uncertainty. Because the prediction error also includes all non-sampling errors, these distributions are better measures of our practical uncertainty than is the sampling error alone.

It is worth admitting up front that proposition polling is unusually difficult from a technical perspective. MysteryPollster has addressed these difficulties here (and in related posts.)
I also address the issue raised here by “sennoma” to look at the change in the error distribution between early and late polling.

The figure above shows the distribution of errors, poll minus vote percentages, pooled across all the referendums. (I omit Proposition 80 which had few polls.) The dashed lines are for polls taken more than 10 days before the vote, while the solid lines are for polls taken in the last 10 days of the campaign. The reason to pool across polls and propositions is that we have no a priori reason to assume one poll is “better” than another and no reason to assume polling is easier or harder on any given proposition. Pooling allows us to maximize cases and assess our uncertainty prior to seeing any results.

The distribution of errors tightens for polls taken in the last 10 days. For “yes” responses the middle 90% of errors from more than 10 days prior to the election range from -16.5 to +17.1. For polls taken in the last 10 days the 90% range is -8.6 to +10.6. The median error shifts from +4.1 to +1.8, showing that the middle of the distribution comes closer to the actual outcome in later polling for the “yes” vote.

For the “no” responses, the 90% range is -20.1 to +6.5 for early polling, and -16.50 to +0.55 for the last 10 days. The median error is -12.5 for early and -8.5 for late polls, showing a systematic underestimate of the “no” vote.

Put another way, the 90% confidence interval for these distributions is 19.2% for yes votes and 17.1% for no votes. That range is quite large compared to the theoretical margin of error for sample sizes typical of these surveys, which are reported to range from 3-6% for a 95% margin of error. The empirical 90% range is +/- 9.6% for yes and +/-8.55% for no votes. This inflation of theoretical margins of error by a factor of 1.5 to 2.0 or more is typical of empirical estimates of survey accuracy as predictors of election outcomes. The inflation reflects sources of error in addition to those due to sampling, such as campaign dynamics, non-response, question wording effects and other sources of error. (And since I'm comparing 95% sampling margins to 90% empirical ranges, I'm being generous to the polls here.)

These results also demonstrate the problem of bias in the survey estimates. While the percentage of “yes” votes had a median error of only +1.5, the error for no votes was a rather large -8.5% median error, a quite large error. While smaller than for early polls, the median of the “no” distribution falls rather far from the actual outcome. One might chalk this up to “undecideds” who ultimately voted no, but this is a problem of modeling from the survey data to a prediction of the outcome--- something that few pollsters actually do in public polling. (MysteryPollster calculates the errors using the “Mosteller methods” that allocate undecideds either proportionately or equally. That is standard in the polling profession, but ignores the fact that pollsters rarely adopt either of these approaches in the published results. I may post a rant against this approach some other day, but for now will only say that if pollsters won't publish these estimates, we should just stick to what they do publish-- the percentages for yes and no, without allocating undecideds.)

The bottom line of the California proposition polling is that the variability amounted to saying the polls “knew” the outcome, in a range of some +/-9.6% for “yes” and +/- 8.55% for “no”. While the former easily covers the outcome, the latter only just barely covers the no vote outcome. And it raises the question of how much is it worth to have confidence in an outcome that can range over +/- 9 to 10%? In general, I'm pretty confident elections will fall between 40% and 60%. That's only slightly larger than the empirical uncertainty in these polls. If anyone would care to pay me for that prediction, I'm pretty sure I will be as accurate as these polls. (Apologies to my pollster colleagues. But don't worry, no one will pay me for my opinion!)

The next post below (I'm posting these in reverse order) looks at polling in the New Jersey, Virgina and New York City partisan elections. These are interesting to compare to the much harder to poll proposition issues I've focused on here. 

Polling error distribution in NJ, VA and NYC

Distribution of polling errors in New Jersey and Virginia Gubernatorial elections, and New York City Mayoral election, November 2005.

As with the California propositions, we can calculate the variability of the errors for the New Jersey and Virginia Gubernatorial and the New York City Mayoral election polls. Again, there is little a priori reason to think the poll performance will differ across these, so I pool over the three elections and all the polls.

The figure above shows the variability of the errors and the shifts in the location of the distributions between early and late polls. For polls taken more than 10 days before the election, the middle 90% of poll errors is -10.70 to -0.67 and -9.04 to -1.65 for polls taken in the last 10 days of the campaign. The medians are -6.65% and -5.19 for early and late polls, a modest improvement but still a substantial underestimate of the final vote.

For the Republican vote, the 90% interval is -7.78 to +4.22 for early polls and -8.53 to -1.86 for the final 10 days. The medians are -2.78 and -2.39 for early and late polls.

These results show a modest decrease in uncertainty in late polls, and small shifts of the median estimates towards the true outcome. However, the median underestimates the Democratic and Republican share of the vote, even in late polls. As with the California proposition polling, these underestimates reflect the failure to allocate the undecided voters in the polls. But this begs the question of how such votes SHOULD be allocated and by whom. We can arbitrarily choose either of the Mosteller suggestions mentioned above: proportional or equal allocation of the undecideds. But the more reasonable method of allocation of undecideds requires access to the raw individual level data, which is not available in general and never before the election.

On the face of it, one might estimate the predicted probability of voting and of voting for each candidate among the undecideds, using the decided voters for the estimation, and from this estimate the final vote prediction. Pollsters are apparently unwilling to do this, preferring to simply report poll percentages rather than make modeling decisions as well. (This seems a reasonable business choice, even if it frustrates election prognosticators.)

The bottom line here is that the empirical margin of error for late polls in these three races is +/- 3.7% for the Democrats and +/- 3.3% for Republicans. Unfortunately, neither of these ranges covers the final vote percentage, so the bias in estimating the voter percentage is larger than the margin of error. Again, without the raw data, we cannot calculate an optimal allocation of undecideds which would reduce the bias estimate. (One can also calculate the “spread” between the candidates, which estimates who is ahead. Those calculations make the polls look a bit better in picking winners, but with still substantial variability, which is my story here, rather than prediction per se.)

Polling for the 2005 elections

Today's elections have produced a good deal of polling and considerable uncertainty as to likely results. MysteryPollster has done several excellent posts on this here, here, here and here. MP stressed the difficulty of polling on California's propositions, and provides a thorough discussion of the implications for poll variability.

Here I want to show you the variability across polls in the propositions plus the NJ and VA governor's races. I'm going to do this a little differently than you usually see it. Here's why.

We can think of each poll as a sample of the likely outcome. The uncertainty arises from both sampling errors, campaign dynamics and non-sampling errors such as non-response, question wording and a host of other polling demons.

It is common to toss out early polling and focus only on the last week or even days of polling. I'm not going to do this. Rather I want to show the total distribution of results across the polls. This reflects the uncertainty due to campaign dynamics, as well as the other factors. The reason I think that's worth doing is that changes over the campaign are unpredictable. What has gone down may come back up in the end, or vice versa. We place a lot of faith in the last week of polling (not wholy misplaced) but in representing how uncertain we are about the likely results, I think it makes sense to consider the total variability in polling. This provides more polls as part of the evidence as well.

Of course where there are clear trends in the polls, this may make the uncertainty appear greater than would be the case only with late polls. That matters if we are primarily interested in predicting winners. But if we want to represent uncertainty, I think my approach is preferable.

So in the graphs below I present the distribution of polls for each California proposition and for the NJ and VA governor's races. Where there have been clear trends, I mention them. But you won't find the usual time series plot here. That isn't my point. Rather look at where polls have been highly variable--- we should be very uncertain there. Where they have not been, we should be less uncertain.

The wide variability in percentages and in spread should be appreciated by readers of polls (and perhaps by pollsters themselves). Sampling error is only part of the story of uncertainty. These plots give a better sense of how much uncertainty comes from all sources, not just sampling.

The conclusions from most of these plots: don't place big bets based on the polls. (But do note that polls DO seem to provide clear evidence in some cases. The exceptions count too!)

Data: See RealClearPolitics.Com for the numbers and for the trends.

California propositions 73 and 74 polling

Polling on props 73 and 74 show a wide spread with no clear leader for either proposition.

Prop 73, parental notification for minors seeking abortions, has produced widely divergent polling results with alternating polls finding substantial leads for both sides. The distribution of the spread shows a range of over +/- 15%, a huge range of uncertainty.

Prop 74, modifying tenure rules for public school teachers, has also produced substantial variation in results. Some of these are due to trends over the fall. Early polling produced a number of "yes"-leading results, but the most recent polling has found "no" leading by single digits. 

California propositions 75 and 76 polling

The polling on Prop 75, controlling the political use of union dues shows an interesting bimodal pattern. Much of the spread is concentrated around zero, a dead tie. But there is a second mode- centered around a 20% advantage for the "yes" side. Much, but not all, of that right tail comes from early polling, but even polls in the last week find results with both sides winning.

The proposal to limit state spending (and modify previous propositions regarding spending), Prop 76 seems likely to lose. This is a case where the simple "no" percentage has been polling well over 50%, rather than both options polling below 50%.