While party identification at the individual level has a strong effect on approval of President Bush, variation in partisanship across polls accounts for relatively little (12%) of the variation in presidential approval ratings. The implication is we should be less obsessed with the party balance across polls. In only 12.1% of polls over the last 15 months has the partisan balance made more than a +/- 1% difference in approval. In NO polls over that time has it made as much as a +/- 2% difference.
Those of us who obsess about polls have a tendency to judge the quality of a poll by its partisan balance, especially if the results displease us. So when we see a poll with 28% Republican identifiers and a low approval rating of President Bush, we quickly dismiss the polls as obviously biased because we know Republicans make up at least 31% of the population and for this reason the poll underestimates support for the president.
What the analysis here shows is that the effect of the partisan distribution in a poll has less impact on that poll's estimate of approval than is often assumed. The effect isn't nil, but neither is it large. While we may still want to consider the partisan balance in polls, we need to take care that we are not confounding house effects, which persist across polls, with random variation which does not.
In a previous post here, I showed that polls vary dramatically with respect to their partisan balance. These house effects largely have to do with how the poll treats independents. Some produce far more independents than others, and so produce fewer partisans of both camps. But random variation also affects polls within survey houses. That variation MAY affect the estimated approval level, but the partisan rhetoric exaggerates this effect.
Survey houses have characteristic levels of partisanship which are due to question wording and training of interviewers, among other things. Obviously different questions will produce different answers. Fox, for example, asks if the respondent is Republican or Democrat but omits mention of the possibility of being independent. Not surprisingly, Fox produces very large percentages of Reps and Dems but few independents. Other houses offer "Republican, Democrat or Independent", while some add "or something else". The latter produce more respondents who don't fit the three standard categories. And some train interviewers to probe for a party if the respondent says "don't know" or "none", while others teach interviewers to code these respondents as "other party". As we saw before, this variation in house effects accounts for a large amount of the variation in partisanship as measured across polls.
The graph below shows that these house effects are positively related between Republican and Democratic identifiers. This is another way of seeing that the primary action here is in the proportion of independents, rather than the relative balance of the two parties. Pollsters are not biased towards "Dems" or "Reps" but rather the house effect is more or fewer independents. The graph shows the estimated house effects for Democrats and Republicans. The strong positive relationship indicates that houses tend to be low on both or high on both--- indicating that how they treat independents is the primary "bias" (that is a technical term but has unfortunately strong negative colloquial connotations. I'll try to avoid it, even though it is a perfectly good term in technical statistics.)
Fox and Greenberg (opposites ideologically, if you buy the common stereotype) both register more Republicans, but they ALSO typically find more Democrats as well. NBC/Wall Street Journal has an amazing ability to find independents, and so they tend to find almost 6% fewer Republicans AND 6% fewer Democrats than the average pollster.
As we saw in the previous post, these house effects lead to substantial differences in partisanship across polls. But how much effect is there on presidential approval?
To look at this, I first estimated models that estimated house effects for approval (accounting for trends and events in 2005-06) and for partisanship (accounting for the trend over 2005-06 plus the house effects). A survey house that systematically includes too many Republicans may overstate President Bush's approval. However, the above figure shows that this isn't what most house effects do. Either the survey house is low on BOTH Reps and Dems, or high on both. Rarely does a house really favor one party over another. The net effect on approval then is relatively modest.
To see this, the figure below plots approval and partisanship, both NET OF HOUSE EFFECTS. I've estimated my standard model of approval which includes linear trends that decline over 2005-06 plus shocks for Katrina, the Libby indictment, a trend up after the president's defense of his policies from 11/11/2005-1/31/2006, and a new decline following the State of the Union Address. I also include house effects for all pollsters. The net effect is then the variation that is not accounted for by the model.
The net partisan effects capture how each poll varies from what we would expect given both house effects and the long shift in partisanship since January 2005. (
Read about that here.) So the figure below shows how polls varied in approval from what we would expect given trends and house effects, as a function of how they varied due to partisanship in the poll, again net of house and trend effects. The figure also shows the covariation of partisan categories across polls.
What is striking in the first row of the figure is that net approval really doesn't vary that strongly with any of the net partisanship figures. Again, think net approval as the variation in approval beyond what we would normally expect for a survey house. Think of net partisanship as variation in party beyond what we would normally expect for that house. So if Gallup is normally 31% Republican, but a particular Gallup survey found 29% Republican, they would be a -2% in the Republican panel above. And so on for the other parties. Likewise if we would predict that poll at that time to be 44% approval but we found 46%, then the "Net Approval" is +2% in the figure.
But what we see is that while there is a statistically significant tendency for net approval to rise with net Republican, the effects are quite small. The effects of net independent and net Democrat are even smaller. If we predict net approval from all three net partisan categories, the model accounts for only 12% of the variance in net approval, a quite modest amount.
Given estimates of the house effects and the partisan composition of each of the 173 polls taken between 1/1/2005-3/31/2006 for which I've gotten both partisanship and approval data, I can estimate how much partisan composition matters in approval ratings. The surprising answer is not so much. The figure at the top of this post shows that only 12.1% of the polls in this time differed by more than 1% of what was predicted due to partisanship distribution in that poll. And not one of the 173 polls differed by more than 2% of what we should have expected.
This may seem puzzling, given that 80% (more or less) of Republicans approve of the job President Bush is doing, while 90% (more or less) of Democrats disapprove. So you'd think that for every 1% change in Republicans you'd get a .8% change in approval and for every 1% change in Democrats you'd get a .9% change. Ahh, but don't forget the independents! They buffer the changes in partisans, and in fact the correlation between percent Rep and Percent Dem in a poll is only about .2-.3, quite a modest correlation. There are also between zero and ten percent "other", depending on the survey house, which further buffer partisanship. So while in a single poll it may well be true that 90% of Dems disapprove and 80% of Reps approve, across polls the effects are not so strong. For one thing, the percent approving WITHIN each partisan category ALSO varies across polls. So the amount of variation across polls is considerably more than you would estimate from looking at the cross tab of approval by party in a single polls.
The bottom line, based on these 173 polls, is that it takes about a 5% shift in Democratic identifiers to produce a 1% shift in approval. Among Republicans, a 3.3% shift in partisans shifts approval by 1%. Since, in fact, few polls vary by much more that +/- 3% from THAT POLL'S normal partisan distribution, the estimated approval rarely differs by more than 1% from what we would expect, and almost never by as much as 2%.
When I run a poll, one of the first things I look at is the partisan distribution. It is an important diagnostic for how well my sample represents the population. But while party identification is powerful at the individual level, there is quite a lot of variation across polls that is not due to partisan balance alone. The current obsession with partisanship in polls has caused us to exaggerate the impact of partisan balance on other estimates in the poll. Of course we should care about the partisan distribution, but all too often we exaggerate its impact. For the last 15 months it has never caused as much as a 2% shift in approval, and only rarely more than a 1% shift.
Compare approval across polls to the results of other polls, their trends and take account of predictable house effects. What's left due to partisan balance is usually modest.
Click here to go to Table of Contents
