There is a lot of interest in the differences among pollsters, and especially what effect they have on the perception of the race. Here at Pollster, the interest is specifically on the question of whether individual pollsters drive our results, and if so by how much.
So here is a quick look at those effects in Pennsylvania's Democratic primary.
The chart above shows the trend estimates that result from dropping each pollster in turn, and reestimating the trend without that pollster. This is a specific test of how much it matters whether we include a particular pollster or not.
Over the 15 pollsters we have represented in Pennsylvania, the estimates excluding each one for Clinton range from 46.6% to 48.3. The estimate with all pollsters included is 47.4% for Clinton.
For Obama, the estimates range from 41.2% to 42.5%, with the estimate for all pollsters at 42.0%.
The upshot is that pollsters do matter, but none drive the results by very much. A 1.7% range on Clinton and a 1.3% range on Obama for the trend estimate is very small compared to the range we see across the raw poll results. Another example of the greater stability of the trend estimators we use compared to the substantial variability across polls.
We can look at the effects of each pollster by comparing the trend estimate without that pollster to the estimate with them. The higher the effect, the more that pollster drives our trend estimate up for that candidate. Negative effects means the pollster drives our estimate down. Again, this is compared to the trend with and without the individual pollster.
The two charts below show these effects for Clinton and for Obama.
For Clinton, SurveyUSA has the highest positive impact on our trend estimate, followed by Rasmussen. At the opposite end, PPP has the largest negative effect on our trend, with Zogby/Newsmax the next largest negative effect. Other pollsters are clustered rather closely around zero.
And it is important to note that even the four largest positive and negative effects are all less that 1 percentage point.
For Obama's trend, Quinnipiac shows the largest positive effect, followed by Zogby and PPP with near identical effects. On the negative end, SurveyUSA has the most negative effect of Obama trend. Again, none of these effects is as much as one percentage point, and SurveyUSA's is less that half a percentage point. Other pollsters have less impact.
We can also look at the joint effects. These are the same as seen individually above but plotted against one another.
Pollsters do matter, and outliers matter even more. But the net effect of any individual pollster on our trend estimates in Pennsylvania are modest, especially when viewed in comparison to the wide range of raw poll results for each candidate. Another advantage of combining information across polls rather than pick single polls we "like".
