(LARGE graphs-- you probably should click once or twice to see them at full resolution in order to see the details.)
A new American Research Group (ARG) poll of Iowa has caused quite a debate in the comments at Pollster.com. Today Mark Blumenthal takes a close look at polling the "Dark Side of the Moon" in Iowa during the holiday season and the unknowns involved, including the unknowns of the ARG poll. So while the pollsters are busy trying to get one more Iowa poll in this week, let's look at the track records of the pollsters in Iowa.
The chart above shows the polling of the Democratic nomination race in Iowa since January. In 2007, twenty four different polling organizations conducted Democratic caucus polls. Of these, only 7 have conducted three polls or more. That means that for most pollsters, we have no way to separate random error from systematic "house" effects. To do that, we need multiple polls, and to do it reliably we need a number of polls from each organization. But we can at least look at the seven organizations with three or more polls, and see how they compare to the trend estimates. The trend, of course, reflects the best estimate across all pollsters, but until we see the actual vote (itself a very slippery concept in Iowa) we can't say if the trend was better than individual polls or not. Still, this reveals when pollsters seem to follow the trend and when they systematically seem to miss it.
The clear result of the comparison above is that ARG generally showed a much better performance for Clinton than the trend in the first half of the year. From January through June, ARG usually had Clinton some 6-10 points above the trend estimate, with one exception in which ARG agreed almost exactly with the Clinton trend. During this period, ARG was the most discrepant of all pollsters from the Clinton trend.
In the second half of the year, ARG's polling has generally been much closer to the trend estimates, usually less than 4 points away from the Clinton trend. During this time ARG has mixed in with other polling pretty consistently.
That is, until the most recent December 20-23 poll, which again shows a large Clinton deviation of about +6 points from the trend. (I'm using the standard blue trend line. The upturn in the red "sensitive" estimate is interesting, but it is also sensitive to the ARG poll, an example of why you might not entirely trust the red estimator.) This is the first large ARG deviation recently, and quite a change from ARG's previous poll of Dec 16-19 which was close to trend.
In an inversion of the Clinton effect, ARG consistently underestimated Obama's support, compared to the trend, in the first half of the year, but their polling fell much closer to the trend during the second half. At least until the latest poll which has Obama some 10 points below the standard trend. (Again, note the downturn of the sensitive estimator, which would be turning down even without ARG, but which is turning down somewhat more because of ARG.)
ARG's polling for Edwards has been more variable. Early 2007 polling fluctuated quite a bit, sometimes below and sometimes above trend. More recent ARG results for Edwards has generally been a few points below trend, with the latest result about as much below trend as has been "normal" for ARG recently.
ARG's results this year have been more heterogeneous than some oversimplifications claim. There was a substantial overestimate of Clinton early in the year, along with an underestimate of Obama's support. That was important during this period because there were relatively few other polls available for Iowa in this period. But in the second half of the year, the ARG results for Clinton and Obama have been much closer to trend estimates, though still with a small average advantage for Clinton and small disadvantage for Obama.
In light of this, the ARG poll for December 20-23 does look out of line with their own previous polling, certainly their polling of the last 4-5 months.
Let's look at their Republican results for Iowa.
On the Republican side, ARG's results have been especially favorable to John McCain, and to a lesser but still substantial degree to Giuliani. Unlike the Democratic results, these effects have not diminished much in the second half of the year. McCain has often been as much as 10 points above trend, with only one poll below trend and one more right on the trend. No other pollster has been so consistently far from the trend for McCain.
For Giulinai, the results are less far from trend, but still quite consistently above trend. Only 2 of the last 10 ARG polls have Giuliani below trend.
Romney has fared close to trend in the ARG surveys, though on average a bit below trend. The discrepancies for Romney are much less dramatic than for McCain or Giuliani.
The last two ARG polls show shifts of -3 and +4 points for McCain and Romney respectively, and a single point difference for Giuliani. (And a -5 and +6 for Huckabee and Paul.) For the Dems the shifts were +5, -6 and +2 for Clinton, Obama and Edwards respecively.
As Mark Blumenthal notes, the reasons for these discrepancies are largely matters of speculation. But the consistency of the ARG house effects are pretty clear in these data. The ARG results currently stand on the same side as their long term house effects: above trend for Clinton, Giuliani and McCain, and below trend for Obama, Edwards and Romney. Compared to other pollsters, these house effects for ARG appear to be the largest of any polling firm in Iowa.
