Political Arithmetik

We've Moved to PollsAndVotes.com

2011-09-17T18:52:00.001-05:00

In the beginning was PoliticalArithmetik, created the weekend Katrina hit New Orleans.

In 1996 I co-founded Pollster.com. In 2010 HuffingtonPost bought Pollster and I moved on.

In 2011 I've created PollsAndVotes.com, a site for more comprehensive analysis of polling and vote data. PollsAndVotes is ramping up for 2012. Come visit.

Meanwhile, this site is the archive for PoliticalArithmetik, my first love.

Undecided Voters and Racial Attitudes

2008-10-29T20:40:00.003-05:00

How will undecided voters break, and will racial attitudes color their votes?

We've seen an enormous amount of speculation but little evidence based on data, so let's try to tip the balance back to empirical evidence.

Thanks to the Diageo/Hotline tracking poll data, we can model individual vote choice and see what we would expect of undecided voters.

During October 3-11 our colleagues at the Diageo/Hotline poll included a racial attitude question we had previously used in the Big Ten Battleground survey in September and which NBC/Wall Street Journal used in January. That question was shown in both the earlier polls to have a statistically significant effect on vote choice, even after controlling for other political attitudes and demographics.

The question text is: "I'd like you to tell me whether you strongly agree, somewhat agree, somewhat disagree or strongly disagree with the following statement. ... African Americans often use race as an excuse to justify wrongdoing"

58% of the total sample, and 61% of whites agree either strongly or somewhat. (For comparison, 56% of hispanics and 40% of African Americans agree strongly or somewhat.)

I model the vote choice for those who expressed a preference with a model using a variety of attitudes and demographics, including favorability to Obama and McCain, party id, marital status, kids at home, education, race, age, sex, church attendance, region and urban, suburban or rural residence. Then I added the racial attitude responses from the "black excuse" question. To check against people hiding their feelings by refusing to answer the "black excuse" question I also included a variable to capture the effect of refusal to answer.

This model produces a predicted probability of voting for McCain or Obama, including predicted probabilities for those who had said they were undecided or who refused to respond to the vote question. From this we can estimate the likely vote of undecided, and compare the estimates to the responses of those who gave a vote preference in the survey.

Bottom line: Undecided and refuse to say voters are estimated to break 50% for McCain and 50% for Obama. As even as it gets. There is no evidence here of a large bias towards McCain that is hidden within the undecided respondents.

Nor is there evidence of a pronounced racial bias among these undecided voters as compared to the public at large. Among the undecided 27% strongly agree and 32% somewhat agree on the "black excuse" item. For the public as a whole 26% and 32% give the corresponding responses.

The model does a good job predicting survey response as well. 97% of both Democratic and Republican voters are predicted by the model to vote that way. For those who say they only "lean" towards one party or the other, 77% of Democratic leaners and 80% of Republican leaners are predicted to vote as they lean. The symmetry of results here suggests that there is not a visible bias in the model estimates for either party or for intensity of preference.

Finally, what happens if we ignore racial attitude and predict vote among the undecided without it? The split is 52% Obama to 48% McCain. So at most the impact of incorporating racial attitude in the model is a rise of 2% for McCain among undecided. Given the sample sizes involved, that is well within the margin of error. And if we take out candidate favorability from the model we get estimates of 52-48 without racial attitude and 53-47 with racial attitude.

So what can we conclude? There is no evidence of a hidden support for McCain among undecided voters. They split more evenly than does the "decided" pool of respondents, who split 54-46 in this sample (Oct 3-11) but that's well within normal expectations and is a modest difference in any case.

Second, the role of racial attitude is important at the individual level, but the aggregate consequence is extremely modest. Some are moved away from Obama yet others are moved towards him. And among the undecided, the distribution of opinion on this measure of racial attitude is virtually identical to that in the population.

In a year of endless discussion about racial effects there has been far more speculation and far less data analysis than is good for us. Let's put our data on the table before continuing to opine about this subject.

Cross Posted at Pollster.com

The Road to 270 for McCain

2008-10-17T00:07:00.002-05:00

(Click chart to see full size)

It takes 270 to win this game. McCain is behind in national polls, down 7.5 in our Pollster.com Trend Estimate. But as you know, this is a game won in the states. So what does it now take for McCain to eek out a 270-268 electoral vote win?

The chart above shows the states allocated to candidates based on my Political Arithmetik trend estimate. The classification of states is the same as Pollsters, but differ slightly in the order.

The states are ordered left to right according to the Obama minus McCain margin. Yellow states are classed as tossups but are allocated to candidates based on who is ahead in the trend, no matter how small that margin may be. The width of each state's block is proportional to the state's electoral vote.

With this classification, Obama has 364 electoral votes to McCain's 174.

So what would it take for McCain to come back at this point? Ohio and Florida, above all else.

Ohio and Florida are the largest states that are in Obama's row but still close. McCain led in both states in August and the first half of September. Without them, it is hopeless. With them, he still needs more, but they are the necessary conditions for a win.

Can he do it? The trends in Ohio and Florida offer a small glimmer of hope. While most states have continued to move in Obama's direction (see PA, MI and WI), these two have leveled off, and in Ohio moved back in McCain's direction.

No matter what, McCain has a long shot to get to 270. But The road has to go through Ohio and Florida and both states are looking better for him than any others he must win. What he is doing in Pennsylvania and Wisconsin is a mystery to me.

Tracking Poll House Effects

2008-10-12T12:31:00.003-05:00

There are six daily tracking polls currently reporting data, up from just two (Gallup and Rasmussen) during most of the year. How are they doing?

Compared to our Pollster.com trend estimate based on all public polls, not too bad. The trend based on trackers only is close to that for the all polls trend, with an average difference of just 0.35 percentage points, a very slight under-estimate of the Obama minus McCain margin. Recently the difference has been negligible, with most of this difference coming early in the year.

A bit of visual inspection shows the GW Battleground poll seemed more out of line until they shifted their party weighting plan after a few days. Likewise Hotline had a couple day "hiccup" but has returned to trend.

What about house effects? The range is moderate, from +4.3 points on the margin for Daily Kos, to -4.2 points for Zogby, though the latter has only just started polling so the confidence interval is wide.

National Sensitivity Comparison

2008-10-06T14:12:00.004-05:00

Now that you can play with the sensitivity of the trend estimate on our interactive charts at Pollster.com, the need for my sensitivity comparisons is somewhat less. But, it is interesting to see that just now the sensitivity used in our national trends makes very little practical difference.

My pals, "Ready-Red" and "Old-Blue" are in good agreement on both the magnitude of the Obama surge in the last three weeks, and the current standing of about a 7 to 7.5 point Obama lead.

Red had a moment of indecision last week when a couple of polls showed McCain ahead, but after barking in that direction, Red came back to Old Blue in seeing continued upward movement for Obama. You gotta love Red for his sensitive nose, but sometimes it distracts him from the big picture.

So here is a question to ponder. How much more upward lift can there be for Obama? He's already well beyond his previous best. Does anyone believe he really can hold a 10 point lead through the election? If not, then we should see some flattening out in the next few days, regardless of the debates.

Campaign '08 vs '04 and '00 Update

2008-10-06T13:41:00.003-05:00

The 2008 campaign had not seen a really big move in preferences until the financial crisis hit three weeks ago today. Since that time, the Obama-McCain margin has shifted almost 9 points in Obama's favor, converting a small McCain lead into a substantial Obama advantage.

This swing reversed the gains McCain made with the Republican convention and the week after during which he picked up about 4 points and took the lead for the first time since March.

I wrote earlier that we had not seen a move in 2008 as large as ones we saw in both 2000 and 2004. That is no longer true of 2004, though the current run is not yet as large as the one Gore mounted in 2000.

The Bush counter-assault in 2000, after Gore's surge, was almost eight points, and began at almost the same point in the campaign, about 57 days out.

Voters are making up their minds at about the same rate as they did in 2000. If this year follows that pattern, look for some serious decision making over the next two weeks.

Interactive charts now at Pollster.com

2008-09-27T10:48:00.007-05:00

If you can't resist the urge to play with the data, we have new charts at Pollster.com that can satisfy your need. These are interactive versions of the charts here and previously at Pollster. You can exclude pollsters or methods. You can change the sensitivity of the fit to either more sensitive or less sensitive than our standard fits here.

Here is a sample chart for Ohio, using our default values:

And here it is again but mail and Internet polls excluded and more sensitivity:

Give it a try and let us know if you run into any problems. You can embed the results of your experiments on your own page, though not the interactive features.

At the moment the embedded graphs are scaled at 450 pixels wide. To fit them here I edited the html code to make the width 400 and the height 308. I hope we can automate that soon, but meanwhile if your width is fixed and below 450, changing two lines in the embedded html will fix it for you.

These charts are available for a growing number of states at Pollster.com. Stop by and play!

Charles

Obama Recovery Across Red, Yellow and Blue States

2008-09-22T11:24:00.003-05:00

Obama has now recovered his lead in national polling, rising at or above his post-DNC, pre-RNC level. This sharp turnaround erases the very sharp pro-McCain/Palin convention bounce we saw in early September.

But the Obama recovery is not concentrated in Democratic states. As with the McCain advantage, this turn is visible in all three types of states-- Red, Yellow and Blue.

The biggest recovery is actually in the Red states, where McCain enjoyed his biggest bounce. Those states are not back where they started, or even a shade less pro-McCain, but they are a far ways from being "close".

The states we have classified as tossup or leans (what I'm calling yellow here, though on the map leans are light red or blue) have also seen a significant Obama recovery. The range of movement is rather modest, but the roughly three point McCain gain has now been balanced by a 3+ point Obama recovery in these most contested states, putting Obama up by just over a point.

Dem states showed a small move to McCain and have now seen an Obama recovery, though with a one point fall off most recently.

McCain Gains Not Limited to Red States

2008-09-15T15:06:00.006-05:00

Obama advisor David Axelrod is quoted in today's Washington Post article by Dan Balz and Peter Slevin:

"I think one of the things driving the national polls is that the red states are redder," said David Axelrod, one of Obama's closest advisers. "In the battleground states, the race has held pretty firm."

An interesting claim. Let's take a look at the data based on state polls, rather than national.

Among the strong Republican states, McCain has gained more than 8 points over Obama since shortly before the conventions, turning a 14 point lead into a 22.5 point margin, a huge gain.

Among the strong Democratic states, the effect of the conventions is a tiny 2 point move in McCain's direction, from an Obama lead of 12 points before to 10 points now.

But the rest of the states, rated lean or toss up, have also shown movement. These swing states had a 1.5 point Obama lead before the conventions, and that has now turned into a 3 point McCain lead, a 4.5 point shift.

So Axelrod is right that the biggest gains for McCain have come in the reddest of states, and those may influence national polling. But the evidence doesn't back his second claim, that the battleground has held firm, unless of course you mean they are still battleground states. But now battlegrounds that on balance favor McCain rather than favor Obama as they did before the conventions.

One caution: the lean and tossup states are themselves heterogeneous, so a single trend estimate such as the 4.5 McCain lead here is itself a simplification. If you wanted to focus on the six or eight states that probably hold the key to the electoral vote, you could slice this more finely.

We currently rate eight states as pure tossup: NH, VA, OH, MI, CO, NM, NV and MT. (Note the last has few polls and the latest 9/8 shows a 53-42 McCain lead. But it does fit our statistical criteria, and Montana was listed by the Obama campaign as a target state.)

When we fit the data to just these eight tossup states, we see a 3.5 point move in McCain's direction, from a 2 point Obama lead to a 1.5 point McCain lead. Only a point less shift than among all lean and tossup states.

No matter how you slice it, the battleground states have a lot of battle left in them, and campaign events are having effects across all states, though larger in some than others.

Alaska Post-Palin

2008-09-12T14:13:00.003-05:00

Alaska has long been a solidly "red" state but this year the troubles of the state's Senator Ted Stevens and Representative Don Young on top of Republican party "brand" troubles suggested there might be an opportunity for the Obama campaign which spent $88,000 in advertising in the state in June and July according to the Wisconsin Advertising Project.

And then came Sarah Palin to the Republican ticket. The impact is obvious in the chart above.

While the state was classified as "lean Republican" prior to the Palin Pick, all but one poll in the state showed McCain ahead. Now there is no doubt.

The politics are obvious, of course, but it is a nice example of how a political event can be instantly reflected in the polling, and bring our trend estimator to a sharply different conclusion.

State and National Polling Post-Convention

2008-09-12T13:28:00.003-05:00

The gains of the Republican ticket continue in both state and national polling as more post-convention state polls become available. We now have 26 states with a post-convention poll and the results mirror the sharply pro-Republican movement seen in national polling since Sen. McCain's acceptance speech.

The McCain lead is now about 3 points in national polling and is just under 4 points for those states with post-convention polls.

An important point is that the states with post-convention polls are not a representative sample of all states, though the differences between states without a recent poll and those with new polls is not large. States without new polling have averaged 2.1 percentage points more pro-Obama than those states with new polling. However, if we plot the trend lines in the chart for all states (with or without new polls) and for those with post-convention polls, we see the two trends have followed similar if not identical trajectories.

Since our last update of this comparison of state and national trends the addition of new states has brought the state trend line a bit below the national polls trend, something not so visible in the earlier post. This difference reflects the selection of states that have new polling, rather than shifts in the vote margin in the states, except of course for the brand new polls.

The bottom line is that the swing towards the Republicans remains strong in both state and national polls , amounting to a 7 point national swing and a 5 point swing based on the state polls. In an earlier post, I pointed out

The bottom line is neither campaign should be complacent or despondent. There is a lot of time left and recent history shows that both up and down swings of 6-9 points are entirely plausible.

Right now that magnitude of swing is looking about right.

The states with post-convention polls are: Alabama, Alaska, Colorado, Florida, Georgia, Idaho, Maine, Maryland, Michigan, Mississippi, Missouri, Montana, Nevada, New Hampshire, New Jersey, New Mexico, North Carolina,North Dakota, Ohio, Oklahoma, Pennsylvania, Virginia, Washington, West Virginia, Wisconsin, Wyoming.

Ins and Outs of Polling

2008-09-10T17:38:00.003-05:00

I did an hour on Minnesota Public Radio on September 10 discussing polling techniques and issues. Here is a link to the audio. Good callers!

The focus on polling starts at 11:00 minutes into the show.

Post Convention Bounce in the States

2008-09-10T15:54:00.003-05:00

The McCain campaign has gained significant support in national polling since the end of the Republican Convention, but what about the state polls? Has the shift also been reflected there?

State pollsters appeared to go on vacation for the conventions, with very little new polls during the two weeks of conventions (and the week before). Now the pollsters are back, tanned and rested and ready to go. We've added 17 new state polls since the RNC ended, and while we'd love to see more, it is enough to get started with some analysis.

The chart above shows the national trend in blue and the trend based on those states with post-convention polls in purple. Over the course of the year, the two trends have followed each other rather well with some small differences in details but qualitatively similar patterns of up and down movement.

Now in the post-RNC period, the states with new polls match the national polls quite closely, both giving estimates of about a one point McCain lead, with the states maybe a shade less than that.

This post-convention bounce may or may not last, but at the moment the evidence is that it is moving across the states (and these are mostly competitive states) at about the same rate as it is for the national polls.

The state polls show no positive DNC upward bounce, but that's because there were practically no new polls during that period, so don't jump to that conclusion. It is an artifact of no polls.

States for which we have new polls are Colorado, Florida, Maryland, Michigan, Montana, New Jersey, North Carolina, Ohio, Oklahoma, Pennsylvania, Virginia, Washington and Wisconsin.

Republican Bounce and Trend Sensitivity

2008-09-08T12:51:00.004-05:00

The post-convention bounce is now moving in the Republican direction, but with an enormous spread in estimates. A Gallup/USA Today (9/5-7) has an enormous 10 point McCain lead over Obama, 54%-44%. In contrast, Gallup's tracker over the same days shows a 5 point McCain lead, 49%-44%. Now would be a good time to note that the tracker is a registered voter (RV) sample, while the Gallup/USAT is a likely voter (LV) sample. LV samples typically are more favorable for Republican candidates, so at least some of this difference is probably due to these different sampling frames. We'll no doubt be talking a lot about this issue in days ahead.

But other polls on the same days show a tied race. Diageo/Hotline has the race 44%-44% and CNN has it 48%-48%. And Zogby's Internet poll done 9/5-6 puts the race at 50%-46%.

All of these are much better for McCain than the 5-9 point Obama leads we saw in the immediate aftermath of the Democratic convention.

So it looks like both parties got nice convention bounces.

Our trend estimate is still hungry for more data. The standard, blue line, estimate is now less persuaded that Obama had a convention bounce OR that McCain is getting one either. That is standard behavior of our estimator which is designed to be a bit conservative when faced with conflicting polls and short term changes of trend.

But that is why we have our "sensitive" estimator for comparison. The red line is a trend estimate that is about twice as sensitive a the blue line. It is considerably quicker to respond to short term changes and to fewer polls. The down side is it will often chase random noise.

Since there is good reason to believe convention bounces are real, it is reasonable to think that the red line's indication that the race has indeed tightened is probably a real signal in the data, and not just noise. On the other hand, the Gallup 10 point McCain lead is out of the range of any other current polling data. So "red" may be chasing that outlier just a bit more than is good for him. As the figure makes clear, red and blue usually agree quite closely after enough data are in hand, but can diverge especially when data are sparce.

A prudent approach is to wait for a few more post-convention and post-weekend interviewing polls to see just how big and how sustained the RNC bounce is. But both estimators agree we have ourselves a real horserace now.

State Battlegrounds and Home Grounds

2008-08-29T07:39:00.003-05:00

A quickie from Detroit Metro Airport.

Mark Blumenthal reported on an interview with Obama campaign manager David Plouffe yesterday at Pollster. Plouffe discussed the 18 states the Obama campaign sees as their target states, and Mark reported what states those were in his post.

Here we take a quick look at the polling in those states. The chart above is sorted by the Obama minus McCain margin, and shows the 95% confidence interval. The dot size is proportional to electoral vote.

Below I show the status of the states based on our polling categorization of each state.

Time to run for the plane.

McCain, Obama and Clinton Favorability

2008-08-27T08:13:00.004-05:00

A little interesting movement in views of the candidates has taken place since the end of the primaries in June. All three candidates, McCain, Obama and Clinton, have seen rises in their favorable ratings and an initial decline in unfavorable views though with a slight upturn recently. McCain and Obama are enjoying essentially identical ratings, with 60% favorable and only 35% unfavorable. Even after a significant amount of negative portrayals of him in RNC and McCain ads, Obama's rating has risen over the summer, and so has McCain's. (According to the Wisconsin Advertising Project, which monitored and coded all 100,000 ad airings in June and July, one third of McCain's ads contained negative information about Obama and 100% of RNC ads were negative. In the same two months, 10% of Obama's ads mentioned McCain.)

Whatever happens after the conventions, both candidates enjoy an enviable standing with voters as attractive figures instead of a pair of lesser evils. The fall capaign may alter this, but even after a hard fought primary season the nominees remain attractive figures.

Meanwhile, Senator Clinton has also enjoyed an upturn in favorable ratings and a decline in unfavorable ratings since the end of the primary season. While improved, Clinton remains a more polarizing figure than either McCain or Obama, with slightly lower favorable but noticeably higher negative ratings.

Senator Clinton is far more popular among Democrats than among either Independents or (especially) Republicans. In that sense, her speech to the Democratic Convention last night was an example of speaking primarily to the party and her supporters, rather than to the broader public. The contast between former Virginia governor and now Senate candidate Mark Warner's speech and Clinton's is a good example of this difference. Warner stressed unifying themes and appeals across political groups, which was greated warmly but which fell short of electrifying the Democratic delegates. In contrast, Clinton played to the party and produced a predictably enthusiastic response within the DNC convention hall. Conventions contain both elements. Monday, the party celebrated Sen. Kennedy's life and family legacy, primarily an inside the family affair, perhaps touching some independents but not likely to attract Republicans. In contrast Michelle Obama's speech could have easily been given at the Republican convention, with its themes of family, hard work, pulling oneself up from working class circumstances. Hers was a speech designed to reach out beyond the party.

The one remaining question from the Clinton speech is whether her supporters also resepect her enough to follow her lead. For Clinton to be a power in the party includes the requirement that she be able to deliver her supporters for Obama. If any significant number of her supporters refuse to be delivered, they reduce her status as a result. This is hard to judge from the cable news coverage, who can easily find individual delegates willing to say they are unpersuaded. But what effect the Clinton speech has with her supporters outside the convention hall will be critical.

How Pollsters Affect Poll Results

2008-08-24T15:29:00.009-05:00

Who does the poll affects the results. Some. These are called "house effects" because they are systematic effects due to survey "house" or polling organization. It is perhaps easy to think of these effects as "bias" but that is misleading. The differences are due to a variety of factors that represent reasonable differences in practice from one organization to another.

For example, how you phrase a question can affect the results, and an organization usually asks the question the same way in all their surveys. This creates a house effect. Another source is how the organization treats "don't know" or "undecided" responses. Some push hard for a position even if the respondent is reluctant to give one. Other pollsters take "undecided" at face value and don't push. The latter get higher rates of undecided, but more important they get lower levels of support for both candidates as a result of not pushing for how respondents lean. And organizations differ in whether they typically interview adults, registered voters or likely voters. The differences across those three groups produce differences in results. Which is right? It depends on what you are trying to estimate-- opinion of the population, of people who can easily vote if the choose to do so or of the probable electorate. Not to mention the vagaries of identifying who is really likely to vote. Finally, survey mode may matter. Is the survey conducted by random digit dialing (RDD) with live interviewers, by RDD with recorded interviews ("interactive voice response" or IVR), or by internet using panels of volunteers who are statistically adjusted in some way to make inferences about the population.

Given all these and many other possible sources of house effects, it is perhaps surprising the net effects are as small as they are. They are often statistically significant, but rarely are they notably large.

The chart above shows the house effect for each polling organization that has conducted at least five national polls on the Obama-McCain match-up since 2007. The dots are the estimated house effects and the blue lines extend out to a 95% confidence interval around the effects.

The largest pro-Obama house effect is that of Harris Interactive, at just over 4 points. The poll most favorable to McCain is Rasmussen's Tracking poll at just less than -3 points. Everyone else falls between these extremes.

Now let's put this in context. We are looking at effects on the difference between the candidates, so that +4 from Harris is equivalent to two points high on Obama and two points low on McCain. Taking half the estimated effect above gives the average effect per candidate. The average effects are at most 2 points per candidate. Not trivial, but not huge.

Estimating the house effect is not hard. But knowing where "zero" should be is very hard. A house effect of zero is saying the pollster perfectly matches some standard. The ideal standard, of course, is the actual election outcome. But we don't know that now, only after the fact in November. So the standard used here is the house effect relative to our Pollster Trend Estimate. If a pollster consistently runs 2 points above our trend, their house effect would be +2.

The house effects are calculated so that the average house effect is zero. This doesn't depend on how many polls a pollster conducts. And it doesn't mean the pollster closest to zero is the "best". It just means their results track our trend estimate on average. That can also happen if a pollster gyrates considerably above and below our trend, but balances out. A nicer result is a poll that closely follows the trend. But either pattern could produce a house effect near zero. For example, Democracy Corps and Zogby have very similar house effects near -1. But look at their plots below and you see that Democracy Corps has followed our trend quite closely, though about a point below the trend. Zogby has also been on average a point below trend, but his polls have shown large variation around the trend, with some polls as near-outliers above while others are near outliers below the trend. The net effect is the same as for Democracy Corps, but the variability of Zogby's results is much higher.

Incidentally, the Democracy Corps poll is conducted by the Democratic firm of Greenberg Quinlan Rosner Reserch in collaboration with Democratic strategist James Carville. Yet the poll has a negative house effect of -1. Does this mean the Democracy Corps poll is biased against Obama? No. It means they use a likey voter sample, which typically produces modestly more pro-Republican responses than do registered voter or adult samples. Assuming that the house effect necessarily reflects a partisan bias is a major mistake.

How can you use these house effects? Take a pollster's latest results and subtract the house effect from their reported Obama minus McCain difference. That puts their results in the same terms as all others, centered on the Pollster.com Trend Estimate. This is especially useful if you are comparing results from two pollsters with different house effects. Removing those house differences makes their results more comparable.

What impact do house effects have on our Pollster.com Trend Estimate? A little. Our estimator is designed to resist big effects of any single pollster, but it isn't infallible, especially when some pollsters do far more polls than others or when one pollster dominates during some small period of time. We can estimate house effects, adjust for these, and reestimate our trend with house effects removed. The result runs through the center of the polls, but doesn't allow the number of polls done by an organization to be as influential.

The results are shown in the chart below. The blue line is our standard estimator and the red line is the estimate with house effects removed. Without house effects the current trend stands at +2.0 while ignoring house effects produces an estimate of +1.7. A little different, but given the range of variability across polls and the uncertainty as to where the race "really" stands, this is not a big effect.

The impact of house effects isn't always this small. Looking back along the trend we see that the red and blue lines diverged by as much as 1 point in late June, an effect due significantly to the large number of Rasmussen and Gallup tracking polls during that time and few polls with positive house effects in that period. A smaller but still notable divergence occurred in late February and early March.

The bottom line is that there are real and measurable differences between polling organizations, but the magnitude of these effects is considerably less than some commentary would suggest. Many of the house effect estimates above are not statistically different from zero. Even ignoring that, the range of effects is rather small, though of course in a tight race the differences may be politically important. Finally, the effects on our Pollster.com Trend Estimate is detectable but does not lead to large distortions, even if we can see some noticeable differences at some times.

The charts below move though all the pollsters and plots their poll results compared to the standard trend and the trend removing house effects. Pollsters with fewer than 5 polls are all lumped together as "Other" pollsters. Once they get to our minimum number of polls, we'll have house effects for them too.

Age, Turnout and Votes

2008-08-11T15:41:00.003-05:00

It's all about who votes. Those that do win. Those that don't lose. The chronic losers in American politics are the young who famously turn out at low rates election after election.

This year, those young people are of great interest. Allegedly they will be mobilized in huge numbers, and allegedly they will vote strongly for Barack Obama. The latest available Gallup weekly estimate (July 28-Aug 3) shows Obama leading 56%-35% among 18-29 year olds, while McCain leads 46%-37% among those 65 and older.

But will the young vote? And how much difference does it make when they don't?

The chart above shows the turnout rate by age for 2000 and 2004, based on the Census Bureau's "Current Population Survey (CPS)", the largest and best source of detailed data on turnout. The most striking result is just how low turnout is among those under 30 compared to older voters. No age group 18-29 managed to reach 45% turnout in 2000, and only two made it in 2004. Not one single age group over 30 fell so low in either year. Despite a little noise for each group, the pattern is a strong rise in participation rates with every year of age at least until the late 60s, after which there is some decline. Yet even among those 85 and over the turnout rate remains above 55%, more then 10 points higher than among their 20-something grandchildren and great-grandchildren.

The second striking feature of the chart is that the young can be mobilized a bit, under the right circumstances. Turnout among those under 30 rose significantly in 2004 compared to 2000. While turnout went up among all age groups, the relative gain was clearly greater among those under 30. While mobilizing the young is difficult, these data show that it is possible to get significant gains, at least relative to past turnout.

Even so, the "highly mobilized" 20-somethings of 2004 still fell behind the turnout of their 30-something older siblings. A supposed Obama-surge among the young may still not catch up with those even a bit older.

The irony is that the young are a large share of the population, but not of the electorate. The chart below shows the population by age in 2004 (it shifts a little by 2008 but not enough to change the story.)

The "boomers" in their 40s and 50s remain the largest group, but for our purposes there are two important points. Those under 30 make up a substantial share of the population, while those 60 and over represent a substantially smaller share at each age.

In 2004 those 18-29 were 21.8% of the population, while those 58-69 were just 13.2%. Add in the 11.5% 70 and up, and you get just 24.7% of "geezers" over 58 vs. 21.8% of "kids". But the sly old geezers know a thing or two about voting. Shift from share of the population to share of the electorate and the advantage shifts to the old: 18-29 year olds were just 16% of the electorate in 2004, while those 58-69 were an almost equal 15.9%. Add in the 70+ group at 13.4% and the geezers win hands down: 29.3% of voters vs 16% for the young. That difference is the power of high turnout. It goes a long way to explaining why Social Security is the third rail of American politics.

High turnout buys "over-representation". Divide share of voters by share of the population and you get proportionate representation. A ratio of 1.0 means a group votes proportionate to its size. Values over 1 are overrepresented groups. In 2004, for example, 55 year olds were represented 20% more than their population would suggest, with a 1.2 score. The youngest voters, 18 year olds, had an abysmal representation rate of 0.49 in 2000, less than half their share of the population.

While turnout rises with age, it is not until we hit 40 or so that we reach "fair" representation (1.0). After that, every age group is over-represented in the electorate. Less than 40, and every age group is under-represented. (Two small exceptions-- so sue me.)

So what are the implications? If you gave me a choice of being wildly popular with the young or moderately popular with the old, I'd take the old any day. They are far more reliable in voting, and while their population numbers are small they more than make up for it in over-representation thanks to turnout differences.

There is much conversation about "youth" turnout this year. Perhaps we will indeed see another rise, as we did in 2004. But unless something truly unprecedented occurs, no one can win on the young alone. The gap in turnout is simply too large.

But is age destiny? If there were constant differences in partisan preference by age, then perhaps so. But there aren't. Despite being supposedly "old and set in their ways", those 60 and up shifted their votes more than any other age group between 2000 and 2004. In 2000, the 60+ vote went to Gore by a 4 point margin. In 2004, however, those 60+ went for Bush by 8 points. That net 12 point swing, multiplied by their over-representation means a lot.

The 20-somethings also shifted, from +2 for Gore to +9 for Kerry. Coupled with their surge in turnout, the younger voters kept Kerry close in 2004 when he was losing in every other age category. But it wasn't enough to win.

The Obama campaign may be right that they can gain votes by mobilizing the young. But the old play a bigger role in elections, and they are not imovable in their vote preferences. Indeed, they make the youngest group seem a bit static by comparison. It is not the candidate's age that will be the key to winning the votes of those 60 and over. Issues and personality will play a large role. Any candidate would be well advised to recognize that the dynamic swings among older voters coupled with their substantial over-representation makes them a potent force for electoral change.

P.S. At the cross post to Pollster.com, Michael McDonald noticed that my plots were based on population percentages rather than percentage of citizens. That's a good catch and I probably should have used citizens in the first place. But the qualitative results don't change at all, so my story remains exactly as it is above. The precise percentages quoted do shift a little bit, but not in ways that change any of the conclusions. Therefore I've left the text above as it was, but append the first three figures, revised to include only citizens, here for completeness. Thanks to Michael for pointing this out.

Polling Trends in 2008 vs '04 and '00

2008-08-04T15:08:00.003-05:00

The most common description of polls is that they are snapshots, not predictions. A good way to look at that in the 2008 election is to compare the '08 campaign with the two that came before.

The chart above shows the trend estimates for each of the last three presidential campaigns. I'm plotting the estimated margin between the two candidates, Dem minus Rep, for each year.

With 93 days to go until the 2008 election, Obama holds a 3.3 point advantage over McCain, though that has been eroding over the past six weeks. If we put a confidence interval around today's estimate, we get a race that is just barely leaning Democratic.

But what about the future? The dynamics of the next 92 days are all important for where we stand on November 4. Since we can't foresee those 92 days yet, let's see what happened during the same time in 2000 and 2004. That gives us a better idea how much change we might anticipate in the next three months.

In 2004, Kerry slowly built a 2 point lead by this time, and held a small lead through much of the summer. But then the race took a sharp turn, with Bush making a 6 point run, taking a four point lead with 50 days to go. Kerry gained back 3 points of that in the polling, but less than 2 points of it in the actual vote, losing by a 2.4 point margin.

In 2000, Bush led in most of the early polls, holding a 6 point lead with 107 days to go. Then Gore moved sharply up, erasing Bush's lead and then adding a 3 point lead for Gore with about 56 days left. Bush promptly reversed Gore's gains with a six point move in the GOP's direction, and led by about 3 points over the last three weeks of the campaign. Of course, the 2000 polls were misleading in predicting a Bush win. Gore won the popular vote by 0.6 points.

So far in 2008, Obama has enjoyed a run up of 5.5 points since his low point in late March. That run is on a par with Bush's in 2004 but still a bit less than Gore's 9 point run in 2000, and on par the Bush's 6 point rebound that year.

Judging from the dynamics we've seen in the past it is quite reasonable to expect the current trend to shift by half-a-dozen points. August and the conventions have been periods of substantial change in both previous elections, so if history repeats itself the next 4 or 5 weeks should be pretty interesting.

The bottom line is neither campaign should be complacent or despondent. There is a lot of time left and recent history shows that both up and down swings of 6-9 points are entirely plausible.

As a P.S. here are the three campaigns with educational confidence intervals around them.

The current 2008 estimate is just barely inside the "lean Dem" range, and will move to toss up if the current trend continues for another couple or three polls.

The 2004 estimate was pretty close to the outcome which was well within the 68% confidence interval around the trend.

The polls in 2000 were troubling for having the wrong popular vote winner, but even there the outcome was inside the 95% confidence interval. With races as close as the last two, it is worth appreciating just how wide those confidence intervals are.

Our efforts to characterize races rely on the best estimates of those confidence intervals, but it is all too easy to focus on who's ahead and not remember how much uncertainty there is. That uncertainty is both about where the current estimate says the race stands today and about how the race may change in coming weeks. The data here show that unless one candidate builds a bigger lead than either has held so far, the uncertainty remains pretty big.

Note: My trend here is slightly different from the Pollster National trend because I'm working off the difference between candidates, not each trend separately, and because I've made 2008 comparable to 2000 and 2004, just a slightly different amount of smoothing compared to Pollster's standard estimator this year. None of those differences change the qualitative picture or shift the magnitude of changes I cite above.

Trends in Party Identification in Wisconsin

2008-06-13T11:15:00.005-05:00

This week my colleague Ken Goldstein and I conducted a Wisconsin statewide survey sponsored by the UW Department of Political Science and WisPolitics.com. So fair warning that I'm a party to this survey rather than an independent observer.

A number of people have commented on the party identification balance in the survey: 38% Dem, 24% Rep, 29% Independent (37% Independent when "no preference/other" are allocated to independent. When this group is asked how they "lean", very few insist on some other party, so this allocation makes sense.) See Alan Reifman's blog on weighting and party id for a good example and discussion of broader issues of weighting to party id.

I want to point out two things here and put our data in the context of other polls in Wisconsin.

The chart above shows party identification trends since 2000 using data from three sources that have done frequent polling in the state. What we see is a relatively stable Dem/Rep parity from 2000-2004, with Dem ID falling a bit around 2004 while Reps moved up slightly.

Starting in 2005, however, there is an initially slow but then sharper shift in partisanship. Republican ID declines from about 30% to about 24% today, while Dem ID rises from about 30% to nearly 40%. After an initial surge of independents, that group has recently fallen off a bit. (You have to squint a bit to see WPRI and Badger after 2005, but they are close to the trend lines during this period, so the changes are not just a matter of house effects or phone vs ivr methods. WPRI, for example, has Rep ID moving from 33% in 2004 to 28%, 26% and 25% in 2005-2007. Their Dem ID rises from 30%-33%-34% then falls to 29% over the same period. The final 29% is a large discrepancy from the trend, of course.)

We did not weight our survey to party identification, and these trends help explain why we have reservations about doing that. While relatively stable, party id does move over time, and by a fair bit, as you can see here. But that said, our unweighted results turn out to be quite close to the estimated trends in partisan categories in any case.

The second point is to compare these trends with those in exit poll measures of party id. In 2000, the VNS Exit poll put Wisconsin pid at 37% Dem, 32% Rep and 31% Ind. This shifted in 2004 to 35% Dem, 27% Ind and 38% Rep. But in 2006 the exit polls found that the balance was 38% Dem, 34% Rep and 27% Ind. Those values all show a smaller share of independents at the polls on election day compared to the polling trend, but that is to be expected given differences in turnout between partisans and independents. The size of the party ID groups grows as a result, but the balance between them is in line with what we see in the trends in the polls, though certainly not an exact match. The polls, after all, are of either adults or likely voters, while the exits are by definition a measure of who actually showed up on election day.

For 2006, the Dem exit percent and the Dem trend estimate are a close match. Republicans gain in the exits, by about 6 points over the 2006 trend estimate. If that holds for 2008, we might expect an electorate more like 38% Dem and 30% Rep. Of course both parties will have very active "ground games" and GOTV efforts to try to change those numbers.

While I'm certainly happy that our party id balance is so close to the trend in all the other polling, the more important point is that party id in Wisconsin has shifted quite a bit over the past four years. The coming campaign may alter that, possibly bringing disappointed former Republicans back home, for example. Likewise a Republican advantage in turnout could bring the exit polls back to closer balance. But as the data show, today the GOP is at the worst disadvantage the state has seen in over eight years.

Let me conclude with a bit of description of the polls used here.

Wisconsin Policy Research Institute ("WPRI") has done some of the longest running polls in the state, usually two a year. Their data here is taken from their annual estimates, which I assume pool the two surveys though they don't say so explicitly. WPRI describes itself as "Wisconsin's Free Market Think Tank".

The "Badger Poll" is conducted by the UW Survey Center. They did more extensive polling in 2002-04 but now do about two polls a year.

SurveyUSA is a well known national pollster that uses "Interactive Voice Response" (IVR) automated interviews. SurveyUSA has done monthly polling in the state since 2005, providing some of the best data on state trends in approval of elected officials and as a byproduct have an excellent data series of party ID.

Finally, there is our new Department of Political Science/WisPolitics poll. Ours uses a commercial call center, not the UW Survey Center or undergrads in a class calling for a grade. WPRI, Badger and our poll all use live interviewers, SurveyUSA uses IVR. Most of these surveys are in the 500-600 respondent range.

Gay Marriage Support and Opposition

2008-05-21T09:07:00.003-05:00

Marriage for gay and lesbian couples has been a hot button issue, most especially so in the 2004 election cycle when 11 states considered and passed referendums banning (in various ways) same-sex marriages. In 2006 an additional 8 states voted on marriage ballot measures, with only Arizona defeating the proposal. In all, 41 states have statutes defining marriage as "between one man and one woman", and 27 states have put that definition into their constitutions. Only five states currently have no law banning same-sex unions (MA, NJ, NM, NY, RI). In 2008, Florida will have a "defense of marriage" amendment (DOMA) on the ballot, while California is awaiting certification of a ballot proposal and Arizona may reconsider its 2006 initiative (currently awaiting state Senate approval). (An excellent summary of the status of same-sex marriage in the states is available here.)

Despite this overwhelming majority among other states, the California Supreme Court last week ruled that the state cannot constitutionally withhold the right to marriage from same-sex couples. (Text of the ruling is here. The LA Times initial report on the decision is here.) Supporters of gay marriage hailed the decision as a breakthrough for fundamental rights, in line with the same California Court's decision in 1948 striking down laws banning inter-racial marriage. Opponents of gay marriage argued the ruling puts the issue squarely back on the table for 2008 and confirmed the opponents argument that only constitutional amendments can prevent courts from overturning popular opinion on this issue. In 2000 California passed, by a 61%-39% majority, Proposition 22 affirming that "only marriage between a man and a woman is valid and recognized in California."

California has one of the strongest domestic partnership laws in the nation, so the Court's decision has the effect of ruling that by withholding the designation "marriage", such domestic partnership laws still fall short of the equal treatment required by the state constitution.

The California decision follows the Massachusetts Supreme Court's ruling of November 18, 2003 which ultimately made Massachusetts the first, and so far only, state to legalize same-sex marriage. (Rhode Island law recognizes same-sex marriages from other states.) Subsequently, the state Supreme Courts of New York, New Jersey and Washington have each declined to find a constitutional right to same sex marriage. Four states have civil union laws providing full state-level spousal rights (CT, NJ, NH and VT) while six have domestic partnership laws that provide varying degrees of spousal rights (DC, HI, ME, OR, WA plus the California law at issue in this decision).

In light of the California decision, let's take a look at public opinion on same-sex marriage and how opinion has responded to past events.

A typical question asks "Do you strongly favor, favor, oppose, or strongly oppose allowing gays and lesbians to marry legally?" (This is the form used by the Pew Research Center polls. There is considerable variation in question wording, but most polling has used a similar dichotomy between favoring gay marriage or opposing it. I've collapsed "degrees" of support or opposition into a dichotomous measure for all polls.) The earliest use of such a question I could find dates back to September 1985, but it was not until 1992 that the question began to be asked regularly. There was a flurry of interest in the question following the Massachusetts ruling and during the 2004 election campaign.

If we rely on that first poll alone, in 1985 82% of the public opposed same sex marriage, while only 11% supported it. By the early 1990s, when the data become richer, opposition was at about 65% while support stood at about 28%. Congress passed, and President Clinton signed, the federal "Defense of Marriage Act" in September 1996, but public opinion trends seem not to have noticed at all, neither rising nor falling around that time. By the week of the California ruling, May 15, 2008, opposition had declined to about 55% while support had grown to 40%. The net effect of some 16 years of public debate was a 10 point decline in opposition and a 12 point rise in support.

But that trend was not uniform. The Massachusetts ruling, and the 2004 election campaign, coincided with a sharp, if relatively short term, disruption of the previous slow but steady decade long shift of opinion. The Massachusetts Court decision placed the issue squarely on the public radar, and the 11 state ballot proposals in the 2004 election created the setting for public debate and political exploitation of the issue.

During the year from November 2003 to November 2004, opposition to same-sex marriage rose by five points, from 55% to just over 60%. Meanwhile support fell by about eight points, from 38% to 30%, then rebounded by a point or so by election day. (These shifts slightly predate the Massachusetts decision, probably reflecting the increased visibility of the issue prior to the Court's ruling.) The impact of these shifts and of the 11 referendums that were passed on the presidential election remains debatable. Initial punditry credited the referenda with helping defeat John Kerry, especially in Ohio. More careful subsequent analysis doubts much of an effect, however.

These sharp shifts in trend reversed direction immediately following the 2004 election, but took more than two years to return to pre-2004 levels. Support returned to 2003 levels in mid-2007 while opposition has only now, in May 2008, declined back to where it stood in mid-2003. Despite this slow recovery from the 2004 "shock", the 2005-08 trend lines make it clear that public opinion returned to its previous trajectory of slowly rising support and declining opposition in the aftermath of 2004. It is also interesting that the 2006 elections, with 8 states voting on referenda, made no discernible difference to the post-2004 trend. In part this may reflect the more limited number of states, but it also reflects some decline in the saliency of the marriage issue.

The California ruling, and the likely campaign over a proposition there to modify the state constitution this fall, will test whether increasing the salience of the issue will result in a replay of the 2003-04 dynamics, with opponents stimulated and supporters in retreat, or if the 2006 experience means that the issue is no longer the motivator it was in 2004. The 2003-04 data clearly show the potential for sharp changes when the marriage issue becomes extremely salient. That the fight will take place in the most populous state in the Union also guarantees national exposure. However, the fact that most states have already settled this issue through law or amendment, and that only three states (so far) are on track to have proposals on the ballot, means that the issue is more localized than it was in 2004.

Opinion now is not much different from where it was in mid-2003, so a similar reaction is possible but there may be an element of "been there, done that" as well. The novelty of the issue is surely much reduced now than it was five years ago, though the record of referenda passing in 7 of 8 states in 2006 certainly demonstrates that opposition to same-sex marriage remained strong even in a very pro-Democratic election year. (Wisconsin, for example, reelected a Democratic governor and flipped a House seat to the Democrats but also modified its constitution to ban same sex marriage or anything substantially equivalent to marriage.)

The big question is whether the marriage issue has any carry over to the presidential vote in 2008. Democratic politicians, including Senators Clinton and Obama, have tried to insulate themselves by opposing gay marriage. Instead, they support civil union or domestic partner legislation. Senator McCain opposes same sex marriage and opposes legal recognition of same sex partnerships, but also opposes a federal constitutional amendment. This line of debate, with both parties opposing marriage, but with Democrats willing to support some legal recognition short of marriage, reflects another way to framing the question, one that is significantly more favorable for limited rights for gays and lesbians.

(Note: This chart is scaled the same as the previous chart so the dynamics and time frame are directly comparable. The large white space prior to 2000 reflects the politically relevant point that in that time period the "civil union" option was not prominent enough to be included in polling questions.)

Beginning in 2004 (with one early exception in 2000), polling organizations began asking a question with three alternatives. The CBS News question wording is representative:

Which comes closest to your view? Gay couples should be allowed to legally marry, or gay couples should be allowed to form civil unions but not legally marry, or there should be no legal recognition of a gay couple's relationship?

When the "civil unions" option is added, opposition to gay rights drops significantly from about 55% to 40%. Likewise, support for gay marriage drops from 40% to 29%. The "comfortable" middle ground is then some 26% who are willing to support civil unions so long as they fall short of "marriage".

This "half a loaf" approach is acceptable to only some in the gay rights community, but it is precisely the politically acceptable position that Democratic politicians think can move them from the losing side of public opinion to the winning side. If we add supporters of marriage to supporters of civil unions, we get the chart below.

This is now a near mirror image of the balance of opinion in the first chart. Now about 53% support either civil unions or marriage, and a minority of 40% oppose any legal rights for gay and lesbian couples. By assuming supporters of marriage will not punish them for the expedient support of only civil unions, Clinton and Obama (and many other Democrats) have tried to turn a losing position into a winning one.

The remaining uncertainty is whether opponents of any legal recognition are more intense than the supporters of civil unions. If so, then opposition groups may still win the battle between intense minority and lukewarm majority. On ballot propositions, the record is strongly in favor of the opponents of marriage and in some cases of civil unions as well.

The Clinton-Obama position will certainly not win over opponents of any form of legal recognition for gays, but then they probably wouldn't win many such voters in any case (an exception is African-Americans, many of whom are quite opposed to marriage or civil unions.) Whether their position provides them popular support in response to attack ads on this issue remains to be seen.

NC and IN Final Sensitivity Comparison

2008-05-06T14:27:00.002-05:00

Both standard and sensitive estimators are agreed in North Carolina. In Indiana there is a little bit of room between them, but not enough to affect conclusions about the probable outcome (if the polls are right!)

The gyrations the Indiana sensitive estimator for Clinton goes through, thanks to variability in polls and relatively few polls, is a good warning that the sensitive estimator may just be a bit too ready to chase after noise.

NC and IN Final Pollster Comparisons

2008-05-06T14:07:00.004-05:00

With the last of the preelection polls in, we can now do our "apples to apples" comparison. Follow each pollster in the charts to see who's high, who's low and who has jumped around.

Note this is for the Obama minus Clinton MARGIN (which makes it easier to plot all the polls in one, still jumbled, chart.)

And check back tonight as the votes roll in to see who nailed it and who missed. In North Carolina all agree on the winner, only the margin is in dispute. But Indiana has a little disagreement on who is ahead. Fun!

How much does the Pollster matter for Trend?

2008-05-05T15:14:00.002-05:00

One of the things we think about a lot at Pollster.com is the quality of polling. Mark Blumenthal's post on the North Carolina poll demographics here is a great example of how much variability we see among polls, all trying to hit the same target population.

This issue is also raised by those who would like to exclude some polls from our trend estimates. If one "bad apple" spoils the barrel, then this is a serious issue for our efforts to estimate the state of the races here.

We've stuck to our principle that we include all available polls without cherry picking (to shift the fruit metaphor!) but we don't do that out of blind faith. Rather we do it because the empirical evidence shows that the effects of single pollsters are generally small, certainly compared to the other sources of uncertainty about the state of the race.

Here I take a look at this issue for North Carolina and Indiana.

There are four elements that affect how much a pollster influences our trend estimate.

First, the pollster's results must be "different" from the trend we'd estimate without them. If a pollster happened to hit our trend dead on every time, their influence would reinforce our trend estimate, but not change it. So for a poll to affect the trend, it needs to be different from what we'd otherwise estimate.

Second, the pollster needs to produce results that are systematically different from the trend. If a pollster bounces around the trend, some high and some low, then the net effect is small, even if individual polls are rather far off the trend.

Since the trend is determined across all pollsters, these first two points are another way of saying that the pollster must differ from what other pollsters are getting.

Third, volume matters. In some states, a single pollster accounts for a substantial proportion of all polling, while other pollsters contribute only a single poll. The former obviously have more potential influence than the latter. But high volume of polls doesn't matter if they are consistently close to (and scattered around) the trend estimate based on other polling. The problem comes when the prolific pollster is also rather different from others, and especially if there are few other pollsters active in the state.

Fourth, polls late in the game can have more leverage on the "current" trend estimate. So a pollster that does several polls but only in the last week before election day can have more influence on the current estimate than they would if those polls were spread over the entire pre-election period. Again, such an effect is only visible if the late polls are different from other polling.

Having an effect on the trend could be a very good thing if the pollster is right while others are wrong. The problem is how do you know a priori which pollster will be right THIS TIME. Experience this year demonstrates that a good day can be followed by a bad day, or both on the same day.

It is also important to put these effects in perspective across all polls we see in a race. The individual polls are highly variable. Our data often finds polls covering plus or minus 5, 6 or even 7 points of our estimated trend for an individual candidate, and double that for the margin between two candidates. There is a lot of noise out there, and the whole point of our trend estimator is to extract the signal from the noise. Our estimator (especially the "standard" estimator I'm using here, as opposed to the "sensitive" estimator we also check) is designed to resist polls that are "way off" (i.e. outliers) but at the same time be able to follow the common trend across polls. (I'm going to not go into the details of our local regression estimator here, which is not a simple rolling average. Let's hold that for another day. The FAQ on this is coming.)

So let's take a look at the North Carolina plot way up there at the top of this post. The horizontal axis is scaled to show the range of poll results we've seen in the state since April 1. This provides perspective on how much variation you see from poll to poll in the raw results.

The red "whiskers" at the bottom of the plot are the individual polls taken over this time. There is a bit more than a 25 point range in the Obama-Clinton margin during this period. Since the trends in the state have been relatively flat, only a little of this variation is due to "real change".

Our trend estimate based on all polls is the vertical blue line, which as of Monday afternoon is +8.6 points in Obama's favor.

How much do individual pollsters matter for this estimate? PPP has done the most polling in the state. If we take them out, the trend estimate drops to 7.0, a shift of 1.6 points on the difference (or an average of .8 points for each candidate, moving in opposite directions of course).

At the opposite extreme, removing Insider Advantage from our estimator produces a 10.7 point Obama lead, a shift of 2.1 points on the difference, or 1.05 points per candidate.

For most other pollsters, the effect is far smaller, even for relatively frequent pollsters such as SurveyUSA and ARG.

So the maximum effect of removing a single pollster is a shift between a 7.0 and a 10.7 point Obama lead. A shift of 3.7 points on the difference can matter in a close race, but that difference is relatively small compared to the variation we see in individual polls. Indeed, the four polls completed 5/4 show a range of +3 to +10 for the Obama margin. (They average a +7.25, compared to our trend estimate of +8.6.)

There is less polling in Indiana, so we might expect more influence since there are fewer polls to stabilize the trend estimator.

Here the current estimate using all polls is -6.2, a lead for Clinton. The range of results we get from excluding pollsters is from -4.1 (excluding SurveyUSA) to -8.7 (excluding Zogby). That is a bit larger than North Carolina, as expected. But put this in the perspective of the range of raw poll results for Indiana, which is from -16 to +5 in polls taken since April 1. The six latest polls as of Monday, all ending on 5/4, range from -12 to +2.

To sum up. Which polls we include affect our results. That both has to be and should be. We WANT the data to matter, and of course it does. What we don't want is for individual polls to make such large differences for our results that inclusion or exclusion decisions become critical. The results we see here show that we SHOULD be somewhat uncertain as to the trend, as it depends upon which individual pollsters are included. What is somewhat different in our approach at Pollster.com is we want to emphasize this uncertainty and put it in perspective, rather than produce a single number and treat that as if it were "certain". That is why we always show the individual polls spread around our trend estimate in the charts. All estimates have uncertainty. We need to understand both the value of the estimate and the uncertainty inherent in it. Pollster effects are part of that story.

However, what is crucial is that these effects on the trend estimate are small compared to the range of variability we see across individual polls. The goal of our trend estimator is to produce a better estimate than what any single poll (or pollster) can provide. By that standard pollster effects on the trend are modest compared to the variability across individual polls.

Evaluating the accuracy of the polls is a different topic, one we'll revisit again on Wednesday.

NC and IN Sensitivity Update

2008-05-05T14:11:00.003-05:00

As we close in on tomorrow's primaries in North Carolina and Indiana, the "standard" and "sensitive" trend estimates have largely converged.

In North Carolina the standard estimator puts Obama at 50.1% and Clinton at 41.5%. The sensitive estimator has it Obama 49.5% and Clinton 42.2%. Or, a margin in the standard trend of +8.6 for Obama vs +7.3 in the sensitive estimate.

In Indiana, the standard estimator puts Clinton up 49.5% to 43.3% for Obama. Switching to the sensitive estimator makes it Clinton 51.2% to Obama's 43.5%. Or a Clinton advantage of 6.2% for the standard estimator versus 7.7% for the sensitive one.

Either way the polls are seeing a split decision tomorrow. Anything else will be a very interesting surprise.

Final Pennsylvania Updates

2008-04-22T06:57:00.003-05:00

Clinton has increased her lead in the trend estimates over the course of the last polls to 6.6 points using the standard estimator, and to 8.4 points using the sensitive estimate. Last minute polls have given her bigger margins.

Now the key question is whether undecideds push her over a 10 point win, or whether increases in turnout by new "unlikely" voters raises Obama's total.

Still a good bit of variation and some pollsters see a strong trend, others not so much.

Pollster variation doesn't make a lot of difference in our trend estimates.

But remember, since the polls don't allocate undecided, both they and the trend estimates are leaving some 8 percent of voters on the table. They will go somewhere, and if they break disproportionately for Clinton you have a "huge win", while if they go overwhelmingly for Obama you have a nail biter or a dramatic come-from-behind win. In previous primaries, the "winner" has usually enjoyed a significant increase in support beyond what the last polls showed.

Pennsylvania Undecided Voters

2008-04-21T13:08:00.004-05:00

Senator Clinton currently holds a 6 point lead over Senator Obama in Pennsylvania, based on our Pollster Trend Estimate, 49%-43%. But that leaves about 8 percent undecided. What they do will determine whether Clinton's vote expands her lead compared to the polls, or if the undecided narrow or possibly reverse, the lead.

My partner at Pollster, Mark Blumenthal, has looked at this using aggregate polling data here and in his NationalJournal.com column here.

In this post I take a look at the individual level, though using data that are three weeks old, so use caution in extrapolating to tomorrow's electorate.

Using data from the Time/SRBI poll of Pennsylvania, conducted 4/2-6/08, I estimate a model of support for Obama compared to Clinton. I use "the usual suspects" as variables predicting vote: partisanship, gender, race, Hispanic ethnicity, region of the state, age, education, religion and income. The data at that time found an eight point Clinton lead, a bit higher than today's trend estimate.

Using the coefficients for "decided" voters, I can estimate the probable vote of the undecided 11% of voters in the poll. This gives us a look at how they would be expected to behave IF they behave like those who have already picked a candidate. (Note the "if" here. As with all models, this assumes stable influence of the variables among the undecided as among the decided.)

The plot above shows the distribution of estimated probability of voting for Obama. Values close to zero are very likely to support Clinton, while values close to 1 are very likely Obama supporters. Those close to .5 are flipping a coin. The shape of the distribution gives a sense of where voters "lump up" in their estimated preferences.

The black line plots the distribution among those who reported a vote preference. The red line plots the distribution of estimated support among those who said they were undecided in early April.

The key point is that the undecided resemble the decided, with a small shift to the left, suggesting they were as a group somewhat more likely to support Clinton. In these data, the primary difference between undecided and decided voters was age, with older voters more likely to say they hadn't decided. As we've seen in virtually every exit poll, older voters are more likely to support Clinton, so the result we find here, that the undecided lean a bit more towards Clinton, is consistent with this result.

Now again for the caveats. These data are three weeks old. The model requires the assumption that undecided voters ultimately behave like those who decided. Different variables as predictors can make a difference. And so on.

The goal here is NOT, NOT, NOT a prediction of tomorrow's vote. Much may have changed since the first week of April.

The point is to illustrate what we can learn about undecided voters beyond the simple fact they say "undecided". In this case, the data suggest they are not wildly different from those who decided, but their older age makes it more likely they ultimately lean more to Clinton.

The Time/SRBI data are archived at the Roper Center for Public Opinion Research. I am solely responsible for the analysis here.

Pennsylvania Pollster Comparison

2008-04-21T12:34:00.003-05:00

Judge for yourself.

Day Before Pennsylvania Sensitivity Update

2008-04-21T11:21:00.002-05:00

The Pennsylvania race has turned slightly toward Clinton over the weekend, with her lead now at an even 6 points in our standard trend estimate. If you believe in taking more chances with random noise, the sensitive estimator has a 6.4 point Clinton lead.

In the rush of new polling over the weekend, it is also good to check how much any of them may be affecting our estimates.

Dropping any single pollster makes only a bit of different to our estimates. The Clinton trend ranges from 48.5% to 49.6%, while Obama ranges from 42.6% to 43.5%. So dropping your least favorite pollster can, at most, account for the difference in a 5 point race and a 7 point one.

And note that we still have about 9 percent undecided. I wonder what they will do?

National Dem Trends, Newsweek Poll

2008-04-19T07:00:00.005-05:00

A new Newsweek poll gives Barack Obama a 54%-35% lead over Hillary Clinton among Democratic voters (story here, detailed results here, and thanks to Newsweek and their pollster, Princeton Survey Research Associates International, for a full and complete disclosure of the details of their survey. A model others should be encouraged to follow.)

The Newsweek poll raised a few eyebrows for its 19 point Obama lead, considerably more than other recent polls, and beyond the 10.4 point Obama lead in our trend estimator. However, a closer look at recent data shows that Newsweek is not far from other recent data. Newsweek is the 6th poll in April with Obama at or above 50%, while five April polls put him below 50%. With Clinton, Newsweek is the 4th April poll putting her at or below 40%, while eight polls have her above 40%. So Newsweek shows a larger Obama lead than others, but it is not as far out of line as may first appear. (Note in the counts of polls above, we only count independent samples of the Gallup daily tracker, so dont' count each of their daily results as new polls.)

As you can see from the plots below, we've not seen many recent outliers in the national Democratic nomination polling, and the new Newsweek is well within the 95% confidence interval.

All that said, our trend estimate for the race puts Obama at 50.2% and Clinton at 39.8%, a significant gain for Obama during the month of April. Since late March, Clinton has suffered a somewhat greater downward slope while Obama's gains have been a bit more shallow, implying a slight gain among undecided voters.

The Newsweek poll also has some interesting internal results. As with virtually all this year's polling, Obama has a substantial lead among Independents who will vote in the Democratic primary or who lean Democratic: 61% to 28% for Clinton. What is a key to Obama's strength in the Newsweek poll is he ALSO leads among self-declared Democrats 51% to 38%, a group Clinton has won in most contests. If real (and I want to see more data before I accept this change) then Obama may be winning the consensus among party rank and file that will be key to persuading Superdelegates to move strongly in his direction. So long as he trails among the strongest party identifiers, that case is less persuasive. Pennsylvania provides a new test of this possible change in support. (Obama continues to trail in our Pennsylvania estimates, so it is unlikely he has so far persuaded a majority of Democratic identifiers there, though stay tuned for Tuesday's exit polls.)

The other important shifts in this national Newsweek poll is that Obama leads among men 57%-31% but also among women 52%-38%. Again this would represent an important gain among women.

The age gradient in Obama support has been interesting all year. In the Newsweek poll, he wins 18-39 year olds by 62%-28%, as usual, but also wins 40-59 year olds by 54-36%. In past exit polls, his "break even point" has varied among age groups from as low as 40 (i.e. losing all groups over 40 years old) to as high as 59 (only losing those over 60 years old.). More astonishing here is he gains a plurality of those over 60, 47%-41%, which if true would be his best performance among older voters all year.

The area of the Newsweek poll where Obama still suffers is among working class or poor whites, where he trails badly, 35%-54%. In contrast he leads 52%-35% among upper and middle class whites. That class divide remains a critical issue for his campaign.

A caution here as well. In any poll with such high overall support, the support almost has to reach across many subgroups (not quite as a mathematical certainty, but as a strong empirical regularity.) So we should be careful not to accept the depth of Obama's support among Democrats, women and those over 40 years old until we have more evidence from additional polling. In the exit polls this year, where we see big Obama wins (VA, MD, WI) we also saw him making strong inroads among these groups. But with the margin he achieved in these states, it would have been hard NOT to have done well across groups. Be careful of the cause and effect attributions here. It is a challenging state like Pennsylvania that can reveal how deeply into the various demographic groups Obama has managed to extend his appeal. But with those cautions, Newsweek's poll shows some evidence that the national Democratic constituency is moving in his direction across a number of groups.

If these changes are real, we'll see new polling that reflects it. If just a favorable poll (though not an outlier!) then new polling will show that these groups are not quite as enthusiastic for Obama as the current poll suggested.

Pollster Effects on Pennsylvania Trends

2008-04-18T12:26:00.003-05:00

There is a lot of interest in the differences among pollsters, and especially what effect they have on the perception of the race. Here at Pollster, the interest is specifically on the question of whether individual pollsters drive our results, and if so by how much.

So here is a quick look at those effects in Pennsylvania's Democratic primary.

The chart above shows the trend estimates that result from dropping each pollster in turn, and reestimating the trend without that pollster. This is a specific test of how much it matters whether we include a particular pollster or not.

Over the 15 pollsters we have represented in Pennsylvania, the estimates excluding each one for Clinton range from 46.6% to 48.3. The estimate with all pollsters included is 47.4% for Clinton.

For Obama, the estimates range from 41.2% to 42.5%, with the estimate for all pollsters at 42.0%.

The upshot is that pollsters do matter, but none drive the results by very much. A 1.7% range on Clinton and a 1.3% range on Obama for the trend estimate is very small compared to the range we see across the raw poll results. Another example of the greater stability of the trend estimators we use compared to the substantial variability across polls.

We can look at the effects of each pollster by comparing the trend estimate without that pollster to the estimate with them. The higher the effect, the more that pollster drives our trend estimate up for that candidate. Negative effects means the pollster drives our estimate down. Again, this is compared to the trend with and without the individual pollster.

The two charts below show these effects for Clinton and for Obama.

For Clinton, SurveyUSA has the highest positive impact on our trend estimate, followed by Rasmussen. At the opposite end, PPP has the largest negative effect on our trend, with Zogby/Newsmax the next largest negative effect. Other pollsters are clustered rather closely around zero.

And it is important to note that even the four largest positive and negative effects are all less that 1 percentage point.

For Obama's trend, Quinnipiac shows the largest positive effect, followed by Zogby and PPP with near identical effects. On the negative end, SurveyUSA has the most negative effect of Obama trend. Again, none of these effects is as much as one percentage point, and SurveyUSA's is less that half a percentage point. Other pollsters have less impact.

We can also look at the joint effects. These are the same as seen individually above but plotted against one another.

Pollsters do matter, and outliers matter even more. But the net effect of any individual pollster on our trend estimates in Pennsylvania are modest, especially when viewed in comparison to the wide range of raw poll results for each candidate. Another advantage of combining information across polls rather than pick single polls we "like".

Pennsylvania Dem Sensitivity Comparison

2008-04-18T12:22:00.003-05:00

Here is an update on the Pennsylvania Trend Estimates for both the standard estimator and the more sensitive one.

The difference is small: 0.3 for Clinton and 0.8 for Obama. But hey, it can be a game of inches. Either way it looks like about a 5 point margin right now.

(See earlier posts for details on the standard and sensitive estimate. The sensitive is about twice as responsive to the data, but tends to chase ghosts.)

Pennsylvania Dem Trend Sensitivity

2008-04-14T14:50:00.003-05:00

Time for a look at the sensitivity of our trend estimators. ARG has a new Pennsylvania poll out showing a 20 point Clinton lead. But Susquehanna Polling has one completed three days earlier with a 3 point Clinton lead and Zogby has one on the same day with a 4 point Clinton lead. Did things shift that swiftly or do we have an outlier?

Our standard trend estimator is designed to resist outliers, and it manages to do so in this case (see the chart above.) With or without ARG (and with or without ARG AND the close Susquehanna poll) the trend estimates only change by 10ths of a percentage point. What does change slightly is the slope of the trend estimate between the solid line with all polls and the dashed and dotted lines without ARG and without ARG and Susquehanna. In fact, the changes are slight enough you need to squint to really see them.

It may surprise you that removing ARG makes Clinton's trend estimate go UP, when you would understandably expect it to go down. The reason is simply that she is trending down either with or without ARG. Removing ARG means that the latest poll for the trend estimate is 3 days earlier, hence higher up on the downward trend. Since ARG doesn't much affect the slope, removing it just "backs the trend up" by three days, making Clinton a tiny bit better off. The same happens in reverse for Obama-- without ARG he is a shade worse off (0.1 percentage points) because his trend is rising either way, so backing up 3 days hurts him a tiny bit. (Removing ARG and the close Susquehanna poll makes similarly modest changes to the trend estimate.)

So for our standard estimator, whether we include or exclude the latest ARG poll makes very little difference.

That is by design. At this point a single new poll that shows a big change should be regarded with caution. It MAY reflect a big shift over the weekend. But it could just as easily be a statistical fluke that will not be replicated in other new polls. Until there is more evidence one way or the other, the trend estimate is designed to not chase after a single poll far away from the other data. But if we get two or three more polls showing similar results, the trend estimator will then be convinced the shift is real, and will turn in the direction of the new data.

But what if we decided to be less cautious and more willing to respond to new polling? For that, we have an alternative estimator that is about twice as sensitive. This one will pick up new trends much more quickly, but will also be misled by a single outlier much more easily. Let's see what this more sensitive estimator thinks is going on.

With the sensitive estimator, the ARG poll makes a HUGE difference. It shifts the difference in trends from a 4.3 point Clinton lead to a 12.6 point lead! That is really responsive. But it also demonstrates what a large difference a single poll can make when we crank up the sensitivity of the trend estimator.

What is more interesting with the sensitive estimator is that taking out ARG, we see a flattening of Obama's trend and even a tiny downturn, though also a continued decline for Clinton. If we also take out Susquehanna we find Obama continuing to rise but Clinton falling at a slower pace than without ARG alone. (You really have to squint to see the dotted line. Sorry about that.) Clinton leads by 4.2 without ARG AND without Susquehanna, by 4.3 without ARG alone.

So perhaps the sensitive estimator is showing us something new-- Obama may have flattened or even turned down a shade recently, but if so Clinton seems to have continued to decline as well. This also illustrates why we prefer our more conservative estimator. The sensitive version is just too dependent on individual polls at the end of the series. It provides a possible early hint of things to come, but is too unstable to count on. Our standard estimator may be a little slow to pick up a change of direction, but it seldom chases after random noise.

As for ARG's result, let's wait a day or two and see what other polls have to tell us and how our estimator responds.

For an animation of the Pennsylvania sensitivity analysis for a wide range of sensitivity, download this Acrobat file and use the arrow keys to control the animation speed.

Bush Approval Falls to 28.3%

2008-04-11T10:15:00.003-05:00

President Bush's approval trend has taken a sharp downturn in recent weeks, to fall to a new low for the administration at 28.3%. This follows a lengthy period of stable approval at around 32-33%.

Recent polls from Gallup and AP/Ipsos put approval at 28%, a new low for the Gallup poll. Harris recently found approval at 26% while CBS News put approval at 28%. Pew similarly has approval at 28%, though the Diageo/Hotline result for registered voters (as opposed to adults in the other polls) has approval at 35%, the only recent poll over 30%.

While the President has taken a back seat to the primaries in recent months, his new approval slump reflects growing pessimism over the economy which has displaced the war in Iraq as the most important problem facing the country. Likewise the percent saying the nation is headed in the wrong direction has continued to grow in recent months, to a high of 81% in a recent CBS News poll.

Most Important Problem, 2005-2008

2008-04-09T17:18:00.003-05:00

The Iraq war and the economy have consistently been the top two "most important problems" facing the nation during President Bush's second term. But the dynamics have changed dramatically over the past seven months.

After near parity in 2005, the war dominated throughout 2006 as far more important that the economy, and with rising numbers of people citing the war as most important. That peaked in early 2007 with concern over the war gradually diminishing through most of the rest of the year.

And then the economy struck. As recently as August 2007 only 8% said the economy was the most important problem. By early September that jumped to 13%, then to 23% in January and now 37% in early April. By contrast the war fell from 34% to 15% over that same time.

It will be ironic if the fall campaigns largely ignore the war to focus on an economy that 12 months earlier had looked fairly good.

California Super Tuesday Pollster Comparison

2008-02-05T13:07:00.001-06:00

There is quite a bit of disagreement in the California polls, so let's do a Pollster Comparison to see who says what.

The Democratic race has the biggest gap across pollsters. Most see Clinton ahead, but disagree on how much the race has tightened. But Suffolk and Rasmussen put Obama just barely ahead in their final polls. And the big difference is in the Zogby/Reuters/C-SPAN poll that sees Obama not only ahead but expanding his lead to double digits.

That gives us a range of results from Obama ahead by more than 10 to Clinton ahead by 10. Somebody is not quite right here. But who? And why?

Across all the polls, Clinton is relatively flat while Obama has gained. But looking at each poll and each candidate reveals more differences across pollsters.

Both Zogby and the Field poll find some decline for Clinton while others have her flat or rising (SurveyUSA and Rasmussen).

Everyone sees Obama either flat or rising, with no pollster (who has at least 2 polls) seeng Obama declining. But the extent of the rise, and how high Obama's starting point was, differs quite a bit across pollsters. Survey USA and Rasmussen both see Obama gains, but SurveyUSA offsets that by measuring gains for Clinton as well, resulting in relatively little net change over their last three polls. Rasmussen, on the other hand, has Clinton gains but even bigger Obama gains, resulting in a net decline and a final poll with Obama just barely ahead.

The Zogby polls in contrast find both Clinton declines AND Obama gains, making for a net change that strongly favors Obama and results in his 13 point lead in the final Zogby poll for California.

On the Republican side, we also see substantial differences. Two pollsters see a dramatic tightening of the race. Rasmussen and SurveyUSA estimate that McCain's earlier lead has now collapsed to a dead heat. (A single ARG poll also sees a dead heat, but has no over time data for comparison of trend.) In contrast the LA Times/CNN/Politico and Field polls both find McCain gaining some ground in their polls. These two are a bit stale now, but showed gains even when Rasmussen and SurveyUSA saw declines, so this is not only a difference of survey dates.

Zogby/Reuters/CSPAN is again quite different, finding Romney leading in all three polls and expanding his lead to high single-digits in the last two polls.

Across the Republican polls, the differences are almost all matters of the degree of increase for both McCain and Romney. All polls agree that McCain has been increasing his support. But they also agree that Romney is rising sharply. The differences in net support have a lot to do with differing estimates of how fast and how far Romney has risen. Those with the greatest Romney rises have him catching McCain or leading him.

Given the length of time it is likely to take to count the California vote, it may be Thursday before we know which of these polls was closer to the end result.

Super Tuesday Endgame

2008-02-03T21:13:00.001-06:00

Updated: 2/5/08 at 1:59 pm est

Now that the first "Super" event of the week is over, time to turn to the next one.

We've got a number of polls in several states, fewer polls in others. Here are the trends in the 17 states for which we have polls since January 1.

In the Republican chart below, I've omitted Utah where Romney holds an 84-4 lead over McCain, according to the one Republican poll since January 1. Expanding the scale to include that data point makes it hard to see the differences in all the other states.

See the individual state charts at Pollster.com for trend estimates in the states with sufficient polling. Here, look at the data and reach your own conclusions. The data here are through Sunday. I'll update the charts Monday evening with the Monday polls.

Super Tuesday Polling at a Glance

2008-01-31T07:07:00.001-06:00

Updated: 2/5/08, 1:58pm est

No time for commentary this morning, so here is an explanation.

It is ironic and annoying that the most important date on the primary schedule is also the date with the fewest polls per state. Just as the campaigns are struggling to run 22 simultaneous campaigns, so pollsters and the media have invested little in comprehensive polling of the Super Tuesday states. Even large states such as New York and California have fewer than 10 polls since January 1, far fewer than we saw last week in Florida for example. As a result, we have many states with no data at all, preventing a comprehensive overview of the prospects for Tuesday. Even where we do have polls, we lack enough to consistently estimate the trend with data taken since Iowa. Where we can estimate trends, we've done so on the "regular" state pages at Pollster.com. You should go there for the best trend estimates we can manage with so little data.

The charts here are a way of seeing the entire set of Super Tuesday states (where we have polling) at a glance.

Rather than plot the usual trends with so few data points, each poll is a point and the darker the point the more recent the poll. The points are also scaled in size to be proportional to the number of delegates at stake in the state.

Instead of a trend estimate, this plot highlights the median of all post-Iowa polling in the state. The shading of points will then let your eye tell you whether there is a visible trend around that median. Be your own data analyst!

When more states become available, they'll be added to updated charts. If a state is missing, we don't have polling for it. (If you think we've missed a state with polling data, let us know!)

Florida Republican Poll Errors

2008-01-31T07:00:00.001-06:00

Polling had a pretty good night in Florida on Tuesday. While early polls understated both McCain and Romney's vote, the polls got better as election day closed in, reflecting a (measured) upward trend in both candidates support. By the last day or two of polling most polls were within the ten-ring, and a number were close to the five-ring.

There was more disagreement among the polls as to who was ahead, and this included some late polls that put Romney ahead of McCain. However, most of these polls reflected a race "too-close-to-call", rather than egregious errors about the leader, with one large exception towards the lower right in the plot.

As we've seen all year, the polls got the vote for third and fourth place finishers very accurately, almost all within the five-ring.

Florida Republican Endgame

2008-01-28T21:33:00.000-06:00

The polls in Florida point to a very close contest between Mitt Romney and John McCain. As of Monday's polling used here (there will be new final polls available Tuesday morning) Romney has a small lead based on our trend estimators (both standard and sensitive). But that lead is so small that a "dead heat" is probably still a good characterization of the race.

As we've seen in several previous states, the final vote has often broken strongly in favor of the winner with little going to the second place finisher. But who will that winner be? Other than our trend estimates, the data have little more to tell us on that score. With Ron Paul polling consistently at between 4 and 5 points, there are at least 10% of voters yet to make up their minds. A strong surge could boost one of these front-runners from about 30% to near 40% and a very significant win. An even split of undecideds will make for a close finish with both around 33-35%.

Both Romney and McCain have been gaining ground in Florida, but Romney's rise has been consistently sharper than McCain's. Moreover, there is little evidence that McCain has enjoyed a post-South Carolina spurt in Florida. Nor does Romney's Nevada win seem to have helped him beyond his already considerable upward trend.

As the Florida race narrowed to a two way contest for first place, Giuliani and Huckabee have subsided into the 12-15 percent range. They too could go either way for 3rd and 4th, though at the moment Giuliani has a small advantage for third place. As mentioned above, Ron Paul trails with 4-5% for fifth place.

The chart below shows how much the polls in Florida have overlapped since the South Carolina primary and Nevada caucus 10 days ago. While the trend estimates are slightly distinct, the huge overlap of polls for Romney and McCain show that we should remain quite uncertain as to who is "really" ahead. Sometime Tuesday night, we'll find out.

South Carolina Poll Errors

2008-01-27T13:55:00.001-06:00

The polls had a bad day on Saturday in grossly underestimating the support for Barack Obama, though they nailed the Clinton and Edwards votes quite well. Not one poll came within the ten-ring, and the final poll of the primary understated Obama's vote by nearly 15 points. Several polls flirted just inside the 20-ring, and one hapless example of the consequences of poor question wording, the Clemson University poll, understated Obama support by nearly 30 points. The Clemson poll allowed 36% to remain "undecided", hopelessly biasing downward their estimates of candidate support, but especially so for Obama.

My colleague Mark Blumenthal has posted a nice comparison of these South Carolina results with those "terrible" polls from New Hampshire. Below is the same poll error chart for New Hampshire, but scaled the same as the one for South Carolina above.

The New Hampshire results were mostly inside the 10 ring and all were inside the 15 ring. And a couple even touched the 5 ring. Judged by distance from the bullseye, New Hampshire doesn't look that bad, certainly not compared to South Carolina.

But there is another difference, and this is where New Hampshire was terribly wrong and South Carolina not so bad: All but one of the New Hampshire polls had the wrong leader. None of the South Carolina polls, not even Clemson's, got the leader wrong.

So while the distance from the bullseye was quite a bit worse in South Carolina, the creation of confounded expectations was not. It was the expectations that were created and then confounded that make New Hampshire a polling disaster, while there has been little said about the polling errors in South Carolina. (Except here, where we care about such things all the time!)

The other interesting comparison is the parallel that the number 2 finisher in both South Carolina and New Hampshire was quite well estimated. The SC polls got Clinton within normal margin of error. And the New Hampshire polls also got the 2nd place finisher there, Obama, within reasonable error.

The problem in both cases is in the substantial underestimate of the first place finishers vote. The final choices of late deciding voters is a challenge for all polling, and perhaps especially so in primaries where there is no "party identification" to come home to if you can't make up your mind. In New Hampshire the Clinton win rested on significantly more voters supporting her than expected. In South Carolina is was the magnitude of the victory, rather than first place itself, that confounded the polling.

Increases in voter turnout in this cycle may be part of the story (a 75% increase in South Carolina), but here we see those late deciders breaking for different candidates, and yet in both cases for the ultimate winner. Second place results may on average be slightly low compared to the polls, but the first place "bonus" seems quite strong. At least for the Democrats. In the Republican South Carolina primary, both first and second place finishers were a bit underestimated, so there was not the same asymmetric error for first place. The New Hampshire Republican race also about equally understated the votes for first and second. The relatively lightly polled Michigan Republican race shows somewhat greater underestimate of first place (Romney) and second place (McCain). And in Nevada, with only 3 late polls, Romney was dramatically underestimated, while Ron Paul finished second but was only moderately underestimated.

So perhaps these reflect pollsters' difficulty in discerning the likely behavior of undecided voters, or perhaps these are last minute decisions to vote by "not-so-likely" voters who are screened out of the sample but who turn out for the ultimate winner in larger than expected numbers.

Turning to 2nd and 3rd place, the chart below shows that the polls had a pretty good day predicting the Clinton and Edwards votes. Despite some chatter about a late Edwards surge and a Clinton fall (including some evidence in our sensitive trend estimates that such a movement was occurring) most of the late polls were within the five-ring for 2nd and 3rd place, and all got the order of finish right.

South Carolina Democratic Endgame

2008-01-25T14:05:00.001-06:00

The South Carolina polling continues to show a substantial lead for Obama, while Edwards' rise hints that he could challenge Clinton for second place.

At the moment, Clinton continues to hold a six-point advantage over Edwards, but Edwards has been rising while Clinton has been moving down. Obama, meanwhile, has been fairly steady at around 40-44% support, though with some hint of a small decline in the sensitive estimator. Note however that the Clemson University poll included here had an amazing undecided rate of 36%. That makes every candidate in their poll look lower than in all other polls that have a much lower rate of undecided. The level of undecided is quite sensitive to how the poll is conducted, including whether respondents are pushed as to whether they "lean" towards a candidate. The Clemson poll apparently didn't push at all among undecided voters. We'd be making a mistake to read their data as indicating a decline of support for anyone.

A second place for Edwards would, of course, be good news for his campaign, while Clinton would no doubt argue she had conceded the state to account for a third place finish. But Edwards still has some ground to make up, and late deciding voters remain an unknown-- if they are unhappy with either Clinton or Obama, Edwards can benefit simply by not being one of them. This may be especially true among independents who vote in the Democratic race, and the expected handful of Republicans who show up. (Republicans and independents can vote in the Democratic primary only if they did NOT vote in last week's Republican primary.)

I think the more compelling story of South Carolina will be the exit poll results. Obama has appealed to white voters in previous primaries and caucuses. The pre-election polls have found him getting as low as 10% of the white vote in South Carolina. The potential for racial polarization in this Southern state could damage his ability to transcend race as a basis of voting. Paradoxically, there has been speculation that Clinton can win the votes of black women, a result that could reduce polarization in the exit poll. We'll know much more about how voters decided by Saturday night.

South Carolina and Nevada Poll Errors

2008-01-21T11:49:00.001-06:00

Polling for the South Carolina Republican primary mostly got the winner right, but large undecided percentages prior to the election accounted for a general underestimate of the final vote for both McCain and Huckabee. Late polling also found the race closer than the eventual outcome.

Pollng for third and fourth place was less successful in detecting Fred Thompson's final strength and the drop of support for Mitt Romney, who abandoned South Carolina to spend time in Nevada, where he scored a strong first place finish. While almost all the polls finished inside the "5-ring", correctly seeing a close fight for the 3-4 spots, all but one poll got the order of Thompson-Romney wrong (and the one that got that order right substantially missed the magnitude of both votes.)

In the Nevada Republican caucuses the polls wildly underestimated Romney's final strength of 51% of the vote. (Note that the rings here have to be rescaled to include the very large errors.) Even the best of the three Nevada polls was more that 15 points off on Romney. The earliest of the three polls, taken before Romney's win in Michigan, was over 30 points low.

McCain and Huckabee ignored Nevada, essentially conceding the state to Romney, but the polling still failed to pick up the magnitude of his support there.

The polling likewise failed to capture Ron Paul's second place strength. Preelection polls put Paul at about 7% compared to his finish of 13.7%.

In terms of erroneous expectations, the polling also put McCain well ahead of Paul, uniformly getting the 2nd and 3rd place finishers wrong.

On the Democratic side, the final poll was inside the "10-ring", and the polling improved as the caucus approached. Here the surprisingly poor showing of John Edwards, and the Democrat's caucus reallocation rules for non-viable candidates helped boost the final percentages away from the polls. The three most recent polls all got the order of finish correct.

South Carolina Republican Endgame

2008-01-18T14:31:00.001-06:00

South Carolina is looking quite interesting for the Republicans. There has been a decent amount of polling since Iowa, with Huckabee getting a brief bounce but subsequently subsiding a bit, while McCain has gained since the first of the year. There is little evidence that Romney benefited significantly from his Michigan win. Fred Thompson has risen a bit, based on the sensitive estimator, but still trails in fourth place.

If we try to pick these data apart a bit, the sensitive red estimator is trying hard to fit Huckabee's varying fortunes. His December rise, and Iowa bump are picked up, but if those were real then so is the decline we see to his current level of about 22%. Whether he has been trending up down, or flat depends on how wide your view of the polling is. The blue estimator has the longest run view, and still see's him improving over his showing in 2007, but that was ages ago politically. The sensitive red estimator is showing a downturn since Iowa, but if you squint hard and ignore the earliest post Iowa polls you might believe you can still see some rise in the last week or 10 days. But do note the bottom line: the two different trend estimates still put his current support at between 21.6% and 23.3%. So while they disagree on immediate trends, the end up close to the same bottom line.

For McCain, there is little dispute that he has surged since early December when he was in the low-teens to somewhere in the mid-to-upper 20s today. The sensitive estimator thinks the rate of climb since Iowa has been more rapid that does the blue estimator, but again both put his support between 26.9% and 29.3%.

One big question in South Carolina is whether conservative criticism of both Huckabee and McCain is having any effect. If Thompson is benefiting from that, his polls only modestly show it. The sensitive estimate suggests a rise from about 10% to about 14%, but there is no polling evidence for a surge that would allow him to compete for first place.

Finally, Romney's Michigan win seemed to help him in Nevada (based only on 3 polls, I should add) but there is no evidence of a bounce in South Carolina. After spending Wednesday and part of Thursday in the state, Romney appeared to concede the race and moved on the Nevada to campaign, where his chances look better. The Romney trends are also in complete agreement: No substantial trend, and both agree on 16%.

One big warning here. South Carolina has shown suprising numbers of undecided. The Fox poll today, for example, has an unbelievable 19% undecided. That is HUGE. And, if you add up the trend estimates for the four top candidates here, you get... 81% leaving the same 19% unallocated. In Fox's poll, less than 10% pick Paul, Giuliani or Hunter combined, so there is a gigantic amount of room for those last minute deciders to break one way or the other.

With a history of under-the-radar negative campaigning in the state, and evidence of the same this year, it would be surprising if we don't see some important shifts here at the end.

In New Hampshire and Michigan, those late shifts have benefited the first place finisher disproportionately.

This chart is an attempt to illustrate the variation in the polling over the past week. (A comment raised some good questions about it, so let me try to explain it better.) The horizontal axis is the current sensitive trend estimate for each of the top four candidates. The vertical axis is the actual poll results since the Michigan primary. The point is that the sensitive trend puts the order as McCain, Huckabee, Romney and Thompson. But the vertical spread of the points for each candidate shows how much variation we've see from poll to poll. While most of the McCain polls are higher than most of the Huckabee polls, there is some overlap. Likewise most of Huckabee's results are higher than Romney. But Romney and Thompson show considerable overlap. The less the overlap, the more reasonable it is to believe the separation between candidate trends is reliable, and the more overlap the greater the uncertainty. If all the polls showed exactly what the trend estimate shows, then the points would all be on the diagonal line, and there would be no disagreement at all. The fact that most of McCain's polls are below the 45 degree line shows that he has been trending up, while Huckabee's points are mostly above the diagonal, consistent with his recent downward trend. Romney and Thompson are about equally above and below the diagonal, showing little trend over the last eight polls since Michigan.

But the simple point is that the more spread you see vertically, the more uncertainty. And the more overlap between pairs of candidates, the more uncertainty.

For me, that 19% undecided in the Fox poll is scary. And fun!

Nevada Endgame

2008-01-18T14:13:00.001-06:00

The Nevada caucuses are upon us, but the polling is scant. As the graphs make clear, we have only four polls since New Years, so the trend estimates here should be taken with more than the usual grain of salt. With just 4 polls, the sensitive red estimator is going to try hard to come close to all the polls, making Clinton and Edwards look like they are experiencing huge trends. Obama's four polls are more clustered, so the red trend is a little better behaved, but still tries hard to find a trend when there truly isn't enough data to support one.

Also, there was no Nevada polling between December 3 and this week, so we have NO IDEA what happened during that time. Both trends are fit to all the data (including polls in 2007 not shown in the plots.) So the blue line is the best conservative guess given all the data in 2007 plus the four polls in 2008. The red line sensitive estimator is, in this case, I think hopeless.

If you force me to choose, I'd take blue in this case. With so little data, you want to be conservative. But you can see the poll-to-poll variation is large for Clinton and Edwards. Perhaps a lot has been changing this week, but you can't be sure it isn't just pollster variation.

And one more caution: as a caucus with very low expected turnout, polling Nevada is at least as perilous as Iowa, if not more so.

For the Republicans, we have only three polls taken this week. The only good news here (statistically speaking) is that McCain, Huckabee and Thompson are all tightly clustered without the large variation we saw for the Dems. Romney, on the other hand, shows a big gap between the first poll, completed BEFORE his win in Michigan and the two polls taken since then. Looks like a bump.

The lack of wild variation lets both trends come closer to the polls, but again just three polls is a ridiculously small number to base much on. I'd stay conservative with Blue here.

Ironic that where uncertainty is intrinsically great because of low turnout and the importance of organization that we also have magnified that uncertainty with so few polls. Makes it hard to be an odds-maker in Nevada.

An Emerging Republican Consensus? Can it be?

2008-01-17T12:17:00.000-06:00

Four events. Three winners. South Carolina and Nevada up in the air. Can there possibly be any reason to think Republicans are settling on a consensus candidate?

Strange as it seems, there is one clearly emerging candidate---he who was declared dead in July: John McCain. (See my premature obituary for McCain here.)

Given McCain's losses in Michigan and Iowa, his one New Hampshire victory is hardly a reason to believe he is an emerging consensus. But the trends in polling across the nation and nine early states points to McCain as the unique candidate in the Republican field who has been strongly rising across all states. Further, his rise is not a "bump" from an early win: the rise predates the primaries, a crucial point.

While McCain does not lead in all these states (and has now lost two of three) his polls now put him in the competitive range in each of these states.

The most striking and compelling feature of the chart is the simultaneous upturn across all states, especially following the long term decline in McCain's support over most of 2007. Why McCain, and why then?

Part of the answer must be the fall of Rudy Giuliani in the fourth quarter of 2007.

After leading in national polls throughout the first three quarters, Giuliani's support took a sharp turn downward in the late fall, closely associated with the timing of the indictment November 8th of his long time friend, partner and associate, Bernard Kerik. (I also think failing to compete in early primaries, and then doing quite badly, is a contributing recent cause of Giuliani's decline. Late win strategies do not have a good track record... ask John Connally.)

McCain's rise comes after Giuliani's decline begins. Given that both candidates appeal more to moderate and somewhat conservative Republicans (as opposed to the conservative base of the party) it is likely that these voters turned from Giuliani and found McCain the most attractive among the remainder of the field.

McCain also shares with Giuliani the advantage of perceived "electability". As Giuliani's fortunes fell, McCain emerged as the candidate Republicans see as having the best chance of defeating any Democrat in November. In primaries, perceived viability is an important asset.

We can also see support for McCain surge in favorability ratings among Republicans. In the Diageo/Hotline national poll taken January 10-12, McCain scores a remarkably high 78% favorable rating among Republicans (with 31% very favorable). Giuliani is at 67% (with only 15% very favorable), Mitt Romney at 55% and Mike Huckabee at 53%. Granted this poll's timing reflects the New Hampshire win and not the Michigan loss, these are still notably high ratings for a candidate who has often alienated important elements of the Republican party.

McCain led the vote choice among Republican primary voters in the Diageo/Hotline survey by 32% to Huckabee's 17% and Romney's 15%. (McCain also lead in the Pew poll 1/9-13 by 29%-20% over Huckabee, and in Gallup/USAToday by 33%-19%. Again, none of these reflect Michigan's impact.)

On electability, McCain was rated most likely to defeat the Democrat by 42% to 17% for Giuliani in the Pew poll. That reversed the result from Pew's November survey that had Giuliani most likely to win by 45% to 16% for McCain.

Among Republican primary voters, more also see McCain as most likely to win the Republican nomination, regardless of their own preference. Diageo/Hotline has McCain with 82% saying likely to win, 25% very likely and 57% somewhat likely. Compare Huckabee at 56% (6%/50%), Romney 53% (6%/47%) and Giuliani 45% (7%/38%).

Also surprising, given McCain's testy relations with so many Republican groups, is the relatively small number who would refuse to vote for him. McCain suffers only 9% of Republicans who would never vote for him in the Pew poll. Huckabee is at 8%, Giuliani at 15% and Romney at a devastating 20%.

Since his nadir in November, McCain has achieved a remarkable recovery among Republican voters. Perceptions of him have changed substantially in the last three months, both in favorability and in electability as well as in support. And those changes are reflected across all of these states and the national polling as well. That is hard to argue with.

But perhaps we still should argue a bit. There are two things that may yet halt the McCain victory. One is the opposition of important organized groups within the Republican party. While the rank and file may have come over to McCain, those groups bitterly opposed to campaign finance reform remain adamant in their opposition, and they wield great influence among party elites as well as some grass roots organizations. The second barrier is the calendar which looks to pit McCain against Huckabee in South Carolina on Saturday. (Romney has apparently conceded South Carolina in favor of a Nevada effort. Fred Thompson has yet to rise in South Carolina polling despite a surprisingly animated debate performance.) South Carolina was McCain's Waterloo in 2000, and its large religious base should be Huckabee's best shot at a win since Iowa. A McCain win can spur his campaign on, but a loss can shake the progress he's made with voter perceptions of his support within the party. (I should add a third threat: McCain's mouth. The "straight talk" he prizes has gotten him in trouble before, and can again.)

A McCain nomination will certainly shake many powerful elements of the party. Whether they can prevent that depends in part on the development of a common choice among the alternatives to McCain. Romney remains anathema to 20% of Republicans. Huckabee is opposed by economic conservatives. Thompson has failed to emerge, and Giuliani has other problems. So while many Republican groups despise John McCain, it is not clear they can unite in embracing one of the alternatives who has also proven attractive to voters.

The candidate profile most like McCain's is Huckabee's. His sharp rise in November and December was similar in appearance, covering a number of states and national polls. But Huckabee's rise stalled in most states and has taken a fall generally since mid-December. His poor third place finishes in New Hampshire and Michigan dampened any Iowa momentum, and he must now win South Carolina based on overwhelming support among conservative Christians to remain in competition. He probably couldn't have a better state to try to pick up that win, but it is a make or break opportunity.

Romney's win in Michigan was a life preserver, but not necessarily a life saver. The Romney decision to concede South Carolina must be bitter given the substantial spending he committed to the state. It is also proof that he has failed to overcome opposition from Christian evangelicals who remain very reluctant to embrace a candidate of the Mormon faith.

Still, Romney's poll profile is at least generally upward sloping. It lacks the amazing coherence of McCain's (or even Huckabee's.) One can still imagine Romney picking up wins in the trench warfare of February 5th, and making the delegate count a serious battle after. But he does not appear to have unified his support across a variety of states.

And Fred Thompson remains the disappointment of the year. In paper qualifications, both in office holding and in ideology, Thompson's campaign came from central casting. But the actor failed to grow into the role. Here the uniformity of polling is a uniform decline across all states since the peak, just before the official announcement in September.

So when we survey the Republican field, only one trend stands out unambiguously and uniformly across states: McCain's rise and emergence as the only candidate doing better and better almost everywhere.

In July I said a McCain recovery would be a miracle of Biblical proportions. Now I've seen, yet I still find it hard to believe.

Michigan Poll Errors

2008-01-16T15:39:00.000-06:00

(Darker points are polls taken closer to primary date, lighter points were taken earlier. All polls were completed after the New Hampshire primary.)

The Republican primary polls for Michigan did better than those for New Hampshire, but generally underestimated support for Mitt Romney. Only one poll got both Romney and McCain's vote within the five point error ring.

All three polls completed on January 14th, the day before the primary, got McCain's vote within a small error, but the Romney errors ranged from small to quite large for these last polls. Earlier polling was generally further from the final vote, suggesting that a trend to Romney was partially captured by the late polls, but only imperfectly.

For the second and third place finishers, the polls were quite good for McCain vs. Huckabee, with all three final polls inside the five-ring and all eight polls inside the 10 ring.

The pattern of underestimating the winner's percentage while doing quite well on the 2nd and 3rd place finishers suggests that much of the undecided category in the surveys eventually went to Romney, boosting his total beyond the support registered in the poll, while barely adding to the 2nd and 3rd place totals.

Michigan Endgame

2008-01-14T12:49:00.000-06:00

Michigan for the Republicans is a critical test for Mitt Romney. After two "silvers" in Iowa and New Hampshire (and a mini-gold in Wyoming which got little attention or credit) Romney badly needs a win in his original home state. With a win, Romney can become the third winning Republican going into the Nevada and South Carolina events this weekend.

But John McCain also needs a Michigan win. Coming back from the dead to win New Hampshire was a huge achievement for McCain, but he needs to prove he is for real outside of his best state from 2000. That year he also won Michigan, thanks in large part to Democratic cross-over votes. This year Democrats are again free to cross over for McCain, thanks both to open primary rules and the fubar Michigan Democratic primary stripped of all meaning by breaking party rules to move ahead in the voting. With no meaningful Democratic vote, those Dems who supported McCain eight years ago are again free to do so this time. Whether or not that happens is just one more nightmare problem for pollsters: how many Democrats will in fact vote in the Republican primary?

And Mike Huckabee could certainly use a strong finish to show that Iowa wasn't his first and last hurrah.

The numbers look not so good for Huckabee. There is the barest hint of a post Iowa bump for him. Rather his gains in November and December seem to be all the rise he's gotten in Michigan. Since Iowa, Huckabee's poll support in fact seems to be falling, as captured by the sensitive red estimator (but not the slow-to-change blue line.) Whether blue at 17% or red at 14%, Huckabee looks likely to be a distant third in Michigan.

McCain and Romney on the other hand are neck and neck and the polling variation is so large there is no way to declare either a leader. Romney has the slightest of leads in the sensitive estimator at 26.1% to McCain's 25.6%, but that difference is meaningless given the spread in polling. McCain has clearly picked up considerable support since Iowa and New Hampshire, but so has Romney. McCain appears to be gaining more rapidly but with so little time since New Hampshire it is impossible to get a reliable estimate of the rate of gain in the last 5 days.

The bottom right panel of the chart shows clearly how uncertain the top two spots in Michigan are. The blue Romney dots mix in with the red McCain dots, overlapping so much that there is clearly no reason to think one is ahead of the other. Even some Huckabee results are within the range of Romney and McCain support.

Most of Huckabee's higher polls are older, and his recent downward trend means that more of his polls are above his current trend estimate. For Romney and McCain, the polls are evenly scattered above and below trend.

Republicans have plenty of reason to turn out Tuesday. They can help make or break the candidacies of McCain or Romney. Those pesky independents and Democrats remain the huge unknown. See Mark Blumenthal's analysis of the recent polling and how many Democrats are included in the various samples for a good look at how squishy that number is.

Polling Errors in New Hampshire

2008-01-09T13:58:00.001-06:00

Hillary Clinton's stunning win over Barack Obama in New Hampshire is not only sure to be a legendary comeback but equally sure to become a standard example of polls picking the wrong winner. By a lot.

There is a ton of commentary already out on this, and much more to come. Here I simply want to illustrate the nature of the poll errors. These show the nature of the problem and help clarify the issues. I'll be back later with some analysis of these errors, but for now let's just see the data.

In the chart, the "cross-hairs" mark the outcome of the race, 39.1% Clinton, 36.4% Obama. This is the "target" the pollsters were shooting for.

The "rings" mark 5%, 10% and 15% errors. Normal sampling error would put a scatter of points inside the "5-ring", if everything else were perfect.

In fact, most polling shoots low and to the left, though often within or near the 5-ring. The reason is undecided voters in the survey. Unless the survey organization "allocates" these voters by estimating a vote for them, some 3-10% in a typical election survey are left out of the final vote estimate. Some measures of survey accuracy divide the undecided, either evenly across candidates or proportionately across them. There is good reason to do that in another post. But what the pollsters publish are the unallocated numbers (almost always) and so it seems fair to plot here the percent of the vote the pollster published, not one with undecided reallocated.

What we see for the Democrats is quite stunning. The polls actually spread very evenly around the actual Obama vote. Whatever went wrong, it was NOT an overestimate of Obama's support. The standard trend estimate for Obama was 36.7%, the sensitive estimate was 39.0% and the last five poll average was 38.4%, all reasonably close to his actual 36.4%.

It is the Clinton vote that was massively underestimated. Every New Hampshire poll was outside the 5-Ring. Clinton's trend estimate was 30.4%, with the sensitive estimate even worse at 29.9% and the 5 poll average at 31.0% compared to her actual vote of 39.1%.

So the clear puzzle that needs to be addressed is whether Clinton won on turnout (or Obama's was low) or whether last minute decisions broke overwhelmingly for Clinton. Or whether the pollster's likely voter screens mis-estimated the make up of the electorate. Or if the weekend hype led to a feeding frenzy of media coverage that was very favorable to Obama and very negative towards Clinton, which depressed her support in the polls but oddly did not lower her actual vote.

On the Republican side we see a more typical pattern, and with better overall results. About half of the post-Iowa polls were within the 5-ring for the Republicans, and most of the rest within the 10-ring.

As expected, errors tend to be low and left, but the overall accuracy is not bad. This fact adds to the puzzle in an important way:

If the polls were systematically flawed methodologically, then we'd expect similar errors with both parties. Almost all the pollsters did simultaneous Democratic and Republican polls, with the same interviewers using the same questions with the only difference being screening for which primary a voter would participate in. So if the turnout model was bad for the Democrats, why wasn't it also bad for the Republicans? If the demographics were "off" for the Dems, why not for the Reps?

This is the best reason to think that the failure of polling in New Hampshire was tied to swiftly changing politics rather than to failures of methodology. However, we can't know until much more analysis is done, and more data about the polls themselves become available.

A good starting point would be for each New Hampshire pollster to release their demographic and cross tab data. This would allow sample composition to be compared and for voter preferences within demographic groups to be compared. Another valuable bit of information would be voter preference by day of interview.

In 1948 the polling industry suffered its worst failure when confidently predicting Truman's defeat. In the wake of that polling disaster, the profession responded positively by appointing a review committee which produced a book-length report on what went wrong, how it could have been avoided and what "best practices" should be adopted. The polling profession was much the better for that examination and report.

The New Hampshire results are not on the same level of embarrassment as 1948, but they do represent a moment when the profession could respond positively by releasing the kind of data that will allow an open assessment of methods. Such an assessment may reveal that in fact the polls were pretty good, but the politics just changed dramatically on election day. Or the facts could show that pollsters need to improve some of their practices and methods. Pollsters have legitimate proprietary interests to protect, but big mistakes like New Hampshire mean there are times when some openness can buy back lost credibility.

New Hampshire Pre- and Post-Iowa

2008-01-08T14:26:00.000-06:00

Here is one more way to look at the impact of Iowa. The plot shows each poll (tracking polls are included only when their samples don't overlap, e.g. every 3rd day for a 3 day track.) Polls ending on the same day are separated in the chart, though the order is arbitrary within day.

McCain had already captured the lead over Romney by the Iowa caucus. While three post-Iowa polls still find a Romney lead, the majority of polls put McCain in the lead. Moreover, the height of the bars is quite similar before and after Iowa, indicating little effect.

That lack of change is interesting since Romney might have been expected to fall due to his disappointing second place finish in Iowa. McCain's 4th place in Iowa wasn't read as a "loss", in the bizarre expectations game. But regardless of their finishes, Iowa seems to have not moved the New Hampshire Republican electorate at all.

On the Democratic side, the large pro-Obama bounce is obvious. If it didn't show up in the first day or two post-Iowa, it clearly appears after that. A Clinton lead of some 10-15 points became an Obama lead of some 8-10 points.

Last Day of the New Hampshire Endgame

2008-01-08T12:50:00.001-06:00

The last New Hampshire polls are in, and there is little change of trend from Monday. The Obama rise continues to look steady and strong. The sensitive trend estimator in red continues fit the sharp turn in Obama support better than the slower-to-change blue estimator, though both end up close to one another in estimating current support.

The current estimates are 39.0% for Obama and 29.9% for Clinton. If you like the more stable standard blue estimate, the numbers are 36.7% and 30.4% respectively.

There is the tiniest of hints that Romney's decline and McCain's surge have both flattened just a bit in the sensitive Red estimator, though the actual difference between the red and blue trend estimates is trivial for each candidate.

The sensitive red trend puts McCain at 33.4% and Romney at 27.5%, while the standard blue estimate has it 34.2% to 27.5%.

The first bit of suspense tonight will be whether McCain succeeds in holding a lead over Romney. The trend data say yes, but there is some considerable variation in the McCain poll results.

On the Democratic side, the question should be the size of the Obama win.

Edwards looks to be a distant third, a result that should be damaging to his campaign to emerge as one of the two "change" candidates.

Huckabee appears to have utterly failed to capitalize on his Iowa success. That is a big deal, in my view, because his campaign needed dramatic successes to bootstrap itself into a national effort. Iowa alone is not enough. Can he recapture the Iowa momentum in Michigan or Nevada or South Carolina in the face of a new look at McCain? (Or a reborn Romney should an upset happen in NH.)

The Thompson campaign gave up in NH, and Giuliani's collapse there just continues the question of whether his once high flying campaign can survive the early series of losses he now seems set for.

Some of these questions will be answered tonight.

New Hampshire Endgame

2008-01-07T13:32:00.001-06:00

The New Hampshire endgame polling presents an interesting contrast. The Republican race shows virtually no hint of an "Iowa Bounce." The Democratic race, on the other hand, is showing a huge bounce for Obama and a drop for Clinton. Edwards is largely unaffected.

The charts also show the better performance of the sensitive red-line estimator when things are as dynamic as they have been since Thursday. The Red estimator catches the upturn in Obama support pretty well, while the blue estimator is trying hard to keep up but its "slow to change" nature means it totally misses the timing of the upswing.

If anyone were actually asking if there has been an Obama bounce, surely they are no longer asking.

The Republican side is a bit more sedate, probably because the leader there, John McCain, was hardly a top finisher in Iowa. The upward trend for McCain, and the downward one for Romney, predated the Iowa caucuses. At most the trends we saw earlier have largely continued. Huckabee appears to be the candidate without a bounce, in fact.

These are dynamics we've seen before when Iowa has had an impact. The short interval between Iowa has been much debated. One side says it doesn't allow enough time for an Iowa bounce to be fully felt. I'm of the opposite opinion. The short interval maximizes the effect of Iowa by not allowing time for losers in Iowa to retool their approach and for "added scrutiny" of the Iowa winner to slow their climb. The issue for Clinton and Romney is how to halt what is beginning to look like disastrous slides. With more time between events they would be better able to recover. If New Hampshire is a second loss for both, then both campaigns have to find ways to recover by South Carolina.

Iowa 2008 Trend, Entrance Poll and Outcome

2008-01-04T02:51:00.000-06:00

Last week I took a look at how the poll trend estimates did in the 2004 Iowa Democratic Caucus. This morning we have two new data points for comparison.

It has been a long day and night, so I won't say much tonight. We will look at the polls in more detail after a bit of sleep.

The bottom line in 2004 was that the polls under-estimated winners and over-estimated losers. (See the plot below for the 2004 comparison.) This year again the poll trend substantially underestimated the size of Obama's win. Clinton was quite well estimated, and Edwards did significantly better than the poll trend estimated.

The complex reallocation of preferences in the Democratic caucus also affected the entrance poll, which was quite close for Obama and Clinton but underestimated Edwards' final share of delegates.

The lower tier of candidates all finished below their poll trend estimates, though at such low levels of support that none of the errors are large.

On the Republican side, with a simpler form of voting at the caucus, the polls did a bit better, except again for substantially underestimating the winner, Mike Huckabee. Other candidates ended up with shares of the caucus vote pretty close to their poll trend estimates. Ron Paul did a little better than the poll trend and Giuliani a little worse.

Iowa Entrance Poll Results: Republicans

2008-01-04T01:11:00.000-06:00

The entrance poll tables can be found at MSNBC here. The widths of the bars are proportional to the size of the groups.

Iowa Entrance Poll Results: Democrats

2008-01-03T22:38:00.000-06:00

The entrance poll tables can be found at MSNBC here. The widths of the bars are proportional to the size of the groups.

Final Iowa Endgame

2008-01-03T14:49:00.000-06:00

The last of the Iowa polls are in (I hope!). Today we got the final Zoby/Reuters/C-SPAN tracker, ARG and an Insider Advantage (based on only the top 3 candidates and with 2nd choices reallocated.) See the posts at Pollster.com for the individual poll results.

This updated "endgame" chart shows the results for both the standard and the sensitive trends. In addition, I've plotted each of the Zogby tracker results. The trend line only uses the non-overlapping tracker results so as to avoid over-counting the number of truely independent surveys in a tracker. For Zogby, this means we use 12/26-29 and 12/30-1/2 as the two trackers that are included when estimating the trend.

The Insider Advantage poll released results for all Republican candidates, but only the top three Democrats. They reallocated the 2nd choices of other Democrats. Today's release from Insider Advantage has the race Obama 34, Edwards 33, Clinton 32. On December 31, Insider Advantage released reallocated support as Edwards 41, Clinton 34 and Obama 25. That's quite a lot of change in 3 days. Since it seems unlikely actual support shifted by this much between 12/28-29 when the first poll was conducted, and 1/2 when the second was done, this is a good example of how unstable results can be when reallocating votes. We should be left with a lot of uncertainty.

The Zogby Tracking poll has moved a bit in the Republican race, and quite a bit in the Democratic. The Zogby results see a sharp move toward Obama and away from Clinton since Monday, with the final day's results moving enormously for a tracking poll. Perhaps this reflects real change, but it suggests that the final night of interviewing was somewhere between 12 and 16 points different from the sum of the other three nights in order to move the Clinton estimate by 4 points in the final poll. Obama also is credited with a 3 point move between Tuesday and Wednesday's results, likewise requiring the Wednesday nights results were far from previous ones.

Finally, ARG's recent results show a bit less net change in the Democratic race, but continue to exhibit the consistent ARG house effect we've seen before. On the Republican side, ARG does see a shift towards Romney but also a late rise for Huckabee, while McCain has moved down in their polling so his ARG results are now in line with the trend estimate.

The standard trend estimate largely ignores this cacophony, with small upward trends for Romney, McCain, Clinton and Edwards, and a slightly downward slope for Huckabee. Obama is awfully close to flat.

The more sensitive red estimator is more excited about recent developments, thinking it sees some late movement toward Obama, but not for any other candidate.

The last word on these graphs is that they also show the relative lack of late polls from a variety of polling organizations. Most polling is again from a handful of sources. This makes meaningful comparisons of house effects all but impossible, and means that our estimates of late trends are much more dependent on those few pollsters than we would wish. The stability of the blue-line standard estimator is the best comfort we have. Even with the reduction on the variety of pollsters, we aren't seeing dramatic changes in the trend estimate, implying the results are in line with pre-Christmas surveys from a larger variety of organizations.

It is 2:30 out here on the West Coast, so 4:30 in Iowa. Two and a half hours until the voters start to vote. Let's see what they think.

Romney's Iowa Campaign

2008-01-03T12:24:00.000-06:00

May 27th, October 14th and November 28th, 2007. Three landmark days for the Romney campaign in Iowa.

From a distant third place in late 2006, Romney's Iowa efforts paid off on May 27th, when his estimated trend passed that of frontrunner Rudy Giuliani, making Romney the leader in Iowa. A campaign between two Northeastern Republicans had tilted from New York to Boston.

Romney held first place for 185 days, along the way picking up a strong win at the Ames straw poll in August. But the peril of becoming the odds on favorite is not getting credit for wins. The straw poll victory was noted, but always with the "as expected" modifier. The peril of the steady front runner is that wins are expected and losses are devastating. Meanwhile Huckabee's narrow win over Sam Brownback for 2nd place was the big news from the straw poll.

Huckabee built on success, and the next turning point for the Romney campaign was October 14th, the day Huckabee passed Giuliani's trend estimate and became the second place contender in Iowa. In the chart, the color changes at that point. Romney held steady in his lead over Huckabee for a while, but as the former Arkansas governor rose in the polls (with Romney steady, neither rising nor falling significantly) the lead evaporated and on November 28th Huckabee's trend crossed Romney's.

Of these three dates, May 27th was according to plan. Romney set out to overtake the national and Iowa frontrunner, and over about six months accomplished that.

What was perhaps underappreciated at the time was October 14th. That was the point at which the Iowa race in the eyes of voters shifted from a Boston-New York rivalry, to a Boston-Little Rock one. The campaigns adjusted earlier, but this was the tipping point for Iowa Republican voters. Now instead of comparing a Massachusetts Governor with a New York Mayor, voters were more likely after October 14 to think in terms of a Southern Governor as the alternative. Rural Iowa voters may have some difficulty telling one Northeastern Republican from another, but comparing one to a rural state Southerner is a somewhat easier task.

Once Huckabee became the primary alternative rather than Giuliani, the terms of the debate changed. This was most evident in the shift of conservative Protestant Christian voters in the state to Huckabee. Such voters may have had a hard time differentiating between the Mormon and the thrice divorced Catholic, but the Mormon vs the Baptist minister was easier to grasp.

The third turning point, November 28th, ended the 185 day reign of Romney in first place. Since then the polls have mostly seen a Huckabee lead, and the trend estimate, at least, sees little change in the small margin. The race remains a choice of Boston vs. Little Rock.

Tonight will determine whose supporters were committed enough to attend a caucus and whose stayed home. Romney's organization and financial advantage may yet pay off there.

But the hidden blessing of November 28th was that it gave Romney the chance to "win" Iowa. Had his lead held up from May 27th until January 3rd, a first place finish would again have been "as expected". Good. Far better than a loss. But not the momentum builder and validation that a candidate hopes for out of Iowa.

November 28th changed that. If Mitt Romney pulls out a victory, however small the margin, he will have "come back" to win, and will get the boost that Iowa can offer to candidates who are not yet national household names. (And remember, Romney still stands in only 3rd place nationally, barely over 15%. He needs what Iowa can do for him.)

In the scenarios of what can happen after tonight, Romney needs the Iowa win to help in New Hampshire against the "risen from the dead" McCain surge. A loss makes the news from now until Tuesday all about the Huckabee's win and McCain's surge-- two things the Romney campaign would not like to carry into New Hampshire.

Until late November, everything had gone according to plan for the Romney campaign. They had built leads in Iowa and New Hampshire and had just moved into first place in South Carolina. While still lagging in national polls, the Romney machine was well poised to sweep three early events and ride momentum into Florida and then Super Tuesday. November 28th changed all that. Now a loss in Iowa threatens the next two states. But paradoxically November 28th also marks the moment Romney got the chance to be a "surprise" winner again.

Which scenario plays out now rests in the hands of Iowa Republican caucus goers.

Iowa Endgame Polling

2008-01-01T11:04:00.001-06:00

Happy New Year, and let's look at the endgame of Iowa polling.

There have been seeming sharply contradictory polls since Christmas. Within the week ARG has had Romney up 9 over Huckabee while Research 2000 had Huckabee up 7 over Romney. We've also seen Clinton up 7 over Obama in ARG and Obama leading Clinton by 7 in the Des Moines Register/Selzer poll. And none of this looks to be because of very strong late trends. Depending on when you start and how hard you squint you can see some modest trends or no trends at all in the polling since mid-December. Our trend estimators, both standard blue and sensitive red, see small trends over each day, but the noise of the individual polls around these trends is large. (These trend estimates are based on ALL the Iowa data, not just the polls shown in the plots. They are the same estimators you see at Pollster.com and here.)

The current trend estimator puts the Republican order at Huckabee 31.2%, Romney 26.8% and McCain 11.9%. For the Democrats the trend estimates are Clinton 29.5%, Obama 26.6% and Edwards 25.2%.

The variability in the polls means that there is lots of room for cherry picking the poll you like. Since the Des Moines Register/Selzer & Co. poll came out last night, the Edwards and Clinton campaigns have found fatal flaws in what was (before Monday night) the most respected poll in Iowa. No surprise there.

(See Mark Blumenthal's write up of our Pollster.com "Poll of Pollsters" on the reputational rankings of the various pollsters.)

The closeness of the Democratic race makes folly of any attempt to crown a winer based on the polls. Aside from the polling difficulties, the unknown ability of the campaigns to mobilize their supports, and the effectiveness of turning out first time caucus goers is enough to make staying up Thursday night worthwhile.

On the Republican side, Huckabee leads in the trend, but it is worth noticing how consistent Romney's poll results have been since Christmas compared to how variable Huckabee's numbers have been. Add to that the bad media days Huckabee has been suffering, including yesterday's press conference to announce he would not run the negative ad he then showed, and I think you have to wonder if this endgame may yet shift by Thursday night.

Sensitive Trends Update: IA, NH, SC and US

2007-12-29T17:05:00.000-06:00

Here is an update comparing the standard "blue line" estimate of poll trends with the more sensitive "red line", and a simple 5 poll moving average for comparison.

I talked at length about these estimators and issues related to them in an earlier post here, so won't comment further. See the earlier post for details.

These data include polls released through December 29, with ARG's 12/26-28, Research2000's 12/26-27, Strategic Visions 12/26-27 and LATimes/Bloomberg's 12/20-23,26 polling as the latest from Iowa.

The Ghost of Caucus Past

2007-12-29T14:32:00.000-06:00

As we pour over the latest data, it is worth taking a look back at a previous year in Iowa.

The Democratic contest in 2004 was strikingly dynamic, with a sudden surge for Kerry and Edwards as Gephardt and Dean slumped. (And note the gap in polling. Those are the holidays we are in right now when polling wasn't done.) By the last polls, Kerry had established a clear lead, but Dean, Edwards and Gephardt were within four points of each other. The polling got the trends pretty close to right. Edwards was clearly on his way up through the last 12 days of the race, as was Kerry. Dean may or may not have reversed his fall in the last week, and the polls said Gephardt was coming down. So far, so good.

But the Democratic caucus process does a lot to change the outcomes, with supporters of non-viable candidates joining forces with their second choices. That process is likely to boost the well off candidate, while robbing the struggling campaigns. And of course there is always the issue of which candidate's supporters actually turn out on caucus night.

The entrance poll is the best measure we have of whose supporters actually show up on caucus night, even with all the appropriate cautions about the entrance poll itself. Comparing the entrance poll to that last estimated trend value for candidate support from pre-caucus polling, we can see how the two rising campaigns did even better than the polling predicted. Kerry's last poll trend estimate was 25.9%, good for first place. But his entrance poll support was 34.8%. For Edwards, his final trend estimate was 21.4%, but the entrance poll found 26.2% supporting him.

For Gephardt the story was the opposite. His supporters stayed home on caucus night. The pre-caucus trend had Gephardt falling but at 17.8%. But the entrance poll found only 10.3% to be Gephardt supporters.

Dean was the only top candidate whose preelection and entrance poll numbers match closely. His trend estimate was 20.3% and the entrance poll put his support at 20.5%.

Caucus night had the effect of stretching out the differences between the candidates, advantaging the top two while damaging the fourth place candidate. (Dennis Kucinich was the exception in the back of the pack, with a trend estimate of 1.7 but an entrance support of 4%.)

Then they vote. And form coalitions with non-viable supporters. And weight the delegates in a complex formula and finally there is an allocation of delegates to the state convention. That allocation is the best we can do to consider a "final" outcome of this process.

The results there further favored Kerry and Edwards. Kerry moved up to 38% of delegates, from 34.8% in the entrance poll and 25.9% in the pre-caucus poll trend. Edwards got 32% of delegates, up from his 26.2% in the entrance poll and 21.4% in the poll trend.

Dean ended up with 18% of delegates, down a bit from the 20.5% in the entrance and 20.3% in the poll trend. And Gephardt's delegates just about matched his entrance poll, 11% of delegates and 10.3% in the entrance poll. A disappointment from his 17.8% final trend estimate.

So let's take one important lesson away from this Caucus Past. The pre-election poll trend got the order of finish right, if only by a point separating Edwards and Dean. But the process of caucus night that makes it tough to come out means that enthusiastic supporters are more likely to turn out than those who are discouraged by recent slippage. That is probably true for both parties. On the Democratic side, the complex voting and coalition formation further exaggerates the lead of the top candidates and diminishes the showing of marginal ones. And that process is seen even in comparison with the entrance polls, let alone the pre-election polling.

In Iowa the "outcome" is quite a few steps removed from the simple balance of preferences among the population. The mechanisms themselves intervene to affect the final delegate counts. So don't expect to see the pre-election polls hit the final delegate percentages very closely. If the polls get the order right, that will be good enough.

For a fair test of how good the polling is this year, wait for New Hampshire where pollsters poll, voters vote, and they just count up the results.

Primary State Voters Making Up Their Minds

2007-12-29T13:32:00.000-06:00

Voters in the early primary states are making up their minds, or at least picking a candidate when pollsters call. For all that has been written about how unsettled the races in both parties have been, we are now seeing a clear decline in the rate of "don't know" answers to the vote question. Interestingly, that trend is clear in the earliest primary/caucus states, but only faintly visible if at all for the nation as a whole. This is further evidence that early state voters really do pay more attention and move to a choice sooner than the nation as a whole. Looming election days concentrate the mind apparently.

In both parties, the undecided rate has turned down the most in Iowa, to about 5% or a shade less. This is down from about 14% early in the year for Democrats and over 15% for Republicans. Even as recently as November 1, over 14% of Republicans were unable to give a candidate choice. Democrats were at about 11% undecided at that time. In both parties the undecided rate has fallen rapidly in November and December.

New Hampshire also shows some fast movement to a decision. NH Dems have been steadily decreasing their don't know rate all year from 17% in January, but it still stands around 10% across the most recent polls. NH Republicans were a bit slower to choose until the first of December. Since then we've seen a rapid decline to about 8% now.

And in South Carolina where there are fewer polls and the undecided rate has been all over the place, we are seeing evidence of more decision making since early November. That pattern has held for both parties, with Dems now at about 9% and Reps at around 10%.

Nationally there is no trend at all in Democratic undecided rates, which have held at 10% all year. For Republicans there has been a little movement nationally, down from about 15% to about 11% since late October.

Pollsters allow voters a variety of ways to say they haven't decided, so there are various ways we can measure the crystallization of preferences. One is to just use the percent who say they are "undecided", which is simple enough. But some voters, especially early on, pick options like "someone else" or "won't vote" and pollsters vary in how the report the no-preference alternatives. So I've calculated the percent who fail to choose any of the candidates the poll asks about. This is the blue "no preference" line. As it happens, these alternative measures track together pretty well, and recently any gap between them has largely vanished.

You can also see large differences across pollsters in how large an undecided rate they produce. A few national polls have zero percent undecided, while the highs at the same time are over 25% for Republicans and over 20% for Democrats. This is one of the sources of house effects in surveys.

There is one methodological issue that we can't address with these data. Near the end of the race some pollsters push voters harder to get a response to the vote question. That would, of course, artificially lower the "don't know" rate, and to some extent may be what we are seeing in these data. Such practices are not normally disclosed so there is no way to statistically adjust for them here.

Graphing Debate Time by Candidate

2007-12-28T11:23:00.000-06:00

A brief break from polls to comment on graphics and politics. Today's New York Times has an op-ed by NBC's political director Chuck Todd and a graphic designed by Nicholas Felton. The text and graphic are here. The text describes the data (quite completely-- an unusual but welcome touch!) noting that candidates are stratified by time in rough line with their poll standing and that debates played a part in both the rise of Mike Huckabee and the slippage of Hillary Clinton.

The graphic is a variation on a pie chart, showing the total number of minutes each candidate spoke in the 21 debates held in 2007. It is an odd fact that statisticians and analysts of statistical graphics universally hate pie charts, while graphic designers for mass media love them. The latter seem to equate statistical graphics with pie charts, while the former have ranted for years about their defects.

The appealing metaphor of a pie chart is its division of a whole into parts. Whatever the slices represent, they have to add up to a "whole" pie. The trouble here is that the text and data are about amounts of time, and only implicitly about the share of a total that each candidate receives. Further, the pie mixes the two parties into one whole pie, but there really should be two pies, each divided within party and assuming we care about shares of the pies, since Republicans can't eat any of the Democratic debate pie, nor vice versa.

When what we want to compare are magnitudes, rather than shares of a whole, the data are more clearly presented as distances rather than areas. It is easy to compare which distances are longer than others, and relatively difficult to see differences between the areas of pie slices, especially when the slices are not adjacent to each other.

So let's look at the same data in a different format and see what we can see.

The chart above reproduces the information given in the original pie chart. It plots the number of minutes each candidate spoke during all debates, and distinguishes party by the use of color (here red and blue, in the pie chart by shading.)

The main points made in the op-ed are also evident here. There is clearly a great deal of stratification across the candidates, with front runners getting more time than the "back of the pack" candidates. But there are a few comparisons I think stand out more in this chart. The advantage of Obama and Clinton over Edwards and Richardson is clear here. Richardson got only about 2/3 of the time of Obama, with Edwards a bit better but still well short of Clinton. The next cluster of Democrats-- Dodd, Biden and Kucinich-- got only about half the time of Obama, with Gravel far behind even that.

On the Republican side, Giuliani and Romney were closely matched with McCain a little bit behind. As with the Democrats, there is then a large gap until we reach Huckabee at about 3/4 of Giuliani's time. A smaller but still clear gap separates Huckabee from the cluster of Paul, Hunter, Fred Thompson and Tancredo. Another gap separates Brownback and then one more puts the short-lived candidacies of Tommy Thompson and Gilmore and the single appearance by Keyes together.

To my eye, these differences are easier to perceive and compare when the number of minutes is simply the location of the dot in the chart above than when it is the area of a pie slice.

There is actually more data presented in the text of the op-ed than is present in the graphic. Todd's text notes differences in number of debates by party (11 for Dems, 10 for Reps), which means Dems should have about 10% more total time available than Reps (if, that is, the debates were equal length, something we don't know from the text but see below). He also gives the data on how many debates each candidate participated in, an obviously important point since we are comparing Keyes' minutes in a single debate with Obama's time in 11 debates. For some purposes we might care only for total time, but for others we might want to adjust for number of debates. I do that in the chart below, which shows the average number of minutes per debate for debates in which the candidate participated.

One immediately clear point in this chart is that Obama and Clinton retain their advantage over Giuliani and Romney even when we adjust for the extra Dem debate. Giuliani and Romney got the same time per debate that Edwards and Richardson received, but that leaves them well back of Obama and Clinton.

The most important shift in the chart is the movement of Fred Thompson to the midst of the top 4 Republicans. Thompson only participated in 5 of the 10 debates, so his total time in the first chart (and the original pie chart) dramatically misrepresents the attention he received after entering the race. Thompson received substantially more time per debate than did Huckabee, though in 10 debates Huckabee had 73 total minutes to the 49 Thompson got in 5 debates, as shown in the first chart.

In his one debate, Alan Keyes got more time than any other second tier Republican averaged over more debates.

But why do Giuliani and Romney continue to get less time per debate than do Obama and Clinton? Todd's text points out that there are 12 Republican candidates but only 8 Democrats who have to divide the time. That seems reasonable, but the data are a bit more complicated. While there are 12 Republicans in the charts, three of them participated in four or fewer debates while all but 1 of the 8 Democrats participated in at least 10 debates. When we count total candidate debate appearances, the Republicans had 88 and the Democrats 81, less than a 10% difference due to number of candidates and appearances, not the 12 to 8 ratio of candidates.

What is quite different is the total number of minutes the candidates of each party spoke. Democrats totaled 794 minutes over 11 debates, a total time per debate of 72.2 minutes of candidates actually speaking. For Republicans, the total in 10 debates was 666 minutes, or 66.6 minutes per debate. So the Democratic advantage in the original pie chart and my first chart above has built into it a longer total for Democrats regardless of the number of debates.

Further, if we divide by number of candidate appearances, we get how many minutes each candidate would have gotten if the total speaking time had been divided exactly equally for each debate appearance. The result for Democrats is 794/81=9.8 minutes per candidate per debate, while Republicans had 666/88=7.6 minutes per candidate per debate. For whatever reasons of debate format and schedule, Democrats enjoyed more time to talk even adjusting for number of candidates and number of debates. (Had Reps had the same 81 appearances as Dems, they would still have only had 8.2 minutes each.)

So the disadvantage in minutes per debate even for Republican frontrunners compared to Democratic leaders is not just an artifact of number of debates or of candidates. It is a real difference and it might be of political interest to know what accounts for it. Did Republican debates run shorter on average? Did questions to Republicans run longer on average, leaving less time for answers? The data don't answer these questions. But over even an equal number of 10 debates and equal number of candidates, Democrats would have enjoyed almost an hour longer to speak (721 minutes vs 666.)

Let's adjust the speaking time to show which candidates got more than their fair share relative to the time available per candidate per debate. In this case, 100% means the candidate got exactly the "fair share", or 7.6 minutes per debate for Republicans and 9.8 minutes per debate for Democrats. On this scale, leading candidates got up to 140% of their party's fair share, while the lowest share was 63%.

This new scale now removes the differences in total time between parties, and lets us compare relative advantage or disadvantage between parties and candidates. The data are plotted below.

Now that we are no longer confounding differences in total time between parties, new perspectives emerge from the data. The top two candidates in both parties were equally advantaged-- all four got about 140% of the time that an equal time rule would have given them. The previous comparisons masked this due to the shorter Republican times. But relative to a fair division of time, the top two were treated almost identically in both parties.
But the 3rd and 4th places were treated rather differently. In the Republican debates, McCain and Fred Thompson received about 120% of a fair share, while on the Democratic side Edwards and Richardson got only slightly more than a fair share would entitle them to, Richardson at 101% and Edwards at 109%. Based on this share of time comparison, then, the debates treated the Republican race as more of a 4 person contest, while the Democratic debates divided a top 2 from an "average" third and fourth.

The extra shares for leaders must come, of course, from the rest of the pack. Huckabee who now threatens to win Iowa has received only a 96% share of a fair time allocation. The bottom 5 among Republicans all got less than 80% shares. Among Democrats the gap between Richardson and the rest is from 101% to 80.7% and below.

So total time favored the Democrats, but did so even after adjusting for number of debates and debate participants. The separation into top 2 vs 3rd and 4th was especially clear for Democrats. The Republican race gave relatively more time to 3rd and 4th place candidates.

The text of the op-ed only contains four sentences that give interpretations of the data:

The front-running Democrats, thanks mostly to a smaller field (but also to one additional debate), got a lot more time to speak than the front-running Republicans.
Not surprisingly, the times for each candidate seem to follow the polls, with the leading contenders getting more minutes. As Mr. Huckabee’s poll numbers rose, his speaking time increased.
The debates had effects on both voters and candidates. Mr. Huckabee’s performances helped him emerge from the pack, and a few tough moments for Mrs. Clinton set the stage for her eventual fall in the polls.

The first point is right that Dems got more time, as the charts all show. But that advantage wasn't entirely due to fewer candidates and one more debate. The advantage was more real than that: Democrats got more time to speak per debate and per candidate. Changing the graph allows us to see this in a way the pie chart did not.

Also revealed by the charts here are systematic differences between candidates that illuminate the nature of the stratification within and between parties. Front runners are advantaged, but the Democratic race was treated as having 2 clear leaders while the Republican race had 4 apparent contenders, based on speaking time. That too is masked by the pie chart.

Finally, two of Todd's three points above are not addressed by the graphic or the data given. While it is clear that the order of speaking times roughly follows support in the polls, this is not entirely the case. For example Biden has more poll support than Dodd, yet Dodd got slightly more speaking time. (For that matter, and more powerfully, Clinton held a large lead in the national polls during almost all of the debates, yet trails Obama in total time.) Moreover, the dynamic element Todd mentions is not illustrated by the graphic at all. If Huckabee's speaking time rose with his polls we can't see it here. And did Thompson's time drop with his polls? Alas, the op-ed doesn't list these data (which would be quite lengthy.) But a graphic could have illustrated this dynamic aspect of the data in no more space than the pie chart.

Nor does the pie chart provide any evidence of the role of debates in the rise of Huckabee or the decline of Clinton. Did the polls move up or down noticeably following any of the debates? Did candidate time in one debate precede a rise in polls, or did a rise in polls precede more speaking time? We could see these things in a graph, but it would take well more than a thousand words to describe them. The better the graph the more words it is worth.

Pollster Variation in Iowa

2007-12-26T13:59:00.001-06:00

(LARGE graphs-- you probably should click once or twice to see them at full resolution in order to see the details.)

A new American Research Group (ARG) poll of Iowa has caused quite a debate in the comments at Pollster.com. Today Mark Blumenthal takes a close look at polling the "Dark Side of the Moon" in Iowa during the holiday season and the unknowns involved, including the unknowns of the ARG poll. So while the pollsters are busy trying to get one more Iowa poll in this week, let's look at the track records of the pollsters in Iowa.

The chart above shows the polling of the Democratic nomination race in Iowa since January. In 2007, twenty four different polling organizations conducted Democratic caucus polls. Of these, only 7 have conducted three polls or more. That means that for most pollsters, we have no way to separate random error from systematic "house" effects. To do that, we need multiple polls, and to do it reliably we need a number of polls from each organization. But we can at least look at the seven organizations with three or more polls, and see how they compare to the trend estimates. The trend, of course, reflects the best estimate across all pollsters, but until we see the actual vote (itself a very slippery concept in Iowa) we can't say if the trend was better than individual polls or not. Still, this reveals when pollsters seem to follow the trend and when they systematically seem to miss it.

The clear result of the comparison above is that ARG generally showed a much better performance for Clinton than the trend in the first half of the year. From January through June, ARG usually had Clinton some 6-10 points above the trend estimate, with one exception in which ARG agreed almost exactly with the Clinton trend. During this period, ARG was the most discrepant of all pollsters from the Clinton trend.

In the second half of the year, ARG's polling has generally been much closer to the trend estimates, usually less than 4 points away from the Clinton trend. During this time ARG has mixed in with other polling pretty consistently.

That is, until the most recent December 20-23 poll, which again shows a large Clinton deviation of about +6 points from the trend. (I'm using the standard blue trend line. The upturn in the red "sensitive" estimate is interesting, but it is also sensitive to the ARG poll, an example of why you might not entirely trust the red estimator.) This is the first large ARG deviation recently, and quite a change from ARG's previous poll of Dec 16-19 which was close to trend.

In an inversion of the Clinton effect, ARG consistently underestimated Obama's support, compared to the trend, in the first half of the year, but their polling fell much closer to the trend during the second half. At least until the latest poll which has Obama some 10 points below the standard trend. (Again, note the downturn of the sensitive estimator, which would be turning down even without ARG, but which is turning down somewhat more because of ARG.)

ARG's polling for Edwards has been more variable. Early 2007 polling fluctuated quite a bit, sometimes below and sometimes above trend. More recent ARG results for Edwards has generally been a few points below trend, with the latest result about as much below trend as has been "normal" for ARG recently.

ARG's results this year have been more heterogeneous than some oversimplifications claim. There was a substantial overestimate of Clinton early in the year, along with an underestimate of Obama's support. That was important during this period because there were relatively few other polls available for Iowa in this period. But in the second half of the year, the ARG results for Clinton and Obama have been much closer to trend estimates, though still with a small average advantage for Clinton and small disadvantage for Obama.

In light of this, the ARG poll for December 20-23 does look out of line with their own previous polling, certainly their polling of the last 4-5 months.

Let's look at their Republican results for Iowa.

On the Republican side, ARG's results have been especially favorable to John McCain, and to a lesser but still substantial degree to Giuliani. Unlike the Democratic results, these effects have not diminished much in the second half of the year. McCain has often been as much as 10 points above trend, with only one poll below trend and one more right on the trend. No other pollster has been so consistently far from the trend for McCain.

For Giulinai, the results are less far from trend, but still quite consistently above trend. Only 2 of the last 10 ARG polls have Giuliani below trend.

Romney has fared close to trend in the ARG surveys, though on average a bit below trend. The discrepancies for Romney are much less dramatic than for McCain or Giuliani.

The last two ARG polls show shifts of -3 and +4 points for McCain and Romney respectively, and a single point difference for Giuliani. (And a -5 and +6 for Huckabee and Paul.) For the Dems the shifts were +5, -6 and +2 for Clinton, Obama and Edwards respecively.

As Mark Blumenthal notes, the reasons for these discrepancies are largely matters of speculation. But the consistency of the ARG house effects are pretty clear in these data. The ARG results currently stand on the same side as their long term house effects: above trend for Clinton, Giuliani and McCain, and below trend for Obama, Edwards and Romney. Compared to other pollsters, these house effects for ARG appear to be the largest of any polling firm in Iowa.

Comparing Primary Trend Estimates

2007-12-21T15:31:00.001-06:00

Lots of exciting movement in the polling in both parties has pushed me to review the bidding on trend estimators. As regular readers know, the "blue" line estimator that is our standard is deliberately tuned to be a bit conservative. It requires a good bit of evidence that a change in direction is "real" before the trend will move sharply. With lots of polling, this estimator has an excellent track record of finding turning points of opinion while not chasing wild geese.

But in a hot primary, with relatively few polls each week (this week has been an exception!) it is reasonable to ask if there is short term change taking place that "old blue" just isn't quick enough to catch. So let's take a look at two alternatives.

Before we go there, let's remember that the variation across polls is quite large compared to some of the changes in support we are talking about. The variation around the trend estimates here is about +/-5 points while the trend differences are often a point or two. That means we are using quite noisy polls to estimate trends that vary much less than do the individual polls. (Mark Blumenthal has spent much of the fall discussing the variation in poll methodology and the implications this has for how uncertain we should be about individual results.)

So let's think of the single most sensitive alternative estimator we could pick: the latest poll. That would certainly move rapidly, and so be "responsive". But it would also reflect individual "house" effects due to polling organization and practices. It would also be highly unstable as an estimate of support, because individual polls vary over that approximate +/- 5 point or more range we see in the plots. In effect, this most sensitive possible estimator would just connect the dots and produce a plot that looks a lot like an earthquake on a seismograph. Lot of noise, but hard to see the systematic trend.

So we'd like to smooth out this random variation (and non-random variation due to house effects). One option is to take a rolling average. The more polls in the average the smoother the result, and you can take your pick of 5, 10 or more polls to smooth over. Of course, the more your include in the average, the more out of date the average is because it includes some polls taken a while back. I've chosen a 5 poll average here, because it should be quite sensitive yet still gain some of the advantages of averaging. Using more polls would smooth more, but defeat the purpose of having an especially sensitive estimate of trend.

The second comparison uses the same "local regression" methodology that is our standard approach here, but sets the degree of smoothing to about half that of the standard blue estimator. This "red line" estimator is more sensitive than the standard, but not as prone to jumping around as the moving average. "Red" should detect short term change more quickly than "blue", but it will also chase phantom changes due to flukes of a few polls that happen to be too high or too low. (The 5 poll average will be even more susceptible to this.)

So what do we see when we compare these estimators in IA, NH, SC and the US?

For the most part, all are in substantial agreement about big trends over the full year. The red line and black moving average show more variation than does blue, and may have picked up some "real" short term change that blue considered noise. But over all the 44 state x candidate x party comparisons, the agreement among estimators is pretty close most of the time.

For the Dems in Iowa, all three estimators are in quite close agreement right now. The differences are in the range of a point of each other. A sharp eye can see some differences of trajectory. For example, Clinton in Iowa is trending down in the blue estimator, while red sees a very recent upward trend while the black moving average fluctuates erratically. But zoom in and the differences are about a half a point or so. My experience is that you just can't reliably estimate such small differences, but feel free to make your own call here.

On the Republican side in Iowa you see similar agreement with small differences near the end. Small differences for Romney and Huckabee are seen in comparing red and blue estimators-- Red and the moving average see a bit of upturn for Romney and downturn for Huckabee in the most recent polls that the blue estimator isn't convinced of. As with the Dems, none of these differences is very large.

In New Hampshire, the picture is essentially the same for the Dems. Blue sees Clinton moving down, Obama up and Edwards gaining more slowly. Red and the MA think Clinton has turned back up in the last few days, while Obama has stalled or turned down. But all these differences are again matters of at most a percentage point difference in estimated current support.

For Republicans in New Hampshire, there are somewhat bigger differences for Romney and McCain, though the trends agree for the other candidates. Red and MA think Romney took a turn down recently by as much as a couple of points, while blue sees a continuing upward trend. The blue and red estimators are still within 2 points of each other, but a real difference in upward or downward momentum would, of course, be important.

For McCain, the latest couple of polls show a substantial spike in support, and red and MA chase that spike, leading to the largest difference we've seen so far among the estimators. All three see McCain gaining, but red puts him about 4 points higher than does the standard blue trend, while MA is a point lower than red.

In South Carolina where there has been less polling and more noise, we see the biggest differences of all. It is worth appreciating what a huge range of results we've seen in recent SC polling for Clinton and Obama. I am not willing to believe that the true level of support has really varied between 20 and 45 points! But this makes the estimation especially tricky (and is why I prefer the stability that the blue estimator provides in the face of extremely noisy polling.

Blue sees Clinton as flat for some while, in the process splitting the difference between some quite high polls and other quite low ones. Either her support has suddently collapsed (but with simultaneous high and low polls) or the best bet is what the blue line estimates. Red and the MA in contrast see a downturn recently from about 43 to about 35, a major drop.

For Obama, all three see an upward trend, but with red and MA moving up much more sharply than blue. Blue puts Obama at 31, while red would go for 35 or 36.

For SC Republicans the big difference is with Huckabee, where red and MA see a very large recent gain, while blue agrees on the sharp trend but doesn't put the support as high yet. Blue puts Huckabee at about 20, while red and MA would go as high as 28.

There are small differences of recent trend for Romney and Giuliani, but these are quite small-- more on the order of the Iowa differences.

Finally, on the national scene, we gain the advantages of more dense polling. The Democratic trends are quite close to one another. And on the Republican side, even the rapid rise of Huckabee is picked up quite well and with close agreement among all three estimators.

The bottom line is that most of the differences we see among the estimators are small-- on the order of a point or two in the estimates. The apparent differences in the most recent trends strike me as generally being too small to reliably distinguish. What constitutes a "real" change in trend is hard to define, but I think most of the current differences are too small to put a lot of faith in.

In the case of the large differences in South Carolina, I'm inclined to pay more attention to the very large spread across individual polls, and demand clearer evidence of change, but the red and moving averages are perhaps telling us that real change is taking place. More polls would help, but in their absence I think a more prudent reading is that it is hard to know exactly what is happening. (And again I'd refer readers to Mark's posts on the differences across pollsters in practices and methods.)

But the bottom line is this is a fun game. I'll be updating with all three trend lines so you can pick your favorite and place your bets accordingly. Starting January 3 we'll begin to see how the polls and the trends line up with actual votes.

Bush Approval: Still around 33%

2007-12-12T08:58:00.001-06:00

President Bush's approval rating remains about where it has been for several months: 33% plus or minus a fraction. The current trend estimate is 33.2%, including December polling from LATimes/Bloomberg, AP/Ipsos, CBS/NYT, ABC/WP and USAToday/Gallup, the last from 12/6-9/07, finding 37% approve and 58% disapprove.

This now represents one of the longest periods of stable approval for Bush. His presidency has been characterized by a long decline from the post 9/11 highs, interrupted by generally short rallies and spikes due to the start of the Iraq war and the capture of Saddam Hussein. The most important exception has been the long rise in approval starting in March 2004 and continuing through the election in November of that year. The second term has been marked more by long term decline, but with rallies up in late 2005 and the late spring and summer of 2006. Rarely has support simply held stable, neither rising nor falling.

The only period of nearly comparable length was the winter and early spring of 2007, when approval held at a steady 34% for nearly 4 months.

The current plateau at around 33% has now lasted slightly longer, starting a rise in late July and stabilizing around 33% by September. Since that time, trend estimates have varied within less than a percentage point of 33%.

The current polling shows considerable variation, from a low of 28% in CBS/NYT to a high of 37% in the latest Gallup poll (both taken within a week of each other.) The CBS/NYT just barely crosses over into outlier territory in the residuals plot below. Even with these two extremes removed, the remaining polls also balance around 33%.

The final plot below shows that the sensitive "red" estimator is in agreement with the more conservative "blue" standard trend estimate. Neither see much change in recent polls.

If you want dynamic exciting polling, better turn to the primary races.

Zogby Internet Poll Trial Heats are Odd

2007-11-26T21:43:00.000-06:00

A new Zogby Interactive poll, conducted using volunteers over the internet, has produced some odd results for trial heats involving Senator Clinton against all four top Republican opponents. What makes this especially odd is that the results are not equally unusual for Obama.

This poll was reported by Reuters' John Whitesides, who also reports on the Reuters sponsored polling Zogby does by conventional telephone methods. The similarities in the reports make it hard to tell, but apparently these results are not part of the Reuters-Zogby polling partnership, but are independent work by Zogby Interactive. Likewise Zogby's website posts the results without mention of who sponsored the work, so presumably Reuters did not.

The Zogby poll was conducted 11/21-26/07 with 9150 respondents who had agreed to take part in Zogby's online polling. This is not a normal random sample of the population. More on the technical issues below.

The hugely surprising result is that the Zogby poll finds Sen. Hillary Clinton losing to all four top Republicans in head-to-head trial heats. What makes that surprising is that Clinton LEADS all four of those Republicans in the trend estimates based on all other polling by between 3.8 and 11.6 points. Zogby also has Clinton losing to Arkansas Governor Mike Huckabee by 5 points. There are too few Clinton-Huckabee trial heat polls from other organizations for me to compute a trend estimate for that comparison.

The chart above shows all the trial heat data from national polling and the estimated trend lines for each pairing. The data points for the new Zogby data are indicated in the charts as "Zogby Inet" in blue for Clinton and red for each Republican.

What is immediately clear is that the Zogby Clinton numbers are well below the estimated trend for Clinton in each of the four comparisons. Clinton is consistently 8-10 points below her trend estimate based on other polling.

In contrast, the Republican results are quite close to the trend estimate in most cases: Giuliani is at 43 in Zogby, with a trend of 44. Romney is 43 in Zogby, 38.3 in trend; Thompson is 44 in Zogby, 41.3 trend, and McCain is 42 Zogby, 42.7 trend. Those Republican numbers are about the kind of normal noise we see around the trend estimate, so don't seem out of line.

Why then is Clinton so far down in comparison to other polls? The Reuters story doesn't note that these results are far from other polling, and instead uses the theme that Clinton is declining to frame these Zogby results:

The results come as other national polls show the race for the Democratic nomination tightening five weeks before the first contest in Iowa, which kicks off the state-by-state nomination battles in each party.

Some Democrats have expressed concerns about the former first lady's electability in a race against Republicans. The survey showed Clinton not performing as well as Obama and Edwards among independents and younger voters, pollster John Zogby said.

While this is certainly a theme of recent reporting, boosted by a pre-Thanksgiving ABC/WP poll showing Obama leading Clinton in Iowa, it is striking that no other poll has found recent results as far from the trend estimates as are Zogby's results and that the Reuters story fails to note that fact.

One answer to why Clinton does so badly MIGHT be that the poll has too few Democrats and thus biases its results. But if that were so, we'd expect Obama to also underperform his trend estimates. That doesn't happen, as the chart below makes clear.

The Zogby results for Obama are all quite close to his trend estimate from all polls:
Zogby has Obama at 46% vs Giuliani, while the trend puts him at 44.3. Against Romney Zogby has Obama at 46%, while trend says 46.6. Against Thompson Zogby has Obama at 47, while trend is 47.0, and against McCain Zogby has Obama at 45 while trend puts him at 43.4.

This is clearly not consistent with a general anti-Democratic bias in the Zogby Internet poll. It is also clear from the graph that the Obama pairings find Republicans doing quite close to the trend estimates as they did against Clinton.

(Trial heats against Edwards are not very common recently, so the Zogby results for him lack much polling for comparison.)

And so we are left with a puzzle: What is it about these respondents that so strongly affects Clinton support but no one else?

We can probably rule out one easy explanation: That Clinton has suddenly collapsed and Zogby is just the first to find it. The reason is internal to the Zogby result. If Clinton really has suddenly become 10 points less attractive, we'd expect all four Republicans paired against her to do BETTER than their trend estimates when facing her. But what happens is Clinton goes down and they don't do any better. That is hard to reconcile with a real change in Clinton's support. (A tortured version would say Clinton must have collapsed among Dems who now say they are undecided while refusing to move towards any of the Republicans. But that isn't usually what happens in real data when one candidate declines sharply. Usually the other moves up at least a bit, drawing not only from unhappy partisans but especially from independents who now are disenchanted with the former front-runner. So while you could make the math work with this story, it doesn't seem very well supported by the data.)

The Zogby Internet polling has a questionable track record in statewide races for Senate and Governor in 2006, where they often far over-estimated the competitiveness of races compared to conventional phone polls taken at the same time. One way to make sense of those problems turns out not to help much here. It is reasonable that the people who volunteer to take political polls over the internet are considerably more interested in politics (and likely more strongly partisan) than is a random sample of likely voters. That should be expected to lead to fewer people with "don't know" responses as better informed and more partisan respondents are likely to both know more about the candidates and to have made up their minds sooner than a proper random sample. That helps explain why Zogby's 2006 internet polls looked as they did.

But this does no good in Clinton's case. What we see is that MORE internet respondents are undecided about their vote between Clinton and four Republicans than the trend estimates based on less involved and partisan phone samples show. The Zogby undecided rates for the Clinton pairings are 20, 17, 17 and 16% (plus 17% undecided in the Huckabee comparison.) The comparable undecided rates based on the trend estimates are 8.2, 12.8, 9.0 and 10.6. That is an average undecided rate of 17.5 in Zogby vs 10.15 in the trends. Likewise the undecided rate is slightly lower for Obama pairings than it is for Clinton: 17, 13, 14, 13, and 14 for Huckabee. How could it be that a sample that is almost certainly more involved, knowledgeable and partisan can be LESS decided about Cinton, the single best known figure in the race? Again, a tortured story might be constructed, but I think a simpler explanation is that this result is not consistent within the Zogby data itself, or in comparison with outside polling.

Where does this leave us? Puzzled. If these results came from voting machines, I'd suspect that something in the ballot design or the recording mechanism caused a modest but consistent undercount of the Clinton support. The effect seems confined only to that one candidate, and not to any others, Democrats or Republicans. And there was no boost in support for the Republicans paired against Clinton. In this case, I'm similarly inclined to wonder if there is the possibility that the Zogby online survey had a glitch that caused a systematic "undervote" for Clinton. Certainly if my research assistant brought me these results, I'd want to check the software for mistakes before I published it.

Let's assume the Zogby organization has checked for any such possible mistakes or glitches and has ruled that out. (One would assume they were as surprised by the data as anyone and since their reputation is on the line, would have checked very carefully before releasing the data.) Is there any reasonable model of how candidate preferences are evolving that might explain this result, and the stability of Republicans paired against Clinton AND the stability of Obama support and that of his Republican pairings?

Without access to the raw data it is impossible to test any speculation here. But here is one possibility: Internet polls, presumably including Zogby's, use weighting to adjust for non-representativeness in their volunteer respondents. (There is a huge debate about whether this, and more sophisticated approaches, can produce generalizable population estimates with good statistical properties, but we'll leave that for another day.) Clinton has more support among women and somewhat older people. Both those groups are likely to be underrepresented in any pool of internet respondents. As a result the responses of those with these characteristics who ARE present in the sample are likely to be weighted up quite a bit to reach population proportions in the weighted sample. If the relatively few older women who are in the sample are ALSO atypical in other ways that both make them volunteer for internet surveys AND be less disposed to support Clinton than are non-internet volunteering older women, then weighting these respondents up won't properly capture Clinton's support and will lead to a systematic underestimate of her support.

That could do it, but it sounds pretty tortured to me.

I'd check the software one more time.

And based on the large outliers the Clinton results produce, I'd hold off on the Reuters headline until I saw some confirmation from other polls.

Recent polls in IA and NH plus US

2007-11-15T10:28:00.000-06:00

I don't have time to write about these, so let's let six pictures do the talking. The blue line is the standard trend estimate, and the red the more sensitive trend.

I've added the names of the last two polls (last three if there is a tie on the date of last interview.) This helps put the latest results in some perspective, both poll vs poll and poll vs trend.

These are relatively high resolution plots so you may need to click once or twice for full resolution.

State v. Nation as Mitt leads (IA&NH) , Fred falters

2007-11-13T07:00:00.001-06:00

The Republican race has had something for everyone this year. Front runners have faltered (McCain, remember, was the widely declared "front runner" until early spring, though he trailed Giuliani in the vast majority of national polls.) The puzzle of Giuliani's national lead continues to confound explanation in a party of social conservatives. Despite the most visible pre-primary season in history, the leader in IA and NH, Romney, remains less known nationally. The reluctant actor waited in the wings and perhaps missed his scene, certainly entering after his peak. And just to round things out there is an Arkansas governor from Hope who is beginning to be taken seriously and a Texas congressman whose internet strength is disproportionate to his polls.

(A technical note: the blue line in the figures is our standard trend estimator. The red line is more sensitive to recent change, but also less reliable because it can respond to "noise" in the data rather than real changes in trend. It is great for speculation to consider the red line, but safer for prediction to rely on the blue line, which has a better track record over the long run.)

The single most important feature for the Republican race is the discrepancy between state polls in IA and NH and the national polls. Nationally, Giuliani continues to hold a significant lead of nearly 2:1 over his nearest rivals. But in the initial states, Romney has established his own 2:1 lead in IA and a smaller 8 point lead in NH, while Giuliani struggles in IA and has remained basically flat in NH. This sets up Romney to run a classic momentum campaign based on two early successes to carry him to national prominence and through the second round of pre-February 5th primaries and caucuses.

The Iowa data in the top chart shows Romney's early success there, reaching 20% by April 1, when his national support was a modest 8%. Romney has now built his support to nearly 29% in Iowa, a significant lead over his rivals there, though not enough to dominate the race. Meanwhile he is at less than half that (12%) nationally.

Romney demonstrated his organizational strength by winning the Iowa straw poll back in August. But that story also demonstrated a potential problem for him. The straw poll win would have been news to many Republican primary voters outside of Iowa, where Romney still needs to build his visibility and support. But the press corps (and pollsters) strongly discounted his win as "expected", and focused instead on the narrow second place finish of Mike Huckabee, well behind first place. This seems to be the danger for Romney in January. He has carefully built a strong Iowa presence and support and if the election were held today would probably win (you do believe these polls, right?). That would be big news to lots of voters, and should dominate the headlines. But Romney's lead in IA has become conventional wisdom among reporters and a win is likely to be treated as the straw poll win-- news, but not surprising. The surprising second place finisher would be Huckabee, based on current polls, and that would likely be the bigger story from Iowa.

The rise of Mike Huckabee in Iowa is correctly seen as a big polling story. With limited money Huckabee has climbed into second place in the Iowa polls, and currently enjoys the sharpest upward trajectory of any Republican candidate there. While still well behind Romney, a Huckabee defeat of both Giuliani and Thompson would be legitimate "big news", and could propel the former Arkansas governor to the kind of national momentum he must have to compete after Iowa. His current trajectory is getting him noticed more, and his second place standing probably deserves even more attention than it is currently getting. Coupled with Thompson's failures, Huckabee's ascent could be come a major asset.

The other bit of news from Iowa is the failure of the Thompson campaign to launch. For all the high expectations built up in the pre-campaign campaign of Thompson, Iowa voters have failed to respond. The trend has even taken a bit of a turn down in recent weeks. At only about 12% support, Thompson trails Huckabee and Giuliani.

And the picture only gets worse for Thompson in New Hampshire where his trajectory looks like a failed rocket launch, now at less than 5% support.

It is debatable whether Giuliani, as national leader in the polls (see below) can survive losses in Iowa and New Hampshire. It seems even more unlikely that Thompson, who is falling nationally as well, can survive two poor early finishes.

This morning's news is that Thompson will receive the endorsement of the National Right to Life Committee, a potentially important boost to his campaign. The organizational strength of Right to Life organizations could be a significant advantage, and might help Thompson secure the status of "choice of social conservatives". However a consensus candidate of social conservatives has yet to emerge, as demonstrated by the scattered endorsements we saw last week. If Thompson is to secure that standing it will come despite his standing in the polls, rather than because of it. Certainly the fear of other Republicans that Thompson would be the late arrival who swept all before him as not materialized.

And then there is John McCain, who has been all but written off by analysts, including me. Yesterday's news that McCain may actually borrow money to finance his campaign through the early caucuses is further evidence that the analysis is not wrong, at least in an organizational sense. McCain's campaign not only lost most of its staff, it has failed to raise money and is back to the days of the bus. Voters, however, haven't entirely gotten that message. McCain's long decline in the polls halted in the third quarter and has made a small gain nationally. A similar rebound may also have occurred in New Hampshire (but only in the more sensitive red estimator.) Still, at 15% there, McCain would be a distant third place, hardly a strong foundation to relaunch a campaign despite previous success in NH. And in Iowa, never his strong suit, McCain is at a dismal 7%, despite his line that he "drinks a cup of ethanol before breakfast every day".

The contrast between Giuliani as national poll leader, while Romney dominates in the first two states, and a possible late emergence of a relative unknown in Huckabee, sets the stage for a candidate to "emerge" from Iowa and New Hampshire. The Giuliani campaign still banks on a "firewall" in Florida and a great February 5th to maintain his campaign, and some chance that a convincing 2nd place in New Hampshire will keep him strongly in the game. For Thompson, South Carolina looks to be his best bet, though he is currently only tied with Giuliani for first place there, and is declining there as well. (A SC win for Giuliani would be huge, of course.) And Romney has had two good a two mediocre polls in SC recently, leaving it unclear if he is moving up there or not. McCain still needs a miracle. But Huckabee has moved in Iowa strongly and a little bit in New Hampshire. Elsewhere he will live or die based on those two states.

As a spectator sport, the Republican race this year has something for everyone, and is vastly entertaining. I can't wait to see what happens next.

Did Clinton take a hit after the debate?

2007-11-12T10:45:00.000-06:00

Since the October 30 Democratic debate, political conversation has focused on how much Sen. Clinton may have been damaged by her first "rough" debate performance. After uniformly positive evaluations of her handling of previous debates, especially in August and September, Clinton stumbled with answers that her rivals portrayed as waffling and inconsistent.

This weekend two new New Hampshire polls appeared that lend credence to the notion that Clinton's standing has taken a dip, though whether due to the debate or not remains "debatable."

The Boston Globe and University of New Hampshire found Clinton falling to 35% 11/2-7/07 from 41% in the CNN/WMUR/UNH poll of 9/17-24/07. Obama had a small rise, to 21% in the latest from 19% in the earlier poll.

Meanwhile, the Marist College poll taken 11/2-6/07 put Clinton at 38%, compared to 43% in Marist's previous reading 10/4-9/07. Obama gained five points over those two polls, to 26% from 21%.

Finally, a Rasmussen "robopoll" taken on 11/5/07 has Clinton at 34%, down from 40% in their 9/16/07 poll. Rasmussen also has Obama rising to 24% from 17%.

So on the basis of these three polls, and "apples-to-apples" comparisons of polls done by the same polling organization, there is pretty good reason to think Clinton has dropped a few points in New Hampshire, and that Obama has gained a few. The Clinton average change in the three polls is -5.7 points, and Obama's gain is 4.7 points.

If we turn to the trend estimates, based on all the polling rather than just the last three, we see a different but still interesting picture. The vertical black line in the charts marks the date of the debate.

With just three new polls, the standard trend estimator will respond to the new polls, but won't "bend". The blue trend is deliberately conservative, and wants more evidence that the trend has actually turned down before it will change direction. However, the level of the estimated trend does respond to new data. For example, on Friday with Rasmussen included but not the Globe/UNH or Marist results, Clinton's trend estimate was 40.2%. With the new polls it is down to 39.3%. Likewise Obama was at 21.0% on Friday and is now at 21.7%.

But the standard estimator may be too conservative. So we have a second estimator that picks up changes more quickly, though it can fall victim to noisy data and "see" a change when there isn't really one. So with some risk, it is fun to compare the more sensitive "red" estimator with the standard "blue" one.

Red reveals two interesting details not visible in the blue trends. Clinton appears to have flattened out, at least, starting in late September. Obama had a noticeable dip in August-September, and since that time has been trending up for over a month.

If we put a VERY sensitive estimate on the Clinton polls, we would even see a parabola shaped trend, one that rose sharply in September and early October, then fell equally sharply in November. This amounts to just connecting the polls, and ignoring all the noise and house effects present, something I am loath to do.

But even being a bit cautious there is evidence that Clinton's good rise (about 5 points) in the third quarter has stopped and perhaps dipped a tiny bit in the fourth quarter so far. If more polls come in where the last three have, then even the blue standard estimator will flatten and turn down. Meanwhile the sensitive red estimator may be prone to chase the latest polls a bit too much. But it's suggested path for the dynamics of the race is neither as foolish as chasing each poll, nor as slow to notice change in trends as the standard estimator. "Red" may have something here.

The story for Obama is also more hopeful in the red estimator. The few low polls in late August through late September suggested a slump down to the mid-to-upper-teens. The sensitive estimator catches this dip, but sees a steady rise since, to nearly 24% now.

From the point of view of even the sensitive estimator, the changes affecting both Clinton and Obama pre-date the October 30 debate, though the estimate is influenced by the three post debate polls. The story "red" tells is that Clinton had a very good third quarter-- good news about her campaign, it's strength, and her good debate performances-- helped raise her New Hampshire standing by five points. Perhaps the same news, or reviews of his failure to make progress, helped sink Obama's support about 4 points during the same third quarter. But since October 1, these patters have changed, with Clinton seeing no further gains and Obama returning to the mid-20s.

Before reaching too strong a conclusion, let's check some other data, and see if we can find any evidence for similar changes in trend in other data.

In Iowa, where they watch the candidates at least as closely as in New Hampshire, there is no evidence of a flattening of the Clinton trend. If anything, the red trend estimate is slightly higher than is the standard blue estimate (30.5% vs 29.9%). Clinton has climbed steadily since the start of the third quarter and both red and blue estimates put that trend at almost the same rate.

Obama has also seen gains since the start of the third quarter in Iowa, though the red estimate thinks the gains started from a bit lower level. At present the two trends agee quite closely, 24.8% for blue and 24.5% for red.

(The Edwards and Richardson campaigns have both suffered losses since the start of the third quarter in Iowa, though slight or no losses in New Hampshire over the same period.)

If we look at the national picture, we see a substantial Clinton rise in the third quarter (with a plateau midway, followed by a surge) and a mild decline for Obama, reversed only at the start of the fourth quarter. The sensitive estimator matches the standard trend pretty well, certainly leading to the same conclusions.

As my partner at Pollster.com, Mark Blumenthal, pointed out in an important post last week, New Hampshire and Iowa are not typical of the nation as a whole and are being exposed to vastly greater advertising and campaign appeals. Citizens there are also paying more attention to the race, given both advertising and their prized "first in the nation" status. So if we are going to detect changes in the fortunes of candidates, these two states are the best places to look. And when we do, we see some evidence of change but only in New Hampshire.

As always, more data can lead to a reconsideration, but at this point I think the evidence favors a view that Clinton's problems in New Hampshire may be more specific to that state rather than reflecting a more widespread change in her campaign's fortunes. New polling for Iowa could change that. At the moment we only have one post-debate poll from Iowa so the trends there are almost entirely driven by earlier data. But the 5 post-debate national polls are included in the national estimate with no effect, so that is some evidence that any impact is still to develop.

Sen. Clinton may have given her opponents an opening in the last debate. It remains up to them to exploit that opening to their advantage, and up to the Clinton campaign to deflect such critiques. At the moment the New Hampshire polls suggest some changes in that state. But not yet elsewhere.

Bush Approval: Trend 32.6%

2007-11-08T15:45:00.001-06:00

New polls: Marist/WNBC (10/29-11/1/07) Approve 34%, disapprove 58%; Gallup (11/2-4/07) approve 31%, disapprove 64%; NBC/WSJ (11/1-5/07)approve 31%, disapprove 63%.

With these new polls, the trend estimate moves to 32.6%. The rise in approval since June has now clearly leveled off and is perhaps starting slightly down. Both the conservative blue trend estimate and the more sensitive red line agree on both the level of the trend and its recent changes.

After a flurry out both high and low outliers in recent weeks, the current crop of polls are generally falling within the 95% confidence interval around the trend. The last 6 polls are all quite close to the trend estimate.

U. S. Monthly Deaths in Iraq

2007-11-06T21:20:00.000-06:00

I wrote yesterday about changes in opinion of the Iraq war. In the process I said that the death toll is rarely presented in systematic form by news coverage. Totals, or totals for the month are frequently mentioned, but seldom in the context of the entire war. Likewise, yesterday's news story was that 2007 has now become the costliest year, with more U.S. deaths than in any previous year of the war. So let's take a little of our own medicine and put the monthly totals in context.

The last two months have indeed seen substantially fewer US losses than any previous months in 2007. April, May and June produced exceptionally high casualty counts with only 5 previous months reaching the same level of over 100 deaths. In contrast, October casualties fell to under 40 deaths, a low matched in only 7 previous months.

But looking at the last 12 months, it is really only September and October that have been well below recent casualty rates. There is some visual impression of a sharp decline since May, but that is deceptive since May was the third highest death toll of the entire war.

The bottom line: deaths have declined recently and especially in the last two months. But that decline is from very high levels compared to the rates in the entire war. Even the "lower" September rate is actually in the middle of rates for the war as a whole.

There is a lot of month-to-month variation in the death rate. That makes it a bit hard to see the systematic trend in the midst of the variability. The plot below shows the trend in deaths, abstracting out the noise of month to month variation.

We see three clear phases. After the invasion, deaths fell to about 45 per month and stayed there until early 2004. Then the rate increased over a period of about 4-6 months to a new and more intense rate of about 67 deaths per month. This became the status quo death rate for the second half of 2004, all of 2005 and the first half or so of 2006. Late 2006 saw an increase which peaked in the first half of 2007, with an apparent fall since. The current trend estimate stands at the low 60s, not much different from the 2004-2006 level. Unless the very low casualty rate of October becomes the new norm, we are back to about where we have been since mid-2004. It seems more likely that October will prove to have been unusually low and November and subsequent rates will return to around 60 deaths per month. (This is a statistical, not a military judgment. At the moment, October looks out of line with the rest of the data, so a good statistical guess is that subsequent months will be closer to the trend. Of course if conditions or tactics have changed in ways that actually reduce the risk, then casualties can remain near October's low level.)

Yesterday my point was that opinion on the war has turned somewhat more positive, but that while changes in the "objective" facts of the war might contribute some to that shift, the stronger driver of opinion was political debate from Washington. Looked at over the entire course of the war, there have been large month-to-month changes in the casualty rate. Opinion on the war shows no evidence of such high variability, as it should if opinion of the war simply and directly mirrored contemporaneous casualty rates. Rather we've seen a pretty consistent long run decline in positive views of the war, but in 2007 a flattening of the trend followed by a recent rise in positive views of the war. Those changes came despite the substantial rise in casualties in 2007. There is little evidence of a direct link of opinion to current casualty rates.

Ten Months of Opinion Change on War and More

2007-11-06T07:14:00.000-06:00

While I've been doing back room work the last two months, some interesting changes have taken place in opinion about the war, the president, congress and the country. It is too early, and the changes too modest, to declare this a "turning point" in opinion, but the changes are consistent enough to take a hard look and ponder if there is still potential for significant shifts over the next 52 weeks until Election Day 2008.

The single most striking shift is the change in opinion about how the war in Iraq is going. After four and a half years of steady downward trends, there has been a reversal of direction since July.

CBS, CNN and Pew have asked "How well is the military effort in Iraq going?" since the war started (with some minor variation in wording. See the details here.) The virtue of this question is its consistent use over time and its summary evaluation of the war.

President Bush's change of policy in Iraq in January, coupling a change of command with a surge of troop levels did not produce immediately positive responses from the public. Likewise the rise in U.S. casualties in the spring following the change in deployment strategy certainly might have been expected to further erode support for the war and for Bush.

But in retrospect the actions have been accompanied by two phases of changing opinion on "how the war is going". From January through June, the long running collapse in positive evaluation of the war (especially in the second half of 2006) halted. The flattening now appears to have clearly coincided with the change in command and troop levels.

This flattening didn't signal rising opinion on the war-- but after dropping over 13 percentage points in six months, simply arresting the collapse was a major plus for the administration. And this is a particularly striking thing given that the spring of 2007 was a focal point for critiques of the war in Congress, with Democratic leadership repeatedly pushing votes that would have required changes in Iraq policy of various kinds. And this flattening came at the same time that casualties rose.

The second phase of opinion change started in early July, when positive evaluations of the war took their first upturn since late 2003 (around the time of the capture of Saddam Husein). The trend estimate has turned up some 8 percentage points since July 1, still not back to early 2006 levels, but remarkable this late in an unpopular war and with a weak leader and determined opposition.

It is also worth noting that this is not just a shift due to "undecided" citizens shifting. The percentage saying the war is going badly also stabilized through the spring and has turned down to about 58%, from a high of 69% at the end of 2006.

Through the spring, conservatives and Republican supporters of President Bush argued for "giving the surge a chance". This rhetoric shifted in the summer to claims that "the surge has worked". Meanwhile Democrats and liberals pushed for a timetable for withdrawal through the spring and early summer. Very few citizens have a clear idea of any quantitative measures of how the war is "actually" going. Even trends in American deaths are rarely comprehensively presented in news reports (though sometimes mentioned in passing as "factoids".) And even among supporters of the war claims of "success of the surge" were rarely supported by direct evidence. (An exception to the lack of evidence was a widely debated op-ed piece in the New York Times by Kenneth Pollack and Michael O'Hanlon of the Brookings Institution on July 30. O'Hanlon has produced the comprehensive "Iraq Index" at Brookings, an invaluable compilation of measurable trends in the war. Ironically, the op-ed piece was based on "anecdotes" from a visit to Iraq rather than quantitative measures.)

But citizens don't shift their opinion based on quantified measures of progress, nor even New York Times or Weekly Standard articles. For most citizens, opinions are driven more by the messages they hear from partisan leaders, with some sifting for credibility of the claims and filtering by predispositions. And, it must be added, by some effects of "reality", whatever that is.

The current upturn in positive views of the war then reflect perhaps some bits of success on the ground. US deaths are down. Iraqi civilian deaths are down. But if it were casualties alone that drove opinion, positive views should have fallen sharply in the spring as the death tolls of both US troops and Iraqis increased. Instead opinion became flat. So it would be too simple minded to imagine a direct causal effect of casualties on views of the war.

So what to make of the upturn in positive views of how the war is going? Republicans (including the president) have made real progress in swaying opinion to their side, while 10 months of Democratic efforts have failed to persuade citizens that the war continues to be a disaster. The war of partisan persuasion has tilted towards the Republicans and away from the Democrats, at least in this particular aspect.

Let's be clear: the trend estimate is that only 38% think the war is going well, while 58% say it is not going well. The balance remains on the pessimistic side and by a 20 point margin. What I am talking about is the change in trend and the shift of marginal opinion. But that is a telling indicator. On election day a year ago today, the partisan war for public opinion seemed to have decisively shifted to the Democratic view. The notion that there was nothing the White House could do to reverse their public losses of support was widespread. But the last 10 months show that indeed there was something that could change and this change is important.

Much could still change before election day 2008, twelve months from today. Either positively or negatively for the war, and even more so for the candidates currently seeking to inherit the war from President Bush. But the past 10 months of opinion on how the war is going should serve as a reminder that the politics of war, like politics in general, is always open to change.

Postscript: Opinion on the war, and on politics and politicians is, of course, complex. I'm confident many will object to what I've left out above (as well as what I've included!). But take a look at the trends presented below. Across other measures of war opinion, a stabilization (not necessarily much of a rise) has taken place. Even evaluation of Bush's handling of the war rose for a while (though is currently headed back down.)

Overall evaluation of Bush has turned up since July.

Right track or wrong direction has flattened recently.

Even opinion of Congress has stabilized (though see the links to party performance from the thumbnails in the column to the right.)

There are some changes taking place in trends that have been taken for granted. It is time to reexamine our easy and comfortable assumptions.

Iraq Opinion: How Well is the War Going?

2007-11-06T06:53:00.001-06:00

Updated 11/6/07

This post is part of a series monitoring opinion on the war in Iraq. It is updated in place as new data are added. The index in the right column provides a direct link from the thumbnail to this page.

There are three pollsters who have asked this question over the course of the war in similar format but with minor variation in question wording:

CBS: "How would you say things are going for the U.S. in its efforts to bring stability and order to Iraq? Would you say things are going very well, somewhat well, somewhat badly, or very badly?"

CNN: "In general, how would you say things are going for the U.S. in Iraq: very well, moderately well, moderately badly or very badly?"

Pew: "How well is the U.S. military effort in Iraq going? Very well, fairly well, not too well or not at all well?"

Click here for all Iraq Opinion Posts

Bush Approval: Trend at 32.6%

2007-10-30T11:30:00.001-05:00

There have been several polls since my last update, and they have varied much more than usual. But the bottom line, with all the data, is a small decline in the trend estimate, and that well within the uncertainty of the trend.

The newly added polls are:

Fox: 10/23-24/07, Approve 35%, Disapprove 55%
LATimes/Bloomberg: 10/19-22/07, Approve 35%, Disapprove 60%
ARG: 10/18-21/07, Approve 25%, Disapprove 67%
CBS News: 10/12-16/07, Approve: 30, Disapprove 61%
CNN/ORC: 10/12-14/07, Approve 36%, Disapprove 61%
Zogby/Reuters: 10/10-14/07, Approve 25%, Disapprove 75%

That is quite a range from two at 25% to a top of 36%. Zogby and ARG are well below the trend estimate, while the two at 35% and one at 36% are only a bit above trend. (An older NPR poll at 38% is well above expectations, but as I explained in an earlier post this has something to do with the likely voter sample NPR favors, compared to the adult samples of most approval polls.)

If we look at the residuals, NPR looks like a high outlier and Harris a low one. But Zogby/Reuters and ARG are even more extreme low outliers. Zogby uses the same 4 point job rating measure that Harris uses, so that partially explains their low value (a persistent question wording effect) but as the plot makes clear, the Zogby reading is even lower than we'd expect from that. The ARG reading is also extremely low, at the same 25% approval rating, and the points and labels are overwritten by each other.

The curiosity is why two (three with Harris) outliers? Did opinion change and these caught it early? Apparently not, judging by other recent polls that are at or even a bit higher than the trend estimate of 32.6%.

In general I think it is a bad idea to seek a substantive explanation for outliers. The most reasonable story is simply "random variation" and we should leave it at that. There ARE some systematic elements, such as the question wording variation or sampling issues I pointed out, but I'm not inclined to say more. To do so becomes a post hoc search for what are most likely statistical phantoms. (Though when history keeps repeating itself we might look into house effects for a systematic effect, possibly due to question wording, sampling, or treatment of don't know responses.)

The bottom line is approval may have shifted down a tad, from 33.0% to 32.6%. BUT, one should consider the gray region around the trend line below. This gives you a good idea of the uncertainty in the trend estimate itself, after squeezing out as much random variation in the polls as possible (at least until next week! Stay tuned for that!). Clearly the change of .4 percentage points is not a clear indication of movement in approval. In fact, given the wide range of current polling, we have an unusually wide uncertainty about where approval actually is at the moment, with 32.6% being our best estimate, but an uncertain one.

We are in a period of relative stability in President Bush's approval rating but considerable polling variation. Waiting for the next "thing" to happen.

Bush Approval: Trend at 33.0%

2007-10-16T12:09:00.000-05:00

President Bush's approval trend has been relatively flat in recent weeks. For the last month the estimated approval has held between 32% and 33% and currently stands at 33.0%, which includes Gallup's 32% approval from polling done 10/12-14/07.

There was a fairly sharp upturn that started in July, which has tapered off but not yet disappeared. Poll to poll fluctuations have pulled the trend estimate around a bit in the 32%-33% range but without clear evidence of significant change.

The last six polls also demonstrate some persistent differences between polls. The NPR Poll, conducted by Public Opinion Strategies (R) and Greenberg Quinlan Rosner (D) has a persistent positive house effect, making NPR one of the higher measures of Bush approval. In the opposite direction, Harris has a persistently negative reading of approval. In the figure below, we see that both are outliers in the current approval trend.

But this is a good example of when being an "outlier" doesn't necessarily mean anything is wrong with the polling. NPR samples "likely voters" and we know from other analysis that samples of likely voters are generally more approving of President Bush. Likewise, Harris (along with Zogby) uses a four point approval scale, "Excellent", "Good", "Fair" and "Poor" while everyone else uses "Approve" or "Disapprove". Harris and Zogby (and me) arbitrarily map "excellent" and "good" into "approve", and "fair" and "poor" into "disapprove". But is that the right way to make different answers comparable? Do the first two of these choices mean exactly what "approve" means? No. This is just the best categorization possible for mapping these four response options into two. And one consequence is that Harris and Zogby tend to come in below the estimated trend.

The choices these polling organizations make to sample likely voters rather than adults, or to use a four choice approval measure, are not in any sense "wrong" decisions. But they do mean that the results these pollsters get will be somewhat different from what others, using different samples and different questions, will find. But the choices DO make these polls more likely to be "far" away from the trend which is based on all pollsters.

So why call them outliers? Simply because they are, in fact, well away from the trend and the distribution of everyone else. In some cases that might indicate a "bad" poll, a sample that for some reason is far from what we would normally expect, or perhaps a biased question wording or sequence of questions. But being an outlier can also result from making choices of sample or question or other procedures which are different from most. Different yes, not necessarily bad.

But different is surprising, nonetheless. My goal here is to put everyone's polling in perspective, and these two do deserve to be labeled "outliers" because they are in fact different. A normal reader of polls without benefit of the perspective we provide here, would reasonably ask how can NPR get Bush at 38% and Harris have him at 27% (and the trend estimator be at 33%?) That surprise factor is also part of what being an outlier means. Without some further explanation (sample, question wording) the results appear puzzling. An outlier may be understandable or explicable, as both of these appear to be. Or in may not. Either way outlier detection draws attention and asks for an explanation or a discounting of the finding.

Bush Approval: 4 New Polls; Trend at 32.5%

2007-09-21T14:08:00.000-05:00

We have four new presidential approval polls out this week that suggest a slowing of the recent upturn in President Bush's job ratings. The new data are a Pew Research poll done 9/12-16/07 with approval at 31%, disapproval at 59%, A CBS News poll done 9/14-16/07 with approval at 29%, disapproval at 64%, Zogby/Reuters conducted 9/13-16/07 found approval at 29% and disapproval at 71% and the Gallup/USAToday poll taken 9/14-16/07 which put approval at 36% and disapproval at 62%.

The result of these three new readings at the low end and one at the high end of recent polling moves the trend estimate to 32.5%. The blue trend line is still showing an upturn but the rate of increase now looks less steep. In comparison, the more sensitive red line estimator thinks approval has leveled off and is no longer climbing. However, red and blue estimates are in close agreement so we need more data to be confident about the leveling off. The situation is further complicated by the fact that three of the recent polls are from organizations whose polls typically have a negative house effect, meaning they tend to run a bit below the trend line. Gallup is the exception which is usually a bit above trend. The accident that 3 of the 4 new polls happen to come from houses with negative effects may mean we are overstating the amount of flattening in approval. This should become clear with a few more polls next week.

Also, don't overstate the size of the house effects here. None are very large. Zogby averages an effect of -2.4 (and uses an unusual 4 point question format that may contribute to this effect), while Pew and CBS are at -1.4 and -0.8 points respectively. Gallup is at +1.3. All those values are relatively small compared to the +/- 5 point spread we see around the trend estimator. (We'll be talking about house effects in more detail soon.)

At the moment we've seen quite a range of recent polls, with Zogby at CBS down at 29% while Gallup at 36% and Fox at 37% represent the high end. Still, this range of variation is about what we've come to expect and none of the recent polls is outside the 95% confidence intervals. So while politically there is a big difference between 29% and 37%, both are quite compatible with a "true" approval rating of 32.5%, given the normal randomness in survey results.

The last bit here is to look at the red dots in the figure below to see how the estimated trend has been changing over the past 20 polls. With only one exception, it has held between 32% and 33%. That is another reason to think that approval is flattening out now.

Short term trend, long term trend and sensitivity

2007-09-20T11:29:00.001-05:00

The trend estimator we use for the presidential nomination races (and for other trends as well) is designed to capture "real" changes in support while ignoring the inevitable random noise that is the result of random sampling of opinion. The "standard" trend estimator (which I fondly call "old Blue") is deliberately conservative in the sense that it takes a good bit of evidence before the estimator will believe a change in the polls is real. In the case of presidential approval, this estimator has an excellent track record of catching changes in approval while not thinking it sees a change when there isn't one there. (For a fuller explanation of the estimators see this post.)

But I've also used a more sensitive estimator ("ready Red") that catches changes more quickly but is more often fooled by random noise into believing a change has occurred that subsequent polls show didn't happen. I've posted the comparison, and a more extensive comparison of a range of sensitivity estimates here and updated that with each new set of polls.

What has become clear is that "old blue" has been too conservative this summer, and has missed an important change in the trend of Rudy Giuliani's support. I want to use this to illustrate what the different estimators reveal and what they smooth out, and to explain a move to a somewhat more sensitive estimator.

To illustrate this issue I've computed three different trend estimates, plotted in the figure above. The blue line is the most conservative and least sensitive. The red line is more sensitive, while the black line is the most sensitive of all. (Red is twice as sensitive as Blue, and Black is twice as sensitive as Red.)

The current estimates of Giuliani's support are Blue: 27.3%, Red: 28.3% and Black: 27.8%. The estimates are all within one point of each other. In the figure it is clear that the three lines usually overlap each other pretty closely, and so much so in Romney's case that they are all but indistinguishable.

The caveat is that the lines can and do disagree with each other when support makes a change of direction. Giuliani's polling has done that at least twice since January. After rising in late 2006 and early 2007, his support peaked late in the first quarter and fell for a while. But the Red and Black estimators agree that the fall was rather sharp and halted by the end of the second quarter after which there has been a mild but noticeable upturn. The more conservative Blue estimator caught the decline but has not shown the subsequent upturn. As a result, it has continued to see a long term decline, rather than a recent stabilization and some improvement. Even though the three estimators currently are within a percentage point of each other in estimating his support, the short term dynamics of the Giuliani campaign are not well represented by the Blue line.

The blue estimator was set to be especially conservative in the face of few polls unevenly spaced, as we had in 2006 and the first half of 2007 . As polling has become more frequent over the summer, that extra degree of conservatism became a liability rather than an asset. With enough new polls, a more sensitive estimator can be used. And both Red and Black pick up clear changes in Giuliani's trend.

You can see more modest differences for other candidates. Red and Black think McCain had a modest bump in late 2006 and early 20007, but blue doesn't. And if you focus on the very latest data red sees a flattened trend, black thinks there is a bit of an upturn, and blue still has McCain in decline. This kind of disagreement at the end of the series and based on few polls is common. There isn't enough data to be sure of the trend into the near future. Still, the McCain estimates are currently Blue: 13.2%, Red 14.2% and Black 15.4%, a range of 2.2 points.

The Romney trend has been so stable that all three estimators agree almost exactly, except at the very end. The most sensitive Black estimator sees a blip up at about the time of the Iowa straw poll, which is believable, but Red and Blue don't see it. And Black sees a bit of decline since then, which Red barely hints at and Blue doesn't see at all. The bottom line is Blue: 9.7%, Red: 9.0% and Black: 8.2%, a 1.5 point range.

The Thompson trends show a bit more short term variability, in part due to fewer polls, but perhaps also reflecting some real short term changes. Both Black and Red see two short term dips in Thompson support (of a couple of points) that Blue smooths out. And Blue doesn't detect the post-announcement blip that Red and Black pick up. And the bottom line for Thompson is Blue: 20.1%, Red: 20.9% and Black 21.2%, a range of 1.8 points in the estimates.

The Democratic trends have been a bit more stable and so the discrepancies between estimators are generally less. Red and Black see a little more dynamic in Clinton's trends early in 2007 and a small blip or two for Obama. And Red and Black think Edwards was up a couple of points in April while Blue doesn't see it. And Richardson hasn't had enough national movement for there to be any discrepancies.

As for the current estimates, Clinton is at Blue: 39.7%, Red: 40.1%, Black: 39.3%, a range of 0.8 points. Obama is at Blue:21.8%, Red: 21.5% and Black 22.0%, a 0.5 point range, and Edwards a bit more variable at Blue: 12.2, Red: 13.7 and Black: 14.3%, a 2.1 point range (and an example of the more sensitive estimators thinking there is a bit of an upturn in Edwards' support.

The estimates of current support vary a bit for each candidate, but far less than the variation among recent polls. That is the point of using the estimator, after all--- to remove random variation and get a better estimate of where support stands. The estimates across all the candidates are within 2.2 points of each other at the moment, which I think is pretty good agreement regardless of how sensitive an estimator we use.

But the visual pattern of the trend, in Giuliani's case especially, is not well captured by the most conservative "blue" estimator. Now that there is more frequent polling, and therefore less chance that a single poll or two will distort the trend, I think we are safe using a somewhat more sensitive estimator for the trends. We'll be a bit riskier in order to also catch shorter term changes better.

As a result of this analysis, we'll be updating our trend plots for the national data to reflect a somewhat more sensitive estimator. However, we won't go too far on that, since we do believe there is a lot of noise in the polls and we want our trend estimates to resist chasing each new poll.

The state trends, however, will continue to use the more conservative estimator until enough polling becomes available to allow us to increase the sensitivity. Some states are developing a good bit of data, but even New Hampshire has only 42 Republican polls, while nationally we have 130.

And as we've been doing all along, we'll assess sensitivity and write posts on differences between the estimators that may suggest short term changes of interest. Even if we can be fooled by a sensitive estimator, it is always fun to speculate about what it might mean, if there really is a change.

Bush Approval: 4 New Polls, Trend at 33.3%

2007-09-13T15:22:00.000-05:00

A very busy week for pollsters with four new polls today. CNN/ORC was conducted 9/7-9/07, finding 36% approval of President Bush's job performance while 61% disapprove. Harris, done 9/7-10/07 gets 31% approval, 67% disapproval, while AP/Ipsos from 9/10-12/07 puts approval at 33% and disapproval at 64%. The latest is the Fox poll taken 9/11-12/07 which finds approval at 37% and disapproval at 58%.

The addition of these four polls raises the trend estimate of approval to 33.3%. This is the first time since May 9 that approval has reached the one-third point.

The new polls are nicely balanced around the current estimate: Fox and CNN are about as far above the trend as Harris and AP are below it. You can get a sense of the house effects of each of these pollsters from the chart below.

The balance of polls above and below trend is also evident in the residuals plot below. No recent poll is close to being an outlier. All of the last 10 polls lie within the normal range of sampling variation.

With President Bush scheduled to address the nation tonight about Iraq policy it will be very interesting to see if the current upturn in approval is sustained. The testimony by General Petraeus and the President's speech could (I stress could) convince some citizens to give the Iraq policy more time, or could push in the opposite direction among those who were hopeful of a faster reduction in forces. (Note I'm talking about those still on the fence enough about Iraq to be affected either way. Obviously a lot of people have long since made up their minds in either direction and are unlikely to be affected by the testimony or the speech.)