Ideology and Republican Losses

Republican losses fell along most of the ideological spectrum, but were a bit more common among moderates than among the more conservative members of the party. The graph above shows the distribution of conservatism in the House of Representatives by party, using the National Journal ratings. (I use National Journal because it is convenient to grab the data, includes more votes than the typical interest group rating and correlates highly with other (more "political science-y") measures such as Nominate scores. Nominate scores are not yet available for the 109th Congress and I don't have time to compute them myself right now. Readers interested in how Nominate scores are computed should see Keith Poole's excellent site here.)

Republican incumbents who were defeated for reelection had an average conservatism score of 66.5, while those who won averaged 74.3, a statistically significant difference. (The average for all Republicans in 2005 was 73.2, and the median was 74.4.)

This does not mean that Republicans were necessarily more vulnerable BECAUSE they were more moderate. What is left out here is the district. Moderate Republicans are more likely to come from less conservative districts, so this result may well simply reflect the more competitive nature of districts that produce more moderate Republican members. I'll take a look at that later, but for now the simple point is that the election defeat did fall more heavily among representatives with somewhat more moderate voting records.

The one Democratic defeat plotted here is Rep. Cynthia McKinney, GA-04 who lost in the primary. The plot above also includes Republican Rep. Joe Schwartz, MI-07, who also lost in a primary.

Ideological shifts in House Committee Chairs

(I'm still digging out from the ton of stuff undone over the past three weeks, so the Table of Contents is still waiting for an update, and reorganization. But I don't want to delay further some of the new posts I had prepared last week but couldn't post while traveling. So here is a start.)

The shift in party control in Congress brings with it new committee chairs. Their outlooks give us one measure of how different the new 110th Congress may be from the 109th. Of course committee chairs are only part of what makes Congress work. We've heard and read that the Democratic leadership does not plan to give free reign to the instincts of the most liberal members. Nonetheless, the shift from generally conservative Republican chairs to generally liberal Democratic chairs is clear. The Science committee is the one substantial exception.

The graph above plots the Republican chair's vs the Democratic chair's National Journal 2005 Conservatism score, where 100 is the most conservative and 0 is the most liberal. I've used the expected chairs as reported by Congress Daily. The Intelligence committee chair appears to be the most uncertain at this point, with Speaker to be Pelosi said to be opposed to current ranking minority member Rep. Jane Harman, (D-CA). I've used Harman's National Journal score none-the-less in this plot.

It is hard to identify the cluster of points to the middle of the graph, so the dotplot below shows the shift to the left for each committee.


























One other way to view this is to compare the distributions of the two party groups, as in the figure below. Here the polarization of the parties, and the modest overlap in the middle, is apparent. This distribution is not unrepresentative of the House as a whole. While the chairs are not an exact match, they are not wildly different from the House as a whole. The Republican chair median conservatism score is 73.3, while that for all Republicans in the House is 74.4. The median Democratic chair's conservatism score is 18.8 while the median for all Democrats is 23.25.

Bush Approval: 6 Post-Election polls

I'm finally back from nearly three weeks on the road. I'm one flight from home returning from Germany where my network connection seemed cursed at hotel after hotel after hotel. I'm very much looking forward to being home and getting to the backlog of post-election posts. Here is a downpayment.

President Bush has suffered a significant drop in approval since the mid-term election. Six post election polls bring the trend estimator down to 35.0%. And that despite one rather high reading at 41%. The huge question now is what happens to presidential approval over the next couple of months leading to the first meetings of the new congress. As it stands, approval is only 1.2 percentage points above the all time low of 33.8% on May 15, 2006. A new downward trend would threaten that record, adding a strongly negative public judgment of presidential performance to the rebuff of Republicans at the poll. A short term downward shock due to the election that quickly stabilizes would not be a strong endorsement but would certainly be better for the White House facing the new congress. How such a stabilization, or even a new upward trend, can be engineered is a problem for the President's political advisers.

Time to run for that last flight to Madison. More after a good night's sleep.

National Forces: In Flux at End

My estimate of net national forces is moving again, but is it signal or noise? We'll know after tonight. With the last polls, through 11/6, the Senate estimate has turned back up. I HASTEN to add that this estimator is sensitive to the last points of data and can take some time to make up its mind about the trend. We have no more time, so here is what we've got. Perhaps the Republican surge ended over the weekend, and perhaps the upturn here is just a fluke of the last polls that came in. I HAVE made this a pretty conservative estimator, not TOO sensitive to short term noise, but with no more data after today it will be hard to know about the last couple of days of th campaign. Perhaps I'll be able to do more on that after the election, when only academics like me care about such things.

The House does not show the same upturn, but does show a change in the slope of the decline, perhaps reflecting a similar upturn in pro-Dem forces, but with fewer House polls we can't detect it.

The Governors races (bottom figure) don't seem to show any change in national forces.

Make of this what you will. And have fun with election day and election night.

Bush Approval 2nd Lowest at Midterm

The last Gallup pre-election poll puts approval of President Bush at 38% (and disapproval at 56%.) As the figure above shows, this is the second lowest approval at midterm of any president since World War II. In previous years, such low approval would be expected to lead to major losses in congress. Whether due to redistricting or something else, few forecasts anticipate losses of the magnitude such an approval rating alone would suggest. Current estimates range from the 20s to the high 30s, with a few above and (even fewer) below that range.

Back in May I wrote about approval and midterms and estimated a forecast of approval in the last pre-election poll based on approval in May. The plot below is from that post, with the actual approval rating added. Not bad for predicting the future more than 5 months out.

























Click here to go to Table of Contents

Bush Approval: Three polls, trend at 37.8%

The last three national pre-election polls put the approval trend estimate at 37.8%. The polls by Fox, CNN and Gallup find approval at 38%, 35% and 35% respectively. All three polls completed interviewing on 11/5.

There isn't anything new to say. This approval level is terrible in historical context for a midterm election. What effect? We'll see tomorrow.

From Poll Margin to Wins: Polls as Predictors

The usual way to look at poll accuracy is to subtract the poll result from the vote result. But an alternative is to look at how the probability that a candidate wins depends on the margin they have in the pre-election polls. Since American elections are "winner-take-all" within districts, this is a good way of looking at the practical power of polls to predict winners.

After all-- a statistician would say a poll was better that predicted 51% for the loser who actually got 49% than a poll that predicted 51% for the winner who got 55%. That's right from one point of view, but not from the perspective of predicting winners right. Here I take a look at the latter view of what is important.

The data are from all statewide polls for Senate, Governor or President from 2000 and 2002.

The figure above plots results by poll margin. The x-axis shows the Dem minus Rep margin in the polls. The y-axis plots the percent of races the Dem ACTUALLY won for each margin we saw in the polls. So imagine I take all polls that found a 5 point lead for the Dem. The y-axis plots the proportion of those polls with a 5-point lead in which the Dem actually DID win. I do this separately for each race, Gov, Sen and Pres. The dots show there is a lot of variation, but the pattern of points, and the black trend line through the data show how the predictive accuracy varies over margins from -30 to +30.

One interesting feature is that a margin of zero (a tied poll) produces a 50-50 split in wins with remarkable accuracy. There is nothing I did statistically to force the black trend line to go through the "crosshairs" at the (0, .5) point in the graph, but it comes awfully close. So a tied poll really does predict a coin-flip outcome.

The probability of a win rises or falls rapidly as the polls move away from a margin of zero. By the time we see a 10 point lead in the poll for the Dem, about 90% of the Dems win. When we see a 10 point margin for the Rep, about 90% of Reps win. That symmetry is also not something I forced with the statistics-- it represents the simple and symmetric pattern in the data.

More practically, it means that polls rarely miss the winner with a 10 point lead, but they DO miss it 10% of the time.

A 5 point lead, on the other hand, turns out to be right only about 60-65% of the time. So bet on a candidate with a 5 point lead, but don't give odds. And for 1 or 2 point leads (as in some of our closer races tomorrow) the polls are only barely better than 50% right in picking the winner. That should be a sobering thought to those enthused by a narrow lead in the polls. Quite a few of those "leaders" will lose. Of course, an equal proportion of those trailing in the polls will win.

So read the polls-- they are a lot better than nothing. But don't take that 2 point lead to the bank. That is a failure to appreciate the practical consequences of the margin for error.


Click here to go to Table of Contents

Table of Contents

This is a mini-table of contents to take us through the election. It contains only the recent, election focused, posts. Scroll down a ways for an earlier and more extensive table of contents. I'll rationalize it all after the election.

Bush Approval: 4 polls, trend at 38.1% (11/5)

Dem wave crested; advantage shrinks(11/5)

House 06: National forces estimate (11/3)

Senate 06: National forces estimate (11/3)

Senate 06: Four critical races (11/3)

House 06: Generic ballot (11/3)

Governor 06: State of play (11/3)

Bush Approval: 8 polls, trend at 37.5% (11/3)


(From 11/3, ca 4:00 a.m. Table of Contents):

No time for a full table of contents tonight. I've been busy working to improve Pollster.com with my partner Mark Blumental. We've got a number of nice new features there that make the effort worthwhile, but it has significantly reduced my ability to do the analysis I'd like here. Do take a look at our poll results there, and the "Scoreboards" for Senate, Governor and House races, plus our very complete set of hard to find House polls.

But I hope you'll also read along below for a new analysis of national forces at play in this election, plus reviews of the generic ballot, Senate and governor's races and (at long last!) a presidential approval update.

I'll return to the Table of Contents as soon as I have time, but for now, just scroll down to see the new posts. When you get to the old table of contents, you've gone far enough (though new readers might want to click an item or two for background reading. <;-)

\chf

Bush Approval: 4 polls, trend at 38.1%

President Bush's approval rating has leveled out after a month of sharp decline. Four new polls, by ABC/Washington Post, Pew, Newsweek and Time find approval at 40%, 41%, 35% and 37% respectively. This puts the trend estimate at 38.1, a small upturn from the last estimate at 37.4%.

While stopping the decline, approval well below 40% is historically quite low and is almost certainly taking a toll on Republican candidates.

The trends of the four polls are shown individually below.


























Click here to go to Table of Contents

Dem wave crested; advantage shrinks

Across the board, in Senate, House and Governor's races, the wave boosting the Democrats crested about 10 days ago. Since then the advantage Democrats have built throughout the year has been reduced by from 1.5 to 3.5 percentage points. While forces are still a net positive to the Democrats, these forces are weaker than they were during the week before Halloween. This implies that the most competitive races will now be harder for Democrats to win and easier for Republicans to hold. This implies that the anticipation of a major surge to Democrats now needs to be reconsidered. While race-by-race estimates still show an 18 seat Democratic gain, and 27 seats as tossups (see our scorecard at Pollster.com here), this reduction in national forces makes it less likely the Democrats sweep the large majority of the tossup seats and could result in total gains in the 20s rather than the 30s or even 40s that looked plausible 10 days ago.

This cresting of national forces has taken place across Senate, House AND Governors races and occurred essentially simultaneously around October 25th. The estimators here, plotted as the blue line in the figures, is a measure of national effects that are common across all races. The estimate uses all polls in all races, but estimates the Senate, House and Governors races independently, yet produce similar results for each in terms of timing, though with some variation in magnitude. (For more on the estimation method, see this earlier post.)
























As of last Thursday's data, the downturn was clear for the Senate but no indication of change had appeared for the House. Adding the polling data from Friday, Saturday and Sunday, the downturn is now apparent there as well. I did not do the Governors last week.
























The congressional generic ballot does NOT show any such change (yet!). In my earlier post I cautioned that the generic ballot might not be reflecting a realistic assessment of the Democratic advantage. It may also not be reflecting the last minute dynamics of the campaign this year.
























So what does this mean? The House still looks likely to go Democratic, but probably by a smaller margin that it might have a week ago. For a while, the Senate looked to come down to who won two of VA, TN and MO. Now MT must be added to that, and TN moved to lean Rep, perhaps requiring a Dem sweep of VA, MO and MT. (Momentum in VA remains pro-Dem, while MO is completely flat and MT is strongly trending Rep.) Possible but more of a trick that 2 of the former 3 states. The shrinking margin in MD may well end with a Dem win, but clearly some races that were viewed as likely Dem pickups or holds are now somewhat more in doubt that before, possibly including RI.

With two days to go there is a time limit on this dynamic. Reps may not have time to profit greatly from this trend, and we've seen sharp changes before so Dems may be able to recover (Republicans had a bad end of the week last week, after John Kerry and the Dems had a bad first of the week.) So no firm prediction here, but the evidence is that the Dems are falling back from their best chance of large gains.

Click here to go to Table of Contents

House 06: National forces estimate

I estimated the net national forces in the Senate rate last night. Here is the same estimation procedure applied to the House. This is based on 86 House races with a total of 380 polls. This is much less dense than the Senate data, but the results are surprisingly stable. Unlike the Senate, however, the data do not extend back in time very far, so the starting point here is June 1, 2006. The size of the effects here are also NOT comparable to the Senate forces, since both are estimated independently and the zero point is arbitrary in both cases. Relative movement is meaningful, so it is fine to say that since June 1 the net national forces in the House (for these 86 races) have risen about 6 percentage points.

As with the Senate, this is a good explanation for why so many House seats held by Republicans are now competitive. UNLIKE the Senate, these effects appear to have continued to grow recently. Even a much rougher fit still produces the upward rise at the end. This also is consistent with the growth in the Democratic advantage on the generic ballot, though the details of the dynamics are somewhat different.

The evidence then is favorable to larger than anticipated Democratic gains in the House, but smaller gains in the Senate, at least as of November 3. Four days to go.


Click here to go to Table of Contents

Sen 06: National Forces Estimate

This is certainly a good year for Democrats, but how good? And what are the national forces at work? I can estimate a summary of national forces to answer these questions.

I estimate a model that pools ALL Senate race polls, then iteratively fits a local regression (my usual trend estimator here) while simultaneously extracting a race-specific effect. This procedure has the effect of removing the difference between PA (with a strong Dem lead) and AZ (with a substantial Republican advantage) and likewise for all the states, effectively centering them at zero. The trend estimate that results then will move up if across most states the trend has been up, while if pro-Dem and pro-Rep movements equal one another, the national trend will be zero. There is no fixed metric for this national force, so it is convenient to pick a zero point for identification, in this case January 1, 2006.

The estimator finds that the Democratic margin has grown by 5 points across all races due to this national force. Where Republicans have enjoyed increased support, they have had to do it in the face of this opposing wind, while Democrats who would have been trailing by 5 points if January conditions still prevailed, will now have a "wind assisted" tossup race.

The dynamics of this national force have been generally increasing all year but with significant partial reversals at times. From a June high of about 4 points, this force shank to 2 points in August, then surged to 5 points by September 1. A brief improvement for Republicans took place in early September. At the time Republicans claimed to see new movement in their favor, and these data lend some support for that claim. However, that trend was sharply reversed after September 24 with the first publication and subsequent release of the National Intelligence Estimate followed by Bob Woodward's book, State of Denial. This was followed a week later by the Foley scandal, and once more Democratic advantage increased to about 6 percentage points. In the last two weeks of October there was a brief move in a Republican direction, then back to favor the Democrats. As of November 2, however, the national forces have again moved in a Republican direction, this time somewhat more strongly. While it is tempting to explain this as a result of Sen. Kerry's verbal difficulties, the downturn started before the joke-gone-wrong, so perhaps the Senator does not deserve all the credit for the 1.5 point decline since mid-October. As it stands, the estimate is only a little under 5 points. However, as a national force, common to all races, this decline of even 1.5 points is enough to be crucial for either party in Virginia and Missouri. If it moves more, it could also affect the Tennessee or Montana races as well (and conceivably Maryland.)

For my money, these are sensible estimates of the magnitude of national forces at work in this election. A gain of 5 points in the margin turns a 50-45 race into a 47.5-47.5 tie. Estimates much bigger than this would seem too large to be plausible as they would suggest too many races become competitive or Democratic leads.

The method I use here does not lend itself to the usual confidence interval estimates. But some sense of the variability of the estimator can be seen below. The estimation errors, indicated by the gray dots, are estimates of where the trend would be IF the series had stopped on the day represented by the dot. This method is sensitive to last observations and while the fit is quite stable when there is abundant data on both sides of a point of interest, it is often a poor predictor of what will come next. The deviations of the gray dots around the line show when the trend would have gone up more, or down more, than the blue trend estimator finally settled on. The errors are worse near points of change in direction, which makes sense. While the variability is not trivial, and indicates considerable uncertainty near changes of trend, the area covered by the gray dots is still relatively small compared to the size of the effect being estimated. The practical implication is that we have to be cautious in suggesting that the current trend will continue, because a change in direction is not well predicted by the model. That said, we can be reasonably confident that the trend estimator would not be radically different if we add more observations. (Which we won't do, after November 7.)

























Click here to go to Table of Contents

Sen 06: Four Critical Races

There have been some important changes in the Senate polling over the past week. Tennessee now appears to have turned against Democratic Rep. Harold Ford, while Virginia has moved away from Republican Sen. George Allen to a clear tossup. From now on, when people use the term "Tossup" they should show the plot of the Missouri race which lacks trends, bumps, wiggles or hints of what is to come. But the big news of today is the move that has been made in Montana where Democrats were ready to claim (amd many Republicans to concede) Sen. Conrad Burns' seat. President Bush visited and apparently money is now being devoted to this new "firewall" seat. A Burns win would require a Dem sweep of VA, TN and MO to manage a Senate majority. That is obviously a much higher burden than the "2 of 3" wins in these states required with MT in the Dem bag. It is worth noting the Burns is still behind in the trend estimate for MT, but clearly the level of competition has risen, and the odds of a Democratic Senate have shrunk. To make matters worse, in Maryland Democrat Ben Cardin still leads Republican Lt. Gov. Michael Steele but that lead has been shrinking steadily and while the normally Democratic state would be expected to go Democratic, the trend here and in the Governors race (see here) suggest that the Maryland race cannot be assumed to be over. The good news for Democrats (other than in VA) is that New Jersey Sen. Robert Menendez appears to have recovered his lead from Republican Thomas Kean Jr.


So as it now stands, the Dems need 3 of the 4 seats in MT, VA, MO and TN, while holding MD. That may be a tall order, and it makes it likely we won't know control of the Senate until the MT vote is in in the wee hours of Mountain Standard Time. Stay up! It will be fun.


Click here to go to Table of Contents

House 06: Generic Ballot

The generic ballot measure of the House has surged up and not stopped rising since September 22. The surge began the week in which the National Intelligence Estimate (NIE) appeared, followed by Bob Woodward's book, State of Denial. A week later the Foley scandal broke, adding to the move that began a week earlier.

The week or so before the NIE was published there was a small trend in the Republican direction which was remarked on in political news, but this very modest movement was abruptly revered. I would not have thought the NIE or Woodward revelations would have had much effect on mass public opinion, but the timing here is pretty convincing that these did in fact play a role. I speculate that was due to undermining the growth in approval of the administration on terrorism which built in late August and early September, though convincing data of this link is missing. Certainly no such inference is needed with regard to the impact of the subsequent Foley scandal.

The generic ballot is, of course, only a rough indicator of election outcomes (see here, but also see the forecasting efforts of Bafumi, Erikson and Wlezien here and Alan Abramowitz here). I also think the current upturn is a political equivalent of "irrational exuberance" in the sense that the run up in the polls seems likely to seriously overstate the actual vote margin. The current 17 point Democratic margin would be enormous, and even applying the "Charlie Cook Correction" of subtracting 5 points would still imply a 56-44 Democratic triumph. It may happen, but the generic ballot has virtually always overstated the Democratic lead, and this overstatement seems to get worst as the polling margin increases.

For a bit of perspective, the figure below plots the generic ballot since 1994. The conclusion is clear-- the poll measure has not been anywhere near current levels in the past 12 years. The practical result of this remains to be seen, but if Democrats fail to capitalize on this opinion advantage there will be some interesting research to understand why the seat gains fail to respond to this advantage in vote intent (well, in generic vote intent, which isn't the same thing.)


























Click here to go to Table of Contents

Gov 06: State of play

Here is a recap of what are (or at least once were!) the competitive Governor's races (AK and ID lack enough data for the analysis and are omitted.) The graph is ordered from the strongest Republican in the lower left corner to the strongest Democratic in the upper right.

Recent action that may affect election day is visible in NV where Republican U.S. Rep. Jim Gibbons is facing allegations of sexual assault. The race had looked strong for Gibbons but has now narrowed, with Gibbon's leading by under 5 points. In Maryland a small but consistent lead for Democratic challenger Martin O'Malley has all but vanished, leaving Gov. Robert Ehrlich a chance to hold on to the office. The reverse has happened in Minnesota where Republican Gov. Tim Pawlenty has lost the small lead he held over Democratic challenger Mike Hatch, with the race now a dead heat. In Iowa, Democratic fortunes have improved to a small lead, as have those of endangered Oregon Democratic Gov. Ted Kulongoski. In Wisconins, incumbent Dem. Gov. Jim Doyle has persistently held on to a small but relatively steady lead, making challenger U.S. Rep Mark Green's chances look longer than many (including me) expected.

No other races give any indication of shifts likely to threaten current leaders. The bottom line should be a considerable gain for Democrats. Our Pollster.com scoreboard shows 28 Dem, 20 Rep with 2 races too close for an assignment. This would give the Democrats a majority of Governorships for the first time since 1994, with potential advantages going into 2008 presidential contests.


Click here to go to Table of Contents

Bush Approval: 8 polls, trend at 37.5%

Over the past 10 days there have been eight new presidential approval polls. (Sorry I've been tied up with Pollster.com and haven't updated as often as usual. I hope to make up for that a little.) The net effect of this new polling is to indicate that the approval drop we saw recently has stabilized but at a low approval trend of 37.5%. That is about where the President stood at the beginning of the summer-- better than the all time lows of May but well below the recent maximum near 41%.

The polls fall above and below that trend, as you can see below. No outliers, though CBS/NYT is relatively low. Bottom line-- low approval for a President at midterm. If a picture is worth 1000 words, let's assume I've got 8,000 words below and let me move on to more posts!

















































Click here to go to Table of Contents