Polling on the Iraq war inevitably become entangled with the politics of the war, with partisans seizing on results that support their preferences. This was vividly demonstrated this week by a Public Opinion Strategies (POS) poll conducted for The Moriah Group, a Chattanooga-based strategic communications and public affairs firm. (N=800, MOE=3.5%, conducted 2/5-7/07.) The POS press release is here. POS is one of the top Republican polling firms in the country.
After a New York Post story on the poll a variety of blogs on the left and right picked up on the results. See (here, here, here and here.) The Media Matters site posted an extensive critique of the poll and coverage of it here.
The ironic bit is that the poll isn't far out of line with other polling that has asked somewhat similar questions, yet those previous polls have not touched off a flurry of debate about opinion on the war. Rather, the strongly worded interpretation of the results, both in the POS press release and in the New York Post story, has provoked a reaction out of line with the novelty of these finding.
A further irony is that the POS survey uses wording for some options that seems likely to draw opinion towards those alternatives rather than others, and yet the results are only modestly different from previous polling. Whatever bias may exist in the wording, it did not produce results dramatically different from others we have seen in the last three months.
How questions are phrased, including about Iraq, can certainly affect the results. For comparison then, it helps if we can look at questions that pose reasonably similar alternatives, even if worded a bit differently. Let's look at how the recent polling has approached the question of what to do in Iraq. Eight recent polls plus POS have asked questions that address the choice among "immediate withdrawal", "withdrawal by a certain date", or "maintain troops until Iraq stabilizes." Some of these also include an option to "increase troop levels".
The POS question is
Which one of the following statements regarding the US involvement in Iraq do you MOST agree with...(The numbering of options is mine, for clarity.) It is a reasonable criticism of this question that options 3 and 4 offer elements that might be expected to draw respondents. Option 3 invites an expression of opposition to the war while supporting continued troop presence, and option 4 links the war explicitly to the war on terror as a rationale for maintaining troops. At the least, we might think this wording would increase support for the third and fourth options.
1) The US should immediately withdraw its troops from Iraq.
2) Whether Iraq is stable or not, the US should set and hold to a strict timetable for withdrawing troops.
3) While I don’t agree that the US should be in the war, our troops should stay there and do whatever it takes to restore order until the Iraqis can govern and provide security to their country.
4) The Iraq War is the front line in the battle against terrorism and our troops should stay there and do whatever it takes to restore order until the Iraqis can govern and provide security to their country.
Other polls have phrased things differently while getting at the same policy options. For example, Fox puts the question as
Thinking about the situation in Iraq, do you think the United States should:Here option 1 is essentially the same as the POS option 1. Option 2 is worded differently but amounts to the same policy option of withdrawal over a fixed period of time. And options 3 and 4 together are an expression in support of continued presence until the Iraqi's are able to take over. (One can argue if more troops is the same as continued presence, but I'm willing to argue that in this context.)
1. pull out all troops immediately,
2. pull out all troops gradually over the next year,
3. pull out after Iraqi troops are capable of taking over or
4. send more troops?
Gallup has offered a similar set of four options:
Here are four different plans the US (United States) could follow in dealing with the war in Iraq. Which one do you prefer--Again there are some variations in wording, but a similar thrust.
1. withdraw all troops from Iraq immediately,
2. withdraw all troops by January 2008--that is, in 12 months' time,
3. withdraw troops, but take as many years to do this as are needed to turn control over to the Iraqis, or
4. send more troops to Iraq?
The George Washington University Battleground poll, conducted jointly by the Republican Tarrance Group and Democratic Lake Research Partners uses this wording:
As you have probably heard, there has been a lot of debate over the past few months about what the United States should do about its troops that are currently stationed in Iraq. I am going to read you four proposals on this issue. Please tell me which one comes closest to your own....Finally the Pew Research Center asks two questions:
1. The US should begin immediately withdrawing all troops from Iraq. This is now a job for Iraqi forces to handle.
2. The US should set a date certain, no more than one year from now, when all troops be withdrawn from Iraq. This process should begin with some troops coming home immediately.
3. The US should keep its forces in Iraq until our military leaders there confirm that the situation in Iraq is stable enough that extremist forces will not be able to seize control once US troops leave.
4. The US should temporarily increase the number of troops in Iraq to help stabilize the situation more quickly.
Do you think the US (United States) should keep military troops in Iraq until the situation has stabilized, or do you think the US should bring its troops home as soon as possible?Combining the Pew responses we get three categories for 1) immediately, 2) within a year or two, and 3) keep troops until stabilized.
(If bring troops home as soon as possible): Should the US remove all troops from Iraq immediately, or should the withdrawal of troops be gradual over the next year or two?
The variations in question wording mean exact comparison is not possible, but the categories are reasonably similar in the policy options they offer.
For comparison, I combine the last two options of the POS survey. Both say we should continue a troop presence in Iraq until the country is stabilized though they offer different rationales for that presence. Similarly, I combine those options in other surveys that say we should remain until the situation is stable or we should send more troops. Again either option implies a continued presence. This gives three policy options that are comparable across all nine polls, regardless of wording:
1) Immediate withdrawal
2) Withdrawal within a specified time frame
3) Continued presence until Iraq is stabilized
The graph at the top shows the results of this. While there is variation across the polls it is modest compared to the stability of results. The immediate withdrawal option wins from 12-21% support, and averages 16.5%. Between 28% and 40% favor withdrawal by some deadline, with an average of 34.1% choosing this option. And those favoring continued presence in Iraq range from 42% to 53%, and average 45.5%.
The POS survey is a little higher (at 50%) for the combined options 3 and 4, but still only 4.5% above the average for this question. (The average is 44.9% without the POS poll.)
The POS result of 17% for immediate withdrawal is close to the 16.5% average, and POS's 32% for withdrawal by a deadline compares to an average of 34.1% preferring that option. (Removing POS from the averages results in 16.4% and 34.4%, not an appreciable change.)
So we return to the irony. Despite question wording that seems extreme compared to the other pollsters here, POS got at most a 5 point increase in support for keeping troops in Iraq until the country is stable. And the POS question produced very little difference from the average results for immediate or timed withdrawal. So critics who have jumped on the question wording bandwagon may be right about the wording, but they are substantially wrong about the effect.
One of the most annoying aspects of question wording effects is that sometimes they are large when you don't expect it, and sometimes they are small even when you are sure they should be large.
It is similarly ironic that those who delighted in the POS results were oblivious to the fact that there was not really any new news here. Apparently other pollsters from the "drive-by media" and elsewhere had been reporting substantially the same results for a while now.
Finally, the results of this question are: 49% withdraw immediately or by a deadline, and 50% stay until Iraq is stable. I'd say a 49-50 split isn't a strong indication of overwhelming support for either side. (There are other questions in the survey that address different issues-- some are more supportive of the war and some are less. Towards the high end is "I support finishing the job in Iraq, that is, keeping the troops there until the Iraqi government can maintain control and provide security for its people" 57% agree. But also: "Iraq will never become a stable democracy", 60% agree. Picking which results you like while ignoring the ones you don't may be good politics but it is bad polling analysis.)
If we phrase the question differently do we get different answers? Sometimes. I could illustrate that with other questions about Iraq easily. Opinion about the war is complex, with shifting coalitions of supporters and opponents depending on how questions slice the policy options. And sometimes choice of wording plays a substantial role in that. But not this time.
