Bialik also makes a good point that rules out random sampling error as the culprit:
If the problem were merely sampling error, you would expect about the same number of seats to erroneously be assigned to Fatah as to Hamas. Yet all of the error seemed to go Hamas's way. Both pollsters showed Fatah beating Hamas by seven percentage points overall, yet Hamas edged Fatah by three percentage points in total votes. And since both survey groups polled more than 10,000 voters across all 16 districts, their margin of error for the overall vote was closer to 2%.
One of the issues the piece races is what to do in the face of potential non-response bias. Bialik quotes Dr. Said as follows:
But here is the rub: How do you know the non-responses are overwhelmingly for Hamas? (Or for George Bush in 2004?) There may be good reason to think this is so, but what data support the inference? In the U.S. we have the advantage of sample precinct returns which provide a check on the exit polls. As data come in from sample precincts they are compared with exit results from the same precinct, which allows estimation of how much non-response might be affecting the exit results. Such an option doesn't exist in the Palestinian case, where the counting process is much slower. So a Palestinian exit pollster is faced with a dilemma: adjust the results based on substantive expectations (really, your best subjective judgement about non-respondents) and admit that your statistical results are shifted by a clearly non-data driven component (which could be wrong). OR, decline to introduce non-data driven elements into the calculations, with the clear risk that your results may be biased by selective non-response. That is a really tough decision. (A third option would be to compare current results with historical election returns, but in the Palestinian case there is very little past election data to use. That will, of course, improve over time if democratic elections continue to be the practice. Such a comparison can't account for across the board shifts, but might provide some leverage on non-response that is otherwise unavailable.)
Still, Dr. Said blames himself for committing a fundamental error in data analysis: He changed his methodology in the face of surprising results. Logic dictated that, with or without Hamas interference, many mainstream voters who backed Hamas as a protest vote wouldn't reveal their choice to a stranger conducting an exit poll, Dr. Said says. And a greater refusal rate than usual -- his was 10%, above his norm of 8% -- would be expected in such a controversial contest. So he initially planned to assign more of the refusals to the Hamas ledger.
But Dr. Said let himself be influenced by the conventional wisdom surrounding the election. "Everyone here, as well as the CIA, Mossad -- everyone thought the best Hamas could do was a tie," he says. So he threw out his plan to assign the nonrespondents to Hamas.
"We should have added to Hamas in the final analysis, but we didn't because we thought the data made sense," Dr. Said says. "This is the first lesson you teach your students: You should not agree with any hypothesis, even that God exists -- it's a hypothesis you have to check."
I hope I'll get to post soon on some more details of the Palestinian election. In the mean time, both the PCPSR and DSP web sites have good post-election diagnostics of their polling which are worth a look.
The NumbersGuy quotes me:
I have a lot of admiration for what they were doing.That is absolutely true. I'm a strong "small-d" democrat. I don't know what Hamas will end up doing in office. And I don't expect miracles. But I do think that over the long haul (say 20-40 years) democratic institutions exert real pressure on political movements that enter the electoral arena. The IRA is a good example. It has taken a long time, but there has been real progress there. I hope the same for the Palestinians, though I think I am realistic about the long time frame required. What I so admire about the PCPSR, DSP and other Palestinian pollsters is that they are providing the independent data and analysis of what the Palestinian public thinks that is a necessary part of democratic institution building. Given the hard conditions in which they work, I'm very impressed with their excellence. I hope that they continue to provide the best data possible so that whoever runs the Palestinian Authority will have to consider what their public thinks.
