View my bio

Now the Details

Media, ethics, and journalism. What works. What doesn't.

Jeffrey Dvorkin

Monday, January 21, 2008

Polling the Pollsters: Let the Flagellating Begin!

Now that we are in full primary mode, the slips and gaffes of pundits and pollsters are there for all to see - thanks, in part to the blog-oisie which will point out the missing information faster than you can say "are you of voting age?"

To be fair, there are some reputable pollsters and polling agencies. But too often, news organizations report the polls with insufficient regard to the methods and too little accountability for the process. News organizations should use polling to help the electorate understand the issues. Too often, the media use polls instead of reporting, because it's just cheaper. In my experience, it's shoe leather that's required to get a clearer sense of what folks are thinking.

I once suggested at NPR that we not report any polls for a couple of weeks. One host was appalled: "But what will be talk about?" Eventually NPR did stop reporting every poll that was slipped over the transom and I think the listeners were better served. But journalists talking about polls give the illusion of "insider-ness" when in fact, more field reporting is required.

Here's why.

  1. Many - if not most polls are badly done. They are short on sample size. These days, a "reputable" national poll is claimed to be around 800 responses. But only a few years ago pollsters were claiming that you could not consider a poll reliable if it had fewer than 1500 respondants. Which is right?
  2. A "national" poll is usually done with the majority of calls to people in New York and Los Angeles, with a dozen in Houston, for example. Hardly a "national" poll.
  3. Is there a "return to sample?" If a pollster calls and no one is at home, does he or she call back later? That might require paying overtime. Instead the same demographic is called (homemakers, usually) thus skewing the sample.
  4. Cellphones? Still no serious attempt by most pollsters to tap into this group. That is about to change, but will it be in time for November?
  5. Push polls. These are polls that ask the question in a manner designed to elicit a specific response. We need to know how the question was posed.
  6. Omnibus polling. This is a cheap way of doing a poll, by paying a commercial pollster to ask whether you "prefer liquid or solid dog food, and who do you like in the upcoming race?" It's another less expensive way to conduct a poll, but it limits the sample size and the demographic.
  7. Lying. More pollsters privately admit that a signficant percentage of answers they get are simply false. The so-called "Bradley" effect (telling a pollster you will vote for a non-white candidate, then doing the opposite in the privacy of the polling booth) was seen again in New Hampshire where Obama was leading in the polls but came in second to Clinton.
  8. Range of accuracy. Most polls claim they are accurate plus or minus 4 percentage points. That range is statistically greater for smaller polls.
  9. Date of polling. When was the poll conducted? In New Hampshire polling stopped on the Sunday before Clinton "emoted." That event seemed to have some significance, but the pollsters missed it.
  10. Who pays? Lobby groups try to get their "polls" released just before an election and at a time when newsrooms are eager for "content." Beware of Sunday night polls released in time to catch the Monday morning editions.
There are ways to make sure that you are getting solid polling information. Dr. Alan Kay, for one, has written extensively about how to seperate the solids from the spin. But as the campaign intensifies, newsrooms should be wary. And so should the citizens. Polls have their uses in providing that snapshot of opinion, but it also depends who's operating the camera.

Polls have a serious downside: one reason why voter turnout is low and getting lower is, I believe because polls are implicitly anti-democratic: they imply that the race is over and decided even before people to go to the polling booths.


  1. Dear Sir:

    I note that you suggest that there is evidence poll results might diminish voting numbers. I have been looking for some peer reviewed papers saying that and have since found nothing, although as a Canmerican you might find interesting.

    Signed: New Reader

  2. Dear Mr. Strauss - In my experience, pollsters are reluctant to poll on this question, since it might adversely impact their livelihoods. But some academics have suggested that under certain situations in which the media has enthroned a particular candidate, the combination is a one-two punch for lowering voter turnout. As a Canadian-American, I think we need less punditry and more reporting. Don't you agree.

    Thanks for writing, and of course, for reading.