Nate Cohn embarrassingly goes for luck rather than rigor

benfell


Fig. 1. Graph showing the deterioration of response rates by the Pew Research Center, February 27, 2019, fair use.[1]

I wasn’t planning on commenting on a story by Nate Cohn in the New York Times on how recent polling results resemble the failures in two recent election cycles.[2] Thankfully, I have my mother to remind me that yes, I probably do need to comment.

I’m a qualitative guy. I’m really not much for getting into the weeds of quantitative work. But I’ve harshly criticized polling as a “non-methodology:” That’s because when the response rate, which should be ninety percent of a representative sample or more,[3] is instead in the single digits,[4] pollsters have a self-selecting sample that cannot represent non-respondents, particularly when, due to non-response, we have no idea, none whatsoever, how members of that self-selecting sample differ from all those non-respondents. This completely invalidates the methodology and I don’t care what rhetoric pollsters deploy to excuse themselves—the claim that polling works regardless[5] is belied by all the instances in which it doesn’t[6]—or what statistical magic they think they can employ to get around this problem—an absence of data remains an absence of data. This is a non-methodology that persists because we are desperate for an imaginary certainty of numbers,[7] and because people have built entire careers around this non-methodology.

And yes, that’s my very slightly modified boilerplate text for this kind of thing. The non-methodology that Cohn relies upon, however, builds on what we can see as an entirely to-be-expected failure of polling to say, we’ve seen these sorts of polling margins before in certain places, and they were wrong. Therefore, they may be wrong again.[8] And well, yes, they could.

But this election cycle has proven starkly different from those other two.[9] And if we’re going to be quantitative, well, two just ain’t much of a sample size. So what Cohn is really doing here is unreliably relying on unreliable results.

Which is to say, that for Cohn’s method (for lack of a better name) to be reliable, the underlying polling data would have to be reliably wrong. We don’t know that. Indeed, as I noted, pollsters continue to defend their results[10] because even though they’ve had some, um, prominent failures,[11] they can selectively observe their successes.[12] Oh, to be a quantitative hypocrite and even to get paid for it.

But it gets worse. Because the base of errors that Cohn is pointing to occurred in a mere two election cycles,[13] as if polling failures are somehow something new. But the response rates have been plummeting for decades—figure 1 only shows the last two decades— and, remember that those response rates are supposed to be at least 90 percent, something that was, by far, no longer the case even when my first methods professor commented on the problem in 2003.[14] Cohn is selectively observing failures and relying on a methodology built off those mere two failures.[15] This is so bogus, it’s astonishing. It’s facepalm on top of facepalm. It’s a made-up story and the New York Times ought to be ashamed of itself for even letting Cohn wander down this path.

Now, like I said, there’s a chance that despite all this methodological nonsense, Cohn is right. But if he is, it’s more luck than rigor.

  1. [1]Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
  2. [2]Nate Cohn, “Yes, the Polling Warning Signs Are Flashing Again,” Washington Post, September 12, 2022, https://www.nytimes.com/2022/09/12/upshot/polling-midterms-warning.html↩
  3. [3]This according to Valerie Sue, the professor in my first research methods class, at California State University, Hayward (now East Bay), Fall 2003.↩
  4. [4]Steven Shepard, “Report: Phone polls aren’t dead yet,” Politico, May 15, 2017, https://www.politico.com/story/2017/05/15/pollsters-phone-polls-238409; Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
  5. [5]Steven Shepard, “Report: Phone polls aren’t dead yet,” Politico, May 15, 2017, https://www.politico.com/story/2017/05/15/pollsters-phone-polls-238409; Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
  6. [6]Dan Balz, “2020 presidential polls suffered worst performance in decades, report says,” Washington Post, July 18, 2021, https://www.washingtonpost.com/politics/2020-poll-errors/2021/07/18/8d6a9838-e7df-11eb-ba5d-55d3b5ffcaf1_story.html; David Byler, “Polling is broken. No one knows how to fix it,” Washington Post, July 22, 2021, https://www.washingtonpost.com/opinions/2021/07/22/polling-is-broken-no-one-knows-how-fix-it/; Mona Chalabi, “The pollsters were wrong – again. Here’s what we know so far,” Guardian, November 4, 2020, https://www.theguardian.com/us-news/commentisfree/2020/nov/04/the-pollsters-were-wrong-again-heres-what-we-know-so-far; David A. Graham, “The Polling Crisis Is a Catastrophe for American Democracy,” Atlantic, November 4, 2020, https://www.theatlantic.com/ideas/archive/2020/11/polling-catastrophe/616986/; Steven Shepard, “Dem pollsters acknowledge ‘major errors’ in 2020 polling,” Politico, April 13, 2021, https://www.politico.com/news/2021/04/13/dems-polling-failure-481044↩
  7. [7]Jacques Ellul, The Technological Society, John Wilkinson, trans. (New York: Vintage, 1964).; Neil Postman, Technopoly: The Surrender of Culture to Technology (New York: Vintage, 1993).↩
  8. [8]Nate Cohn, “Yes, the Polling Warning Signs Are Flashing Again,” New York Times, September 12, 2022, https://www.nytimes.com/2022/09/12/upshot/polling-midterms-warning.html↩
  9. [9]David Benfell, “The really, really, really wild wildcards in the 2022 and 2024 elections,” Not Housebroken, August 17, 2022, https://disunitedstates.org/2022/08/17/the-really-really-really-wild-wildcards-in-the-2022-and-2024-elections/↩
  10. [10]Steven Shepard, “Report: Phone polls aren’t dead yet,” Politico, May 15, 2017, https://www.politico.com/story/2017/05/15/pollsters-phone-polls-238409; Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
  11. [11]Dan Balz, “2020 presidential polls suffered worst performance in decades, report says,” Washington Post, July 18, 2021, https://www.washingtonpost.com/politics/2020-poll-errors/2021/07/18/8d6a9838-e7df-11eb-ba5d-55d3b5ffcaf1_story.html; David Byler, “Polling is broken. No one knows how to fix it,” Washington Post, July 22, 2021, https://www.washingtonpost.com/opinions/2021/07/22/polling-is-broken-no-one-knows-how-fix-it/; Mona Chalabi, “The pollsters were wrong – again. Here’s what we know so far,” Guardian, November 4, 2020, https://www.theguardian.com/us-news/commentisfree/2020/nov/04/the-pollsters-were-wrong-again-heres-what-we-know-so-far; David A. Graham, “The Polling Crisis Is a Catastrophe for American Democracy,” Atlantic, November 4, 2020, https://www.theatlantic.com/ideas/archive/2020/11/polling-catastrophe/616986/; Steven Shepard, “Dem pollsters acknowledge ‘major errors’ in 2020 polling,” Politico, April 13, 2021, https://www.politico.com/news/2021/04/13/dems-polling-failure-481044↩
  12. [12]Steven Shepard, “Report: Phone polls aren’t dead yet,” Politico, May 15, 2017, https://www.politico.com/story/2017/05/15/pollsters-phone-polls-238409; Courtney Kennedy and Hannah Hartig, “Response rates in telephone surveys have resumed their decline,” Pew Research Center, February 27, 2019, https://www.pewresearch.org/fact-tank/2019/02/27/response-rates-in-telephone-surveys-have-resumed-their-decline/↩
  13. [13]Nate Cohn, “Yes, the Polling Warning Signs Are Flashing Again,” Washington Post, September 12, 2022, https://www.nytimes.com/2022/09/12/upshot/polling-midterms-warning.html↩
  14. [14]This was Valerie Sue, the professor in my first research methods class, at California State University, Hayward (now East Bay), Fall 2003.↩
  15. [15]Nate Cohn, “Yes, the Polling Warning Signs Are Flashing Again,” Washington Post, September 12, 2022, https://www.nytimes.com/2022/09/12/upshot/polling-midterms-warning.html↩

One thought on “Nate Cohn embarrassingly goes for luck rather than rigor”

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.