It’s time for a conversation about that oh so scintillating topic: research methods.
I know, if you made it to the third year of a four-year university program, you probably took such a class. And slept through it. And I don’t blame you. I really don’t. In fact I remember lying in bed, reading a paragraph from the textbook, nodding off, waking up, reading another paragraph, nodding off again, ad nauseum until I finally made it through the assigned reading. So believe me, I get it. Research methods are boring.
But things are getting out of hand lately. Let me start with a trivial example from an unreliable right-wing source. If a Daily Caller report is to be believed, researchers set up—the headline notwithstanding, “roadblocks” is the wrong word—well-marked roadside stands in Alabama, offering fairly substantial compensation: $10 for mouth swabs (possibly to collect DNA) and $50 for blood samples. We are to understand that this was for research into driving under the influence.
This is why they make you take research methods classes, damn it. Because this is wrong in so many ways. First, a study of drunk or drugged driving has no need for DNA evidence. This does not even begin to make sense. If they’re interested in a genetic link to alcoholism, they would recruit participants from Alcoholics Anonymous, or drunk driving programs, or in any of a number of other ways that would offer a more representative sample. That’s crucial, because any sample is somewhat self-selecting anyway, in that it is, if conducted even remotely ethically, drawn from sufficiently diverse volunteers so as to make up a representative group. Not just from people crazy enough to pull over and be tested by, we are told, off-duty cops. (The story omits any mention of whether the cops were in uniform, how potential participants were to know they were off duty, or how they could be assured that even off-duty cops might not arrest someone they suspect of being inebriated.)
Second, the compensation level for a blood sample is excessive. That compromises the voluntary nature of participation because to offer $50 crosses the line into an inducement. Even the $10 for a mouth swab is questionable. These amounts, in combination with the means of recruitment, are highly likely to skew the sample toward needy participants who are willing to accept a number of risks, including the risk of having samples drawn not in any hygienic clinical setting but at a road side stand.
Third, this is so wildly implausible that the study is almost certainly not what it claims to be about. That means the researchers—reportedly the Pacific Institute for Research and Evaluation funded by the National Highway Safety Administration—are employing deception. This is a violation of federal ethical rules for research with human participants, and only might be justifiable. Any such deception, if allowed, should be followed up with a “debriefing” in which the true nature of the study is revealed to participants and—a debriefing should be offered for this anyway—any necessary after-care is made available. But fourth, again according to the story, these participants were to be anonymous, meaning that no identifying information would even be collected (this is distinct from confidentiality, in which identities are protected). Except that these samples are of bodily fluids, which contain DNA—they can’t be anonymous.
That’s just off the top of my head. Now, let’s turn to a more serious matter. It is now exceedingly well established that, elite denials and obfuscations notwithstanding, agents of the U.S. national security complex are engaged in a vast data collection effort, invading the privacy of just about everybody, certainly not just people outside the U.S. This data is being collected and mined, allegedly to protect us from terrorism. To make a very long story short, this amounts to drawing—and acting upon—spurious conclusions from superficial data.
First, this is a necessarily quantitative analysis, which is a dubious methodology with humans. There’s a lot wrong with it that I’ve gone into elsewhere. Even within a quantitative paradigm, there are serious issues of validity. I mentioned with the Alabama study that research participants tend to be self-selecting. Further, with any human being, there is a vast number of variables, only a minuscule portion of which can be controlled for, which means that controls—essential to scientific methodology—are (to be polite) weak in any human research. This inherently undermines the legitimacy of conclusions generalized to entire populations. On top of this, it is an ecological fallacy to then apply a conclusion from one unit of analysis—a smaller or larger group— to any other unit of analysis, not least to an individual. But this is precisely what the recently exposed programs do.
But what the NSA is allegedly doing is even worse, because they aren’t trying for a representative sample. They’re trying for the world population. To the extent that they fail to collect data from the entire population, or collect it unevenly—really now, will they collect the same quality of information about a person in remote, unconnected tribal areas as they do in urban, highly-connected areas?—they will misrepresent that population.
So the methodology being employed by three-letter agencies and their private contractors is inexcusable. It gets worse. We now learn that,
Among the questions is how a contract employee at a distant NSA satellite office was able to obtain a copy of an order from the Foreign Intelligence Surveillance Court, a highly classified document that would presumably be sealed from most employees and of little use to someone in his position.
A former senior NSA official said that the number of agency officials with access to such court orders is “maybe 30 or maybe 40. Not large numbers.”
In other words, these people are unbelievably, abysmally delusional. That casts considerable doubt on their ability to ferret out any non-spurious conclusions from their inexcusable methodology even if they exist. To the extent that they act on any conclusions they do reach, it will be wildly, astronomically improbable that they will not bring the full horrifying weight of the national security state to bear on entirely the wrong people. And that, by itself, whatever one may think of the trade-off between civil liberties and national security, should scare us all shitless.
- Caroline May, “Off-duty cops collect DNA samples at Alabama roadblocks,” Daily Caller, June 10, 2013, http://dailycaller.com/2013/06/10/off-duty-cops-collect-dna-samples-at-alabama-roadblocks/↩
- May, “Off-duty cops collect DNA samples at Alabama roadblocks.”↩
- May, “Off-duty cops collect DNA samples at Alabama roadblocks.”↩
- James Bamford, “The NSA Is Building the Country’s Biggest Spy Center (Watch What You Say),” Wired, March 15, 2012, http://www.wired.com/threatlevel/2012/03/ff_nsadatacenter/; Dashiell Bennett, “Obama Administration Defends Its Right to Take All Your Phone Records,” Atlantic Wire, June 6, 2013, http://www.theatlanticwire.com/politics/2013/06/obama-administration-verizon-phone-records/65954/; Alexander Bolton, “Senators: NSA phone sweeping has been going on since 2007,” Hill, June 6, 2013, http://thehill.com/homenews/news/303891-senators-nsa-phone-sweeping-has-been-going-on-since-2007; Democracy Now!, “Exposed: Inside the NSA’s Largest and Most Expansive Secret Domestic Spy Center in Bluffdale, Utah,” March 21, 2012, http://www.democracynow.org/2012/3/21/exposed_inside_the_nsas_largest_and; Democracy Now!, “National Security Agency Whistleblower William Binney on Growing State Surveillance,” April 20, 2012, http://www.democracynow.org/2012/4/20/exclusive_national_security_agency_whistleblower_william; Democracy Now!, “Whistleblower: The NSA Is Lying–U.S. Government Has Copies of Most of Your Emails,” April 20, 2012, http://www.democracynow.org/2012/4/20/whistleblower_the_nsa_is_lying_us; Democracy Now!, “NSA Whistleblowers: “All U.S. Citizens” Targeted by Surveillance Program, Not Just Verizon Customers,” June 6, 2013, http://www.democracynow.org/2013/6/6/nsa_whistleblowers_all_us_citizens_targeted; Democracy Now!, “Glenn Greenwald on How NSA Leaker Edward Snowden Helped Expose a ‘Massive Surveillance Apparatus’,” June 10, 2013, http://www.democracynow.org/2013/6/10/glenn_greenwald_on_how_nsa_leaker; Democracy Now!, “‘On a Slippery Slope to a Totalitarian State’: NSA Whistleblower Rejects Gov’t Defense of Spying,” June 10, 2013, http://www.democracynow.org/2013/6/10/on_a_slippery_slope_to_a; Democracy Now!, “‘You’re Being Watched’: Edward Snowden Emerges as Source Behind Explosive Revelations of NSA Spying,” June 10, 2013, http://www.democracynow.org/2013/6/10/youre_being_watched_edward_snowden_emerges; Ryan Gallagher, “Details Revealed on Secret U.S. ‘Ragtime’ Domestic Surveillance Program,” Slate, February 28, 2013, http://www.slate.com/blogs/future_tense/2013/02/28/deep_state_book_uncovers_details_on_ragtime_domestic_surveillance_program.html; Barton Gellman and Laura Poitras, “U.S. intelligence mining data from nine U.S. Internet companies in broad secret program,” Washington Post, June 6, 2013, http://www.washingtonpost.com/investigations/us-intelligence-mining-data-from-nine-us-internet-companies-in-broad-secret-program/2013/06/06/3a0c0da8-cebf-11e2-8845-d970ccb04497_story.html; Glenn Greenwald, “Are all telephone calls recorded and accessible to the US government?” Guardian, March 4, 2013, http://www.guardian.co.uk/commentisfree/2013/may/04/telephone-calls-recorded-fbi-boston; Glenn Greenwald and Ewen MacAskill, “NSA Prism program taps in to user data of Apple, Google and others,” Guardian, June 6, 2013, http://www.guardian.co.uk/world/2013/jun/06/us-tech-giants-nsa-data; Mark Hosenball and Susan Heavey, “Obama administration defends phone record collection,” Reuters, June 6, 2013, http://www.reuters.com/article/2013/06/06/us-usa-wiretaps-verizon-idUSBRE95502920130606; Natasha Lennard, “‘Now we are all persons of interest’,” Salon, June 6, 2013, http://www.salon.com/2013/06/06/no_surprise_says_nsa_whistleblower_thomas_drake/; Natasha Lennard, “Revealed: “Boundless Informant,” NSA’s powerful datamining tool,” Salon, June 9, 2013, http://www.salon.com/2013/06/09/revealed_boundless_informant_nsas_powerful_datamining_tool/; Niels Lesniewski, “Durbin Predicted NSA, Verizon Controversy in 2009,” Congressional Quarterly Roll Call, June 6, 2013, http://blogs.rollcall.com/wgdb/durbin-predicted-nsa-verizon-controversy-in-2009/; Declan McCullagh, “DOJ: We don’t need warrants for e-mail, Facebook chats,” CNET News, May 8, 2013, http://news.cnet.com/8301-13578_3-57583395-38/doj-we-dont-need-warrants-for-e-mail-facebook-chats/; Claire Cain Miller, “Tech Companies Concede to Surveillance Program,” New York Times, June 7, 2013, http://www.nytimes.com/2013/06/08/technology/tech-companies-bristling-concede-to-government-surveillance-efforts.html; Alex Pareene, “The government has all your info,” Salon, June 6, 2013, http://www.salon.com/2013/06/06/the_nsa_has_all_your_info/; Bill Quigley, “Twenty Examples of the Obama Administration’s Assault on Domestic Civil Liberties,” Truthout, December 7, 2011, http://truth-out.org/news/item/5454:twenty-examples-of-the-obama-administrations-assault-on-domestic-civil-liberties; Steven Rosenfeld, “How Obama Became a Civil Libertarian’s Nightmare,” Alternet, April 18, 2012, http://www.alternet.org/story/155045/how_obama_became_a_civil_libertarian%27s_nightmare; Charlie Savage and Edward Wyatt, “U.S. Is Secretly Collecting Records of Verizon Calls,” New York Times, June 5, 2013, http://www.nytimes.com/2013/06/06/us/us-secretly-collecting-logs-of-business-calls.html; Charlie Savage, Edward Wyatt, Peter Baker, and Michael D. Shear, “Obama Calls Surveillance Programs Legal and Limited,” New York Times, June 7, 2013, http://www.nytimes.com/2013/06/08/us/national-security-agency-surveillance.html; Bruce Schneier, “What We Don’t Know About Spying on Citizens: Scarier Than What We Know,” Atlantic< June 6, 2013, http://www.theatlantic.com/politics/archive/2013/06/whats-scarier-than-the-verizon-leak/276607/; WashingtonsBlog, “Americans Are The Most Spied On People In World History,” December 5, 2012, http://www.washingtonsblog.com/2012/12/americans-the-most-spied-on-people-in-world-history.html; Shaun Waterman, “Whistleblower’s NSA warning: ‘Just the tip of the iceberg’,” Washington Times, June 7, 2013, http://www.washingtontimes.com/news/2013/jun/7/the-national-security-agencys-collection-of-phone-/; Rachel Weiner, “Mark Udall: I tried to expose NSA program,” Washington Post, June 6, 2013, http://www.washingtonpost.com/blogs/post-politics/post/mark-udall-i-tried-to-expose-nsa-spying/2013/06/06/6e567a52-cee8-11e2-8573-3baeea6a2647_blog.html↩
- David Benfell, “For those who still aspire to a natural science approach,” November 8, 2011, https://parts-unknown.org/wp/2011/11/08/for-those-who-still-aspire-to-a-human-science-approach/↩
- Peter Finn, Greg Miller, and Ellen Nakashima, “Investigators looking at how Snowden gained access at NSA,” Washington Post, June 10, 2013, http://www.washingtonpost.com/world/national-security/investigators-looking-at-how-snowden-gained-access-at-nsa/2013/06/10/83b4841a-d209-11e2-8cbe-1bcbee06f8f8_story.html↩