So by now you've probably heard all about the story of a gang rape at a frat party at UVA that was published in Rolling Stone in December of last year. And you've probably heard about the subsequent investigation into the story which showed that the attack probably didn't happen, at least not remotely close to the way the purported victim described. Rolling Stone writer Sabrina Erdely was rightly criticized for publishing the story with insufficient fact checking and a healthy dose of deception (such as attributing quotes that were gathered second hand, without noting that the speaker hadn't ever been interviewed).
After the fall out from the article, Rolling Stone asked the Columbia School of Journalism to do a full investigation to see just what exactly went wrong. Though Columbia points to several problems in the initial fact gathering and fact checking, the main problem can really just be explained with the opening paragraph of the report:
Last July 8, Sabrina Rubin Erdely, a writer for Rolling Stone, telephoned Emily Renda, a rape survivor working on sexual assault issues as a staff member at the University of Virginia. Erdely said she was searching for a single, emblematic college rape case that would show "what it's like to be on campus now … where not only is rape so prevalent but also that there's this pervasive culture of sexual harassment/rape culture," according to Erdely's notes of the conversation. [Rolling Stone]
Every college freshman will recognize this style of writing. Conclusion first, then find whatever quotations are needed to support your case. The result is generally a bunch of crap.
But, that's not what we're going to focus on today. While Columbia generally gets its criticism of Rolling Stone right, they also get a bit sloppy with the facts and fail to do their own checking:
Erdely and her editors had hoped their investigation would sound an alarm about campus sexual assault and would challenge Virginia and other universities to do better. Instead, the magazine's failure may have spread the idea that many women invent rape allegations. (Social scientists analyzing crime records report that the rate of false rape allegations is 2 to 8 percent.) At the University of Virginia, "It's going to be more difficult now to engage some people … because they have a preconceived notion that women lie about sexual assault," said Alex Pinkleton, a UVA student and rape survivor who was one of Erdely's sources.
Social scientists analyzing crime records did not report that the rate of false rape allegations is 2 to 8 percent. What they did report is that 2 to 8 percent of cases are classified as false rape allegations. Might not sound like much of a difference, but it turns out to be a very substantial one. According to the article published in the journal Violence Against Women: "To classify a case as a false allegation, a thorough investigation must yield evidence that a crime did not occur."
The 2-8% figure is not the rate of false rape allegations, but instead the rate of allegations where there is substantial evidence indicating that it's probably a false allegation. Something close to the Clear and Convincing standard.
To see just what makes Columbia's claim so absurd, just imagine running the numbers in the other direction. How many investigations produce significant evidence that the allegations are true? According to RAINN, for every 32 reports to the police, 7 will lead to an arrest, 3 will be referred to a prosecutor, and 2 will result in a felony conviction. Assuming only cases with substantial evidence are referred to prosecutors, the rate of true allegations would only be about 10-22%.
But how could 2-8% of cases be false while only 10-22% are true?
Because in the vast majority of cases, we don't know either way. That's not at all how the Columbia School of Journalism presented it though. And you'd think in a story about bad fact checking and shady reporting practices, they wouldn't have played so fast and loose with the facts. Again, we see the same tricks employed by your typical college freshman: Find a source that comes close, misrepresent what it says by just a little bit, and hope the professor doesn't bother to do a careful check.