‘Weapons Of Math Destruction’ Outlines Dangers Of Relying On Data Analytics

Heard on  All Things Considered

NPR's Kelly McEvers talks with data scientist Cathy O'Neil about her new book, Weapons of Math Destruction, which describes the dangers of relying on big data analytics to solve problems.

KELLY MCEVERS, HOST:

We are in a time of big data. In recent years, NPR's done stories about how data analytics are being used to help political campaigns, rally supporters, compare the cost of similar surgeries in different cities, track public buses in real time and even maybe identify police officers at risk of committing misconduct. But the question is are we putting too much faith in big data? That's the question we're asking in this week's All Tech Considered.

MCEVERS: In her new book, mathematician Cathy O'Neil says we are in a techno utopia. And she does not mean that in a good way. Her book is called "Weapons Of Math Destruction: How Big Data Increases Inequality And Threatens Democracy." And she is with us now. Welcome to the show.

CATHY O'NEIL: Honored to be here, Kelly.

MCEVERS: So tell us what you mean by techno utopia.

O'NEIL: Well, techno utopia is this idea that the machine-learning tools, the algorithms, the things that help Google, like, have cars that drive themselves, that these tools are somehow making things objective and fair when, in fact, we really have no idea what's happening to most algorithms under the hood.

...

O'NEIL: Yeah, well, so everybody knows about the sort of decades-long attempt to improve public education in the United States. It goes by various names like No Child Left Behind, you know, Race to the Top. But at the end of the day, what they've decided to do in a large part is to sort of remove these terrible teachers that we keep hearing about. And the way they try to find these terrible teachers is through something called the growth model. And the growth model, mathematically speaking, is pretty weak and has had, like, lots of unintended consequences.

When I say weak, I interviewed a teacher from New York City public schools named Tim Clifford. He's been teaching for 20 years, multiple awards, he's written quite a few books. He got a 6 out of 100 one year and then a 96 out of 100 the next year. And he says his techniques didn't change. So it's very inconsistent. It's not clear what this number is actually scoring in terms of teachers and the teaching ability. I interviewed a woman named Sarah Wysocki in the D.C. area who actually got fired because of her low growth model score.

MCEVERS: There must be other examples, though, where people, you know, good teachers got good scores.

O'NEIL: Yeah, I mean, there certainly are, but I would say it's relatively close to a random number generator. So the fact that some good teachers got good scores doesn't say enough. I guess the point is that you might have some statistical information when you hear a score, but it's not accurate enough to actually decide on whether a given teacher, an individual teacher is doing a good job. But it's treated as such because people just trust numbers, they trust scores.

Continue reading or listen to the interview...