Needlestack

The Dallas Observer carries a story about Michael Bills, who was working for the TSA. He was fired after a year working there because they just found out that he had a marijuana arrest and was a child molester.

So what’s the big deal? Well the way they found out about his pot past (after a year of work) was by reading about it on his application, which he truthfully listed the arrest on. I’m not quite sure how they found out that he was a sex offender but they failed to notice that the child molester was Michael Douglas Bills, not the Michael Shane Bills who worked for the TSA.

So a federal agency was dumb, it wouldn’t be the first time. Again, what’s the big deal? Well the TSA is trying to get into data mining with the CAPPS II program. It hopes to catch terrorists by looking at various bits of information from the commercial sector as well as government records.

The problem with these data mining programs is that they don’t work. From Bruce Schnier’s How We Are Fighting the War on Terrorism / IDs and the illusion of security:

But any such system will create a third, and very dangerous, category: evildoers who don’t fit the profile. Oklahoma City bomber Timothy McVeigh, Washington-area sniper John Allen Muhammed and many of the Sept. 11 terrorists had no previous links to terrorism. The Unabomber taught mathematics at UC Berkeley. The Palestinians have demonstrated that they can recruit suicide bombers with no previous record of anti-Israeli activities. Even the Sept. 11 hijackers went out of their way to establish a normal-looking profile; frequent-flier numbers, a history of first-class travel and so on. Evildoers can also engage in identity theft, and steal the identity — and profile — of an honest person. Profiling can result in less security by giving certain people an easy way to skirt security.

There’s another, even more dangerous, failure mode for these systems: honest people who fit the evildoer profile. Because evildoers are so rare, almost everyone who fits the profile will turn out to be a false alarm. This not only wastes investigative resources that might be better spent elsewhere, but it causes grave harm to those innocents who fit the profile. Whether it’s something as simple as “driving while black” or “flying while Arab,” or something more complicated such as taking scuba lessons or protesting the Bush administration, profiling harms society because it causes us all to live in fear…not from the evildoers, but from the police.

There simply isn’t enough data to build a good terrorist model. Let’s take two recent American terrorists: John Allen Muhammad and Timothy McVeigh. What did their records have in common before they acted? The only common data point between the two is that they both served in the military. If we had a system that could spot these two men, it would also falsely identify every single male who served in the US Military.

That of course assumes that the data is properly mined and analyzed. But let’s go back to the initial story, where we find out that the TSA sucks at analyzing data. Where does that leave us?

Some might say finding an evil-doer among regular people is akin to finding a needle in a haystack. I say that since there’s no way to tell the bad from the good it’s closer to finding a specific needle in a needlestack. Is that really worth giving up our privacy for an illusion of security?