Terrorists and false positives

Matt Yglesias calculates:

…monitoring the UK’s 1.5 million Muslims is a lost cause. If you have a 99.9 percent accurate method of telling whether or not a given British Muslim is a dangerous terrorist, then apply it to all 1.5 million British Muslims, you’re going to find 1,500 dangerous terrorists in the UK. But nobody thinks there are anything like 1,500 dangerous terrorists in the UK. I’d be very surprised if there were as many as 15. And if there are 15, that means you’re 99.9 percent accurate method is going to get you a suspect pool that’s overwhelmingly composed of innocent people. The weakness of al-Qaeda’s movement, and the very tiny pool of operatives it can draw from, makes it essentially impossible to come up with viable methods for identifying those operatives.

Comments

I think young Yglesias needs to visit a British University--you could probably find 15 terrorists-in-training at each of them. They would likely come from a certain religious/cultural student society.

Well, as long as Matt Yglesias would be surprised if there were more than 15, I guess it's case closed. I'd be surprised if the UK doesn't already arrest way more than 15 dangerous terrorists who are ultimately found guilty every year. That said, it is indeed impractical to track 1.5 million people individually, but that's knocking down a straw man.

The comment by Desiavenger is to the point. Yglesias seems a bit clueless about the facts on the ground. Let's take his comment "nobody thinks there are anything like 1,500 dangerous terrorists in the UK." Well, how about the head of MI5, Eliza Manningham-Buller, who said in 2006 that her agency was tracking almost 30 high-priority domestic terrorist plots involving 200 networks and 1,600 suspects? In his defense,though, Yglesias was most likely not primarily concerned about facts in this post. He only recently learned about Bayes Rule and has been quite keen to show off his new toy. He does these little calculations every now and then. Quite charming really, in a pathetic kind of way ...

I think he rightly observes that tools like no-fly lists are either going to not work or contain many innocent people, but that doesn't mean there's nothing can be done to find members of Al-Qaeda or other violent groups in a civilian population.

Rather, I think that it suggests better results could be had with standard counter-intelligence methods, i.e. getting sympathetic people to try to join militant groups and cause disruption.

You boys ever hear the story of the elephant and the mouse?

Its a shame, to say the least, that we are witnessing a "debate" by "acadmeics" which assumes that all 1.5 million "British Muslims" are suspect terrorists? What happened to our notions of rational choice?

"I think young Yglesias needs to visit a British University--you could probably find 15 terrorists-in-training at each of them. They would likely come from a certain religious/cultural student society." - Um, he's explicitly saying that Muslims are the target group, so you don't need to make implicit innuendoes like that. The point is that in Muslim Students' Societies, like any other Muslim group, the probability that you will observe a terrorist is very, very, very low.

"I'd be surprised if the UK doesn't already arrest way more than 15 dangerous terrorists who are ultimately found guilty every year." - You know, you can look these things up to support your case rather than quasi-emote about what would happen if your assumption were true. It's fewer than 15.

"There is an infatuation, especially on the left, of the young bloggers who really have no track record, other than blogging." - There is an infatuation, especially on the right, of the ageing talk radio hosts who really have no track record, other than hosting talk radio.

This pseudo-intellectualism on lefty blogs drives me crazy. David Wright's comment above nailed it.

In every loose group of people who are onto something, there will be individuals who will act like "hubs" of communication, planning and execution.

Identifying those living communication hubs should be the primary goal of any serious counter-extremism effort.

I, personally, tend to think in categories of extremism rather than terrorism. Every terrorist is an extremist by definition, but it can't be told beforehand which extremist will turn into terrorist or active aide, and which will stay within the legal mantinels of obnoxious pontification, studying militant literature and sporadic shouting of slogans. So, it makes sense to think about monitoring extremists in general, and keeping an eye for signs of the transmutation from extremist into outright violent actor.

"Look down 2 posts from yours. You're wrong, and you're obnoxious about it."

1. You mean 1 post.
2. Over the nearly five-year period from 01-2004 to 09-2009, there were 43 convictions, which means at most 43 arrests over about five years, because "Some individuals will have been charged with more than one offence". That means, on average, 9 "dangerous terrorists who are ultimately found guilty every year". 9 is fewer than 15.
3. That doesn't mean I am right and you are wrong. My belief might be wrong and yours might be right. It just means that we have different beliefs, and you need to provide evidence to back up your case, rather than tell us that you'd act surprised if it weren't true, which to be honest is irrelevant information since we're all anonymous blog commenters with no authority.

@ Millian

1. The post 1 below refers ONLY to people stopped at ports under a certain act. That is a small FRACTION of all arrests, just people passing in and out of the country and not related to specific investigations, and it's already at 9. Under 15 is not in the ballpark, and if you were ever interested to know that you would.

2. You insist on demanding that I provide evidence, while you provided no evidence for your assertion of less than 15, and Yglesias provides absolutely no support. That's a double standard and is obnoxious.

3. Indeed, David Wright wins the thread. Read that post (as the post *2* below your original suggests) and it's clear why, as I said, Yglesias is knocking down a straw man.

David Wright,

The trouble with these independent filters at the 90% and 99% level is that we don't have any.

Our entire sample of confirmed terrorists is a few guys from the middle east, a few anti-abortionist males, a few homophobic rednecks, and a handful of randoms. What filters do you propose to extract from that dataset with respect to the general population?

About the only clear data I see is that domestic terrorists seem to be really killing Americans, while foreign ones seem to have switched to an odd form of performance art on airliners in which they bring inadequate amounts of explosives onto planes, attempt to ignite them in full view of other passengers, and get subdued, causing vast media attention and airport annoyance to the general public.

Matt is displaying a very profound ignorance about the threats posed by terrorism, and the means to combat it.

He's focused on the Crotch Bomber, and saying "Hey, we'll never catch all the individuals like this. If we try, we'll harass too many innocents. So we're hosed."

This is insane. The issue is not that this depressed, gullible kid flipped out and we didn't see it coming. The issue is that jihadist imams specifically look for guys like this and turn them into bombs. I'm sure Norway has plenty of depressed, gullible kids. You don't see them blowing up airplanes. Why is that?

You don't need to track 1.5 million people. You need to track the radical imams and the people that follow them.

As for the "What's the big deal about 300 dead people... planes crash all the time, we always endure" comment, that idiocy speaks for itself.

I'm really surprised anybody reads this guy.

@ db:

Matt is the one being naive about type I and type II errors. This:

"The point is that no matter how you try to find terrorists you are going to end up with an overwhelming number of false positives (even if your detection method is almost 100% accurate)"

is not true. To say that statisticians have not considered problems such as these and found solutions for them is absurd; David Wright outlined the basic principle underlying the fix. Matt's argument is essentially that rare events are impossible to detect without an unreasonable number of false positives; this is bogus.

much less several independent relatively high quality discriminators.

Emphasis on the "independent". Time and time again we have system where we get "one in a million failures" in a system with redundant protection where it turns out they all have exactly the same failure mode.

I would be startled if someone could come up with 3 completely independent discrimination schemes and would be willing to bet a considerable sum that the overlap between the false positives in scheme 1, 2 and 3 would be well over 50%.

"A fellow worshipper reported him make an inflammatory speech", "a policeman heard him make an inflammatory speech", "a surveillance team saw him talking with a bunch of young people after making an inflammatory speech."

It is rather amusing to see so many commenters insulting Yglesias' reasoning as amateurish without noting how Tyler, who is obviously not merely a "blogger" with no expertise in this subject, obviously considered the thought well-reasoned and interesting enough to quote and link.

I'll add that there almost certainly are somewhere in the range of 15-25 people in Britain at any given time who are going to personally launch dangerous terrorist attacks. Dangerous terrorist attacks aren't THAT hard to commit; clearly, the problem for an organization like al-Qaeda is a lack of qualified attackers in western nations, or there would be many upon many more attacks.

(Gah. That was the point of the quote. Kindly
ignore me. )

C'mon, Tyler, you know better than this. If you don't, ask Robin Hanson about Bayes. All you can do in life is play the odds, and the ban on profiling at airports worsens the odds.

We currently harass a huge number of innocent people at every airport. We just don't do it intelligently. To avoid discriminating, we give the third degree to people like Joe Foss, a retired general on his way to West Point to address the troops when his Congressional Medal of Honor sets off the metal detector.

Under the Clinton Administration, airport security paid more attention to people with Muslim names -- it was one factor in the risk profile. George W. Bush denounced airports profiling of Muslims and Arabs in his second debate with Al Gore in 2000 (Karl Rove thought that was crucial to winning the Michigan vote). Bush's transportation secretary Norman Mineta started a crusade against profiling people who look like terrorists.

The clerk, Michael Tuohey, who checked in Mohammed Atta on 9/11/2001 told Oprah four years later that he never saw anybody who looked more like an Arab terrorists, but then he gave himself a "politically correct slap" and decided not to single out Atta for attention by security.

"Why is that a good idea?" Because it is expected that they'll vote Labour.

To put a fine point on it, in TWOT you have the principle that to measure something affects the system, and in this case, the measurement method increases the products of the reaction.

Why did Britain allow 1.5 million muslims to settle in the country? Why did many other European countries do the same?

My hypothesis is that left-wing politicians have enabled mass immigration to enhance number of their voters. Does anybody other benefit from it?

Let's all be glad that Matt has never been in charge of taking on the Mob in New York City.

"We can't possibly monitor all the Italians in this city... there are over a million of them! And I doubt more than 15 are true crime leaders. How will we ever sort them out? What's the point of even trying?"

The logic of Matthew Yglesias is impeccable and irrefutable (even if all his details/hunches are wrong). If you have a method to try to screen terrorists, and that method is not 100% effective, and you apply that method to some subset of the population, you're more likely to trap more 'false positives' than actual terrorists. So true!

I assume therefore that Matthew Yglesias is highly opposed to the current security system we have in place at e.g. airports. This onerous screening method involves forcing those suspected to take off their shoes and belts, to put their luggage through machines, and disallows them from taking certain objects (e.g. Swiss Army knives, etc) onto the airplane. The subset of people this method is applies to is 'everyone'. And 99.99999% of the people who fall into this net are false positives (because virtually all of us aren't terrorists)! What a waste! How horrible!

Wait a minute though. If it's false positives that Matthew Yglesias opposed, shouldn't he favor any method that reduces their number? Our current methods seem to maximize their number by making everyone a positive. How is that better?

Pavel Kohout,

"Why did Britain allow 1.5 million muslims to settle in the country?"
Maybe somebody in England cares about what happens to people outside of the country.

"Why did many other European countries do the same?"
Maybe people in other European countries feel the same.

"Does anybody other benefit from it?"
You mean other than the 1.5 million Muslims? But who cares about them, right?

Sonic Charmer,
"Wait a minute though. If it's false positives that Matthew Yglesias opposed, shouldn't he favor any method that reduces their number? Our current methods seem to maximize their number by making everyone a positive. How is that better?"
Why are you arguing against a point that Matthew Yglesias never tried to make? Matthew Yglesias never claimed he agrees with current screening methods. He's simply stating that expecting screening systems to catch lone gunmen like the underwear bomber is unrealistic.

When did this blog get overrun with neocons? I hadn't previously noticed.

There really are a disconcerting number of neocons on these comments, and an alarming amount of Islamophobia, too.

The innumeracy on display in Yglesias's column starts with his offhand assumption that "99.9%" is a reasonable number for something that's "about as good as it can be". That's a common flaw - people who don't work regularly with statistics, experimental data, control systems, sampling errors, and other mathematical/scientific/engineering processes and tools often make that kind of mistake.

In the real world, we can and do demand precision much higher than that in all sorts of fields. For example, any company which strives for "Six Sigma" quality is looking for defect rates no greater than 3.4 parts per million. Expressed as a percentage, that would be 99.99966%.

What's a reasonable error rate for a terrorist screening program? Is 99.9% a reasonable number? Or is it 99.99%? Or 99.999%? Yglesias seems to have just pulled his number out of a hat and then attempted to build a rather weak argument around it.

Even after he built his argument around an arbitrary number, the answer he got, that there were 1500 potential terrorists in the UK, should have immediately clued him in that the problem was manageable, because a screening system that can get you down to a range of a thousand people now becomes amenable to human filtering (i.e. put a bunch of people in a room and have them start doing background checks and other means of determining whether the 1500 are really dangerous).

Another example of how he went wrong - is in his assumption that 1500 is unacceptably high. Not just because his estimate of 15 is unreasonably low, but even if 1500 was the number, and it represented a 10X over-estimation of the terrorist threat, is it really a burden on the system to single out that many people? A large majority of them won't fly in any given year. We'd be talking about employing additional security on maybe a few hundred people a year out of tens of millions who fly. This doesn't seem outrageous to me.

Yglesias's whole post was flawed from top to bottom.

Comments for this post are closed