Wednesday assorted links


1) I read this yesterday. I am sure many of the details are over my head, and perhaps the author has been battered by painful examples of bad RCTs ... but big picture the whole thing looks like a losing rant to me. This is what you do when you've lost the war. You say things like "RCTs can't invent the iPhone."

What about this statement?

"There is an inverse correlation between the external validity of a RCT and the operational complexity of an industry"

"Perhaps either districts know something that these RCTs aren’t picking up, or perhaps districts are so poorly run that it takes a dramatic intervention to get them to adopt effective programs that have been around for 10+ years."

The author thinks this is a good argument against a robust finding from randomized control trials. A giant, government run monopoly doesn't innovate therefore the RCT's must not have external validity? What?

There are definitely external validity issues with RCT's, but like anon says, this is more rant than substance.

This statement is of course true.

As always, we have limited information. We should, when dealing with any complex system, have a non-intervention bias which favors stable, self-correcting solutions — large-scale adoption of best practices may have unintended consequences, suffer from errors of external validity, etc.

Having a bias toward non-intervention, though, is not the same as holding non-intervention as a pillar of faith. Any good Bayesian consequentialist should weigh the current level of pain and suffering against the possible benefits and downfalls of intervention (taking into account limited information and evidence from past well-meaning interventions).

I think this is the center of Kingsland's argument:

> In complex systems with complex organizations, evolution is a better change mechanism
> than running RCTs and implementing best practice adoption, especially in policy areas
> where some type of accountability (user choice, output measurement, etc.) can “kill off”
> bad ideas.

To which I say: it depends. For example, I'd favor a substantial carbon tax, while acknowledging that it will have widespread economic consequences that we can at best speculate about.

That one stumped me. It seems quite plausible, but I really struggle to wrap my head around why it should be so. Even in a very complex industry, can't you make a very specific thing a subject of RCT analysis?

Because the thing itself is subject to all sorts of complex interactions that confound the value of the outcome. For example, RCTs clearly tell you that single-payer health systems in Nordic democracies produce better population-level health outcomes than employer-focused market-driven health systems, a la the U.S., but one key claim of people who favor the U.S. model is that the Nordic systems benefit from drug development incentivized by the U.S. market. An RCT has a hard time telling you whether or not that claim is correct. It can tell you that the average Nordic citizen does better, but it might be that the outcome effectively depends on a form of free-riding. Or maybe not. The complexity of the system confounds the value of the RCT.

Basically, RCTs are great at differentiating between outcomes, but in complex systems with interconnected interactions, the RCT can't really tell you what factors led to the outcome. So you can make a very specific thing the subject of RCT analysis, but specific things are often an ineffective way to analyze highly complex systems and interactions, because the specificity of circumstances makes the experience non-repeatable, non-transferable, or both.

It occurs to me that a simpler way to say this would be, "Randomized Controlled Trial" is actually a misnomer, because the number of factors you would actually have to control is literally impossible, and it confounds the outcomes. That's why RCTs in social science are often of limited value. They are still useful, and still extremely valuable, and I think using them is better than not using them, but they have significant and very real limitations.

Fryer acknowledges weaknesses of RCTs in footnote 3:
"3) To be clear, randomized trials are not a panacea. There are important limitations to randomized controlled trials, ...First, many questions that are potentially interesting to economists may not be answerable with a randomized trial. .... Second, as with all statistics – the evaluation of field experiments has implications for the mean of the population and may have little value in predicting individual behavior. ...Third, and likely most constraining, are a host of important caveats which center on external validity. ...Fourth, Deaton (2010) expresses many concerns about the analyses and implementations of RCTs – exploring heterogeneous treatment effects can be viewed as data-mining and researchers should explore the implications of testing a large number of hypotheses in their studies; ..... Fifth, spillover effects could lead one to misstate a program’s overall effect. ...... Sixth, RCTs evaluating programs are considered “black boxes” that do not reveal the true mechanisms of interest. ..... Finally, Deaton (2010) and others argue that in an effort to overcome the above issues, RCTs can become prohibitively expensive. .. .. "

Of course, but it is answering a backlash that is weird. They say that to the man with a hammer every problem looks like a nail. This looks like a patient response saying "no, not every problem is a nail."

It doesn't say nails don't exist, or that hammers shouldn't be in tool boxes.

Neerav Kingsland, and a lot of people, hate hammers right now. That's weird.

Maybe they worry that if hammers get too popular everyone will be using them.

4. Of course, driverless vehicles have been around a long time. They are called elevated trains and subways. This whole discussion of "driverless cars" will one day make for a great study in human behavior: the human capacity to believe nonsense. I understand that some of the Silicon Valley "tech" companies actually employ really smart people whose job is to sit around and think of neat stuff to do, driverless cars being pretty neat. Of course, it's also ridiculous. Why "tech" companies employ really smart people to think of neat stuff is an interesting question. I suppose the smart people might even come up with something useful, like elevated trains and subways. Or it could be to convince people that these "tech" companies really are tech companies rather than glorified advertising platforms. They certainly have the money to employ these smart people, with all those profits generated from advertising.

Good points. Maybe I'm just a typical millenial who hates the aesthetic and cultural nightmare of suburban sprawl, but I get really depressed when I hear people say driverless cars are the way of the future. As if we could technologize our way out of the disgusting mess that public overinvestment in highways versus trains has wrought.

"aesthetic and cultural nightmare"

lol. "My worst nightmare is people having children and living in close-knit neighborhoods with yards. It's so ugly, compared to the run-down dirty crap urban areas! The culture is so terrible, where are all the nihilists and radical atheists??"

I think you have a very particular sense of what a city is like and a very particular sense of what the burbs are like, perhaps based on personal experience, but which are not quite representative of the range of realities of urban and suburban existence across this country.

Strip malls, miles of parking lots, and neighborhoods with infinitely repeating patterns of the same 3 split level floor plans, punctuated by the occasional office park, are less than ideal aesthetically than even a dirty city. Sorry Cliff.

And culturally, well there is no culture, because population density is so low and travel is so car-centric that you never actually come into contact with human beings other than the ones who live in your house (which is fewer and fewer these days, notwithstanding Cliff's assertions about "people having children"), other than the occasional finger given to an anonymous driver in a spat of road rage. For culture there must be community, and the suburbs have helped to kill that, along with the decline of the family which the rise of suburbia has failed to stem.

People driving from the nth Burger King straight into their garage doesn't make for community. Nor do scheduled play-dates and soccer practices -- how many kids today are allowed to leave the house and play with the kids in the neighborhood (if there even are any)?

If you like crap food, crap architecture, and miles of identical wall-to-wall carpeted central-AC closed-window houses where the only human interaction comes from a 50 inch TV screen, just say it with me, "I like crappy things and crappy culture."

Or it may be that the suburbs are more affordable?

Arguably, it is artificially affordable. Excessive building restrictions in cities decrease relative cost of suburbs, same with subsidized car transportation. And "move to the suburbs because they're cheaper" hasn't been the narrative -- it was "do well enough, and maybe you can move to someplace quiet in the suburbs." Because inexplicably, people actually thought that was desirable.

Nigel if cities are so great and full of culture of good thinking about transportation, then why don't we have a 2nd Avenue subway in NYC after 90 years of waiting. Too many artisanal hipsters in Brooklyn making pickles and telling each other they are non-conformist culture carriers.

Hiawatha: Non sequitur. Also, looks like you will, soon.

Or maybe different people have different preferences and rather than trying to lecture each other on The One True Path we just agree to live and let live?

Open up the zoning rules and let the market decide what the world should look like.

It's hard not to admire a society where the everyday goal of everybody isn't good health, knowledge or general happiness but instead finding a place to park the car as close as possible to their destination in order to walk as short a distance as possible. The "employee of the month" gets to park for thirty days right next to the boss by the front door. People that suffer strokes are given physical therapy as soon as practicable so they can get behind the wheel with a handicapped parking sticker so they needn't stagger across the parking lot to the supermarket. In Heaven there will be an open parking spot wherever you want to go.

Wait until you hit your late twenties. Things will start to feel different.

It's been fashionable for self-styled non-conformists to express contempt for suburbs at least since Pete Seeger sang "Little Boxes," in 1962.

Yet those who bash them (suburbs, and those who live there) sometimes seem just a little desparate to confirm their strict conformity to the conventions of a "non-conformity" that's become not only conventional but more than a little stale.

That's a great song.

And can't be about conformity, because clearly nobody agrees. Apparently they prefer to fight through hellish traffic twice or more every day so they can have a couple hundred extra square feet and a yard they have to mow every weekend in the summer.

You can count on me to vote with my feet. I moved out of the burbs as soon as I could and God willing I'll never have to live in a suburb again.

Just 7 years ago I felt exactly like you. Hated the burbs. Everything about it. Even had "little houses" on my ipod. I never would have guess how completely my views could I despise the city, love the suburbs, and think "little houses" is the type of pretentious song only the upper middle class can afford to uncritically enjoy.

Don't get married, and don't have kids.

"God willing I’ll never have to live in a suburb again."

Yeah, wait until you grow up.

There are places with low crime, unique and old architecture, short commutes, community, cultural appreciation, and affordable living. Small and medium cities in the middle of the country. And you don't have to be a wage slave to live a nice life here.

Thanks for the back-and-forth on this, guys. Very entertaining.


Subways are not generally driverless.

"Of course, driverless vehicles have been around a long time. They are called elevated trains and subways. "

Both elevated trains and subways almost always have drivers. There really wasn't much need to continue reading a post that starts with such an obvious fallacy.

Even if they were driverless, there's a world of difference between rolling down a track and rolling down a busy street.

I often refer to the Silicon Valley whiz kids as the Mad Men of today, their business being mostly high tech advertising. Of course, Mad is short for Madison, as in Madison Avenue, the street in NYC where the advertisers were once located. The Silicon Valley ad men deserve their own designation. How about Valley Men? No, it reminds me of Valley Speak, that horrible way some girls in California talk. How about Sili Men. No, it makes no sense. Con Men? That's it!

Sand Hill road. Sand men.

Besides the fact that subways are not generally driverless, there's a big difference between driving a vehicle on tracks that go specific from A to B to C to ... Z, with a handful of track swtiches, and autonomous driving in a "free range" road setting where people are swerving, changing lanes, and children or pets may jump out in front of you at any moment.

All those millions of riders on elevated trains and subways driving the damn things. All at once. How did they ever reach a destination?

We already have driverless cars, they're called cars. Cars have passengers.

#1 A betting market on Fryer's policy recommendations would be very interesting to me.

Have betting markets ever shown long term predictive value? Seems to me that their value is in something else, a rapid response meter to changing conditions.


A little more data please? Oil is always given as the counter example. It is an extremely important betting market, but futures represent nothing more than current sentiment.

Oil is not a good counter example:

But this bit from that page reinforces what I was saying above, about a rapid response meter to changing conditions:

"And it turns out that they all do worse than one simple forecast: the current oil price. That’s right: the most accurate forecast of oil prices over the next month, year, or quarter is the current oil price. We call this the no-change forecast."

Current price does better than futures.

But futures have a cost of carry included. I don't think your criticism changes the overall picture that the oil futures market does a pretty good job of predicting, welp, the future. If you can do better, you'd be a very, very rich person

6) Correlation is not causation.

$4 how does the Coase theorem and offsetting behavior play out with self driving cars. If they are more ridged in their driving get out of their way.

#4 fits my back of the envelop. If google accidentally kills one person (not, ya know, impossible) they have to drive on the order of 10 Billion miles safely before it's a slam dunk again. So far they've done ~1.5 million, which is notably smaller.

Current figures are approximately 1.1 fatalities per 100 million miles traveled in the United States ( Over 10 billion miles (10000 million), that would be approximately 110. I would think that the standard should be to at least match the human driven results.

Not necessarily. If a robot is driving those 100 mn miles then that's saving something like 2.5 mn man-hours of driving time which would be valued in the tens of millions of dollars range.

I mean, I agree that politically you probably have to promise that it's at least as safe. But yeah.

" 2.5 mn man-hours of driving time which would be valued in the tens of millions of dollars range"

Or maybe nothing at all, depending on whom the driver might be.

The rand study is misleading. Fatalities and injuries are considerably more rare than accidents but are clearly highly correlated. If we evaluate driverless cars on accident rates rather than fatality rates, an order of magnitude less data is required.


Near misses tell you a lot:

"It is true that if the probability of failure was as low as 1 in 100,000 it would take an inordinate number of tests to determine it ( you would get nothing but a string of perfect flights from which no precise figure, other than that the probability is likely less than the number of such flights in the string so far). But, if the real probability is not so small, flights would show troubles, near failures, and possible actual failures with a reasonable number of trials."

"The Rand study is misleading."

It seems highly misleading. The US had 0.77 injuries per Million Miles traveled in 2014.

"Autonomous vehicles would have to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their reliability in terms of fatalities and injuries."

You wouldn't need to travel hundreds of millions of miles to demonstrate reliability. You could probably get a good statistical base line within 30 million miles.

#4 - the study isn't about the safety of driverless cars, it's about the quantity of evidence that would be needed to conclude that autonomous vehicles are safe. And I agree with Parick that it seems to wildly overestimate the amount of evidence needed.

RAND study doesn't look meaningful. Accidents to miles driven is skewed by lots of low accident travel, but there is tons of data on where and when accidents are more likely to occur. If you could demonstrate safety in the worst environments then you can extrapolate to safe environments.

Of course you need a level playing field across new auto introductions. So the Google car just needs to show that it drives more safely than those driving the Ford Model in 1916. Not a very high bar.

The figures needed for high confidence aren't all that difficult to find. For example, all Tesla model X and S cars collect massive amounts of data about the environment the car is travelling in as well as what the driver AND autopilot do in the various situations. There's been roughly 125,000 Teslas sold so far (and that number is going to increase at an increasing rate). That's only one manufacturer.
We will get the needed dataset pretty quickly.

As a reliability engineer in the automotive industry, the RAND article was interesting to me. They address problems that all reliability engineers face on a day-to-day basis: how do we account for rare-events? Using strictly frequentist statistics, as the RAND article does, you quickly find that demonstrating a change in rate for any sufficiently rare event is practically impossible. Instead, we turn to Bayesian statistics and investigate the underlying causes that lead to such failures (i.e. fatigue, intoxication, etc.). When we eliminate a failure mode (say driver inattention), then that small slice of the total rare event pie is taken out. If we reduce a failure mode (accidents caused by poor visibility), then we proportionally reduce the size of the slice and as a result the entire pie shrinks. We use accelerated testing to push components/systems beyond their in-use limits and drive the failure rate to a level that can be address through frequentist statistics. Correlation between stress and life allows us to take these accelerated tests and make predictions at the in-use levels. So nothing in the RAND paper is wrong per se, but it's not really how things are done anymore. I imagine that just by eliminating fatigue, inattentiveness and intoxication, autonomous vehicles will have a significantly lower accident/fatality rate, though we will likely add some small proportion of new "failure modes" that couldn't be anticipated during development.

Another question I have is the appropriateness of using a 95% confidence level. That is a FAR more stringent requirement than people apply in their every day lives, and it sets the bar artificially high for autonomous vehicles. Will you only ride in the cars of people who are 95% percentile drivers? Is it somehow worse to get into a car accident when a computer is driving than a human?

In your final paragraph you confuse a 95% confidence interval with being in a top 5% driver. It's more like saying "I require X evidence that so-and-so is at least a median driver before I drive with him." Which people may still not do!

An interesting question is, suppose Google went to you and said, "Hey, here's an autonomous car. It's not as good a driver as you are when you're sober, alert, and have good visibility -- like, it's more likely to get in a crash than you are in those conditions. But it's never intoxicated or sleepy, and let's say for the sake of argument that it always has good visibility." Are you comfortable driving in such a car? It's an interesting question.

"Are you comfortable driving in such a car?"

Absolutely Not! ... But I'll take one for my wife. /ducks

Yes, you're right - my comparison of confidence level to driver performance really isn't correct. But it seems you (and hopefully others) got my point. Now please excuse me while I estimate the effectiveness of my apology in disclosure of my mistake :-)

I live 6 hours from the beach. I will definitely take the car that lets me come home from work on Friday, relax, eat a nice dinner with my family, pack, hop in at bedtime, and when everyone wakes up we are at the beach and it's Saturday morning. Likewise with the return trip on Sunday night.

As it is, every weekend trip to the beach eats at least one vacation day because I've got to spend 12 hours driving, and the more pleasant I try to make it for the kids, (by scheduling drive time during bedtime) the more dangerous it is due to my own tiredness.

Yes, give me the 90%-as-good-as-me autodriving car, particularly if I can be relatively certain that on a Friday night, most (all?) of the drunks will be in 90%-as-good-as-me autodriving cars, instead of driving themselves.

"Yes, give me the 90%-as-good-as-me autodriving car, particularly if I can be relatively certain that on a Friday night, most (all?) of the drunks will be in 90%-as-good-as-me autodriving cars, instead of driving themselves. "

Assuming the upgrade doesn't add more than 20% to the price of the car, I think it would be an easy sale. I've commonly (once a month) driven when I knew I was tired. I've looked down at a text that I've gotten on my phone and typed out a quick response (two or three times a week). And while it's been a long time since I've driven at or close to the alcohol limit, I'd like to be able to drink a third drink sometimes, instead of reflexively stopping at two because I have to drive home.

6. Sloppy with its use of "collusion." There is explicit collusion that is definitely forbidden by the antitrust laws, there is tacit collusion which isn't considered illegal at present but arguably could/should be, and there are a range of facilitating behaviors that can make either type of collusion more likely.

The observed effects could be the result of increased tacit collusion due to cross-holdings by institutional investors, rather than explicit collusion that might be observed in shareholder-management communications.

I find that the good folks at the Japanese Tradition have gotten apologies down to a science:

How completely sad that there had to be a university conducted on how to apologize. The only thing more sad that I can think of is that a professor actually got paid and used university resources to conduct this study. Oh well, bump the tuition up a notch so that more insightful research like this can be done.

...a university conducted study on how to...

5) Ridiculous. What's hard about it? If there are scalpers, just send some plainclothes officers every day and make arrests until it stops. It's not like there aren't thousands of police officers available in the city already doing not particularly useful things.

I live in a pretty minor city, and I've been told that I need "guanxi" to get an appointment. It strikes me more like people first play into the system, because having guanxi is something to brag about, and gives you some power over your social connections. However, the times I've had to go to a doctor, I walked in and had no difficulty getting either a) immediate service, b) a ticket to wait in a line that was maybe 10-20 minutes waiting, or c) booking an appointment within a reasonable timeframe.

Are you using any white privilege? (If you're white?)

I have to wonder if its a very popular doctor/hospital or something.

Cops don't stop it because he's paying them off.

I'll tell you my best standing in line story. So, I fly to Manila from Taiwan to get a new tourist visa. I go to the Taiwanese embassy in Manila, and there is simply a sea of people in line - out the door into the hallway.

I'm a little worried because I've got like 2 workdays to get my visa, and this line is not moving.

Security guard sees me, and I explain I'm trying to get a visa. He asks my nationality, and when I say USA, the security guard announces in a really loud voice "American?!?!?!? Front of the line!" and guides me over.

Later on I figured out the Filipinos were all trying to get guest worker visas, so it kind of made sense to prioritize travelers like me. Man, it was embarrassing...and yet necessary.

I always wondered why he said it in such a gleefully loud voice.

It would be better with a system where first there is a line where they just find out what you're applying for and give you a ticket. Then they can program such preferences instead of the guard doing that (and probably pissing of a lot of other people in the meantime). My last several visa/passport experiences were precisely like this. Among other things, it allows them to direct you to the desk with the staff member which is most specialized in your precise type of situation.

It would be better to sell priority access. The poor can pay with time, the rich can pay with money, everyone wins.

Will they lower fees for poor people as a result? Unlikely, and so the poor will pay more in the form of waiting longer and the rich will gain.

White privilege in China is undeniable, and I definitely benefit from it. I'm immediately ranked above any non-white person with equivalent experience and qualifications, there are many opportunities in acting and modelling where you get paid to show up for a few hours just to look white and get paid the equivalent of the monthly minimum wage. If I ever return to any restaurant or hotel, staff immediately remember all my preferences, and there's no tipping in China. People go to great lengths to be patient with my poor quality Mandarin. The list goes on ... (but does not extend to preferential access to business licenses, etc.)

When it comes to hospitals, I wonder if it might be related to the fact that even a white guy without a lot of guanxi is liable to raise a big stink if things aren't "right". Not sure ... it might be a different story for more major medical services which have more scarce supply of specialized doctors.

3) I've always been pretty indifferent about apologies. Also, I have no interest in "forgiveness", but am quite happy to acknowledge appreciation of an improved situation. The relevant thing is 1) admitting that it happened and 2) not doing it any more (allowing that some things are habitually ingrained, and may not entirely change, even if someone makes a pretty serious effort at reform, e.g., classic disagreements about housekeeping stuff, or perhaps stuff like staying out with friends a little later than you're "supposed" to).

I think of it this way: "just look yourself in the mirror and do better". I do not expect perfection, nor do I expect people to share my view of "perfection", but I expect people to at least try. An insincere apology (except for trivial things), I think is a sign that someone is not worth having anything to do with. I would rather a shrug of the shoulders than an insincere apology.

#2 The study does not control for any other technologies that may have been adopted simultaneously.

#4) This is what happens when you don't have an engineer around to advise the statistician on common practice and testing methods.

There are of course many corresponding examples of where a statistician could have told an engineer what numbers actually mean.

Comments for this post are closed