Alex Tabarrok


by on March 15, 2018 at 7:25 am in Data Source, Economics, Education | Permalink

The Richmond Fed has a good overview of apprenticeships in the United States and some of the academic literature:

According to a 2013 World Bank and International Labour Office study, only about 0.3 percent of the total U.S. workforce is in registered apprenticeships — about a 12th of the share in Germany. But some states, including South Carolina, have expanded “dual system” apprenticeships in recent years by building partnerships between colleges and firms and, in some cases, offering tax credits. Through the state’s “Apprenticeship Carolina” program, about 27,000 workers have been trained since 2007, including many at foreign-owned firms. Nationwide, there were about 505,000 registered apprentices in 2016, according to the U.S. Labor Department.

The review offers some useful ideas on why apprenticeships are less common in the United States. One problem is cultural:

In other countries, it’s more likely that college is seen as one option among many, and apprenticeships are con­sidered a worthwhile route to middle-class employment. In the United States, parents are more likely to see college as a vital investment without considering other alterna­tives…

As I said in Launching the Innovation Renaissance:

The U.S. has paved a single road to knowledge, the road through the classroom. “Sit down, stay quiet, and absorb. Do this for 12 to 16 years,” we tell the students, “and all will be well.” Most of them, however, crash before they reach the end of the road — some drop out of high school and then more drop out of college. Who can blame them? Sit-down learning is not for everyone, perhaps not even for most people. There are many roads to knowledge.

If you want to lower the price of housing and still house lots of people there is really only one way: build more housing. Yet politicians and voters continually seek to repeal the laws of supply and demand. A case in point, many states reduce property tax rates for seniors, veterans or the disabled or combinations thereof. Great for seniors, veterans and the disabled, right? Wrong. If supply doesn’t increase, lowering property taxes simply increases the price of housing.

If the property tax relief is targeted to a very small group then demand won’t increase much and the benefits will accrue to the targeted group but seniors and veterans are both a significant fraction of the population and an even more significant fraction of homeowners. Thus, we might expect that a significant fraction of the tax relief will be capitalized into housing prices–that’s exactly what Moulton, Waller and Wentland find in a new paper:

While property tax relief measures are often intended to aid specific groups, basic supply and demand analysis predicts that an unintended consequence of this particular kind of tax relief is that, on the margin, it increases demand for homeownership among its expected beneficiaries. Accordingly, we examine two property tax relief measures in Virginia that applied to disabled veterans and the elderly, finding that these policy changes had an immediate effect on home prices after the
voters approved them on Election Day. Overall, we find that home prices rose by approximately 5 percent in response to the increase in demand for homeownership. Indeed, the tax relief policies provide a unique, quasi-experimental methodological
setting where the treatment is exogenously assigned to specific groups within this market. We find that the effect was as much as an 8.1 percent price appreciation for homes in areas with high concentrations of veterans, 7.3 percent in areas with
more seniors, and 7.4 percent for senior preferred homes in all areas. The effect was highest, 9.3 percent, in areas with high concentrations of seniors and veterans, which translates to about $18,900, or roughly full capitalization, for the average
home. Conversely, the tax relief measures had little if any effect on homes in areas with fewer potential beneficiaries….

A cynic might argue that the true intent of the policy is to raise housing prices but this gives politicians and voters too much credit. The intent is sincere, it’s the means that are false.

Every improvement in computing power and artificial intelligence raises anew the claim and, for some, the hope that now we can centrally plan the economy. I was asked at Quora whether this will ever happen.

I will begin by accepting that there is nothing inherently impossible about an AI running an economy so, for the sake of argument, let’s say it could be possible using today’s computing power to run a small economy in say 1800. Nevertheless, I assert that an AI will never be intelligent enough to perfectly organize a modern economy. Why?

The main reason is that AIs will themselves be part of the economy. Firms and individuals use AIs to make decisions. Thus, any AI has to take into account the decisions of other AIs. But no AI is going to be so far advanced beyond other AIs that this will be possible. In other words, as AIs increase in power so does the complexity of the economy.

The problem of perfectly organizing an economy does not become easier with greater computing power precisely because greater computing power also makes the economy more complex.

Hat tip: Don Lavoie.

A new paper (another summary) in Nature reports on what is perhaps the world’s biggest field experiment which has successfully shown how to, at scale, increase crop yields and reduce fertilizer usage in China. The scope of the 10 year experiment is astounding. The researchers first conducted thousands of field experiments all over China to discover and validate best practices:

A total of 13,123 site years of field trials were conducted from 2005 to
2015 for the three crops (n=6,089 for maize, 3,300 for rice and 3,734 for wheat), with sites spread across all agro-ecological zones…Each field trial included two types of management: conventional farmers’ practice (control) and ISSM-based recommendations (treatment; developed specifically for a given area). The recommended practices were discussed with local experts and participating farmers. Adjustments were made when necessary. Finally, the agreed-upon management technologies were implemented in the fields by the farmer; the collaborators provided guidance on-site during key operations, such
as sowing, fertilization, irrigation and harvest. Campaign collaborators recorded fertilizer rate, pesticide and energy use, and calculated nutrient application rate. At maturity, grain yield and above ground biomass were sampled by the collaborators for plots with a size of 6m^2 for wheat and rice, and 10m^2 for maize. Plant samples were dried at 70 °C in a forced-draft oven to constant weight, and grain yield was standardized at 14% moisture for all crops.

With validated best practices in hand the researchers and tens of thousands of collaborators then fanned out across the country to convince farmers to adopt the best practices.

During the campaign, about 14,000 training workshops, 21,000 field days, and more than 6,000 site demonstrations were organized by campaign staff; more than 337,000 pamphlets were distributed….During the campaign, we also encountered barriers and experienced challenges.  For example, we observed that some farmers appeared indifferent during some  outreach events. We later learned that it was mainly, because they could not comprehend the scientific content that we were trying to deliver. We solved the problem by having local (county or township) agents acting as an on-site ‘interpreter’ in  ways that speaks/connects with those farmers.

This was amusing:

It is also worth noting that the interests of agribusinesses do not always align with those of our campaign staff. For example, one of our main strategies used in the campaign was to select a site (for example, a village) for a given area, establish the base with field demonstrations of ISSM-based practices, then attract and engage more farmers from the same as well as neighbouring villages, creating  a snowballing and lasting effect. But sometimes, our partners in the private sector were more interested in changing sites so as to reach more farmer-clients. Vigorous  debates and discussion ensued. Eventually, the private sector personnel conformed to our reasoned schemes while using the established sites as demonstrations for  visitors from other areas.

Outputs and inputs among the treatment and control farmers were then measured (here I would have liked more information about the randomization. A lot can go wrong or be mismeasured at this stage.).

Farmers conducted all field operations. Campaign collaborators and/or extension agents were responsible for information and data collection. Typically, 10–30 farmers were randomly selected per ISSM-adopting site; another group of randomly selected 10–30 farmers from a nearby village without ISSM intervention served as a control/comparison. From the selected pool of farmers (roughly 14,600 paired data points), information on key management practices were obtained through a questionnaire survey, including crop varieties, planting densities, planting dates, fertilizer rates and harvest dates. For some sites, grain yields were directly measured in the same way as the field trials (see ‘Field trials’) for the selected 10–30 farmers. Yield and nitrogen rate were then averaged for each site.

The results were impressive.

Aggregated 10-year data showed an overall yield improvement of 10.8–11.5% and a reduction in the use of nitrogen fertilizers of 14.7–18.1%, when comparing ISSM-based interventions and the prevailing practices of the farmers. This led to a net increase of 33 Mt grains and a decrease of 1.2 Mt nitrogen fertilizer use during the 10-year period, equivalent to US$12.2 billion.

The entire experiment cost on the order of $56 million and generating $12.2 billion dollars of increased output, not including any environmental gains.

As if this weren’t enough the researchers then surveyed over 8 million smallholder farmers in China to estimate how much output could increase if the intervention were fully scaled.

What’s especially encouraging about this project is that no new technologies, seeds or infrastructure was involved–just basic science and a tremendous outreach campaign. Moreover, since the campaign increased profits it may continue to generate gains in the future even without further intervention as the practices spread. Repeated interventions will be necessary as climate changes, however. Information technology may makes this easier. China can be intimidating.

Researchers at the Reserve Bank of Australia estimate that house prices in major Australian cities are pushed well above the cost of production, including the land, by zoning regulations such as floor space index (video link) restrictions.

Zoning regulations provide benefits, but they also restrict housing supply and hence raise prices. This paper quantifies their importance by comparing prices to the marginal costs of supply at different points in time. For detached houses, marginal costs comprise the dwelling structure and the land that other home owners need to forego. Relative to our estimates of these costs, we find that, as of 2016, zoning raised detached house prices 73 per cent above marginal costs in Sydney, 69 per cent in Melbourne, 42 per cent in Brisbane and 54 per cent in Perth. Zoning has also raised the price of apartments well above the marginal cost of supply, especially in Sydney. We emphasise that this is not the amount that housing prices would fall in the absence of zoning. The effect of zoning has increased dramatically over the past two decades, likely due to existing restrictions binding more tightly as demand has risen.

Hat tip: Matt Yglesias.

A lot of psychological research has failed to replicate, throwing cold water on the entire field. “Grit” and the “growth mindset”, the two taglines of superstar researchers Angela Duckworth and Carol Dweck, checked all the boxes for predictive failure including the requisite TED talks (Duckworth, Dweck), best-selling popular books (Duckworth, Dweck) and genius awards and, to be sure, there has been lots of puffery about the “incredible potential” and “profound impact” of grit and the growth mindset. But, to their great credit, Duckworth and Dweck have taken the replication crisis to heart and have sought to address it. Working with a large team (PI David S Yeager), the authors have tested a growth mindset intervention in 65 randomly chosen schools with over 12,000 students representative of the United States grade 9 population.

Here is what is notable: The analyses were pre-registered, the data were collected by independent researchers and key parts of the model were analyzed by independent statisticians in a blinded dataset.

To achieve arms-length independence, a research firm not involved in designing the materials or study hypotheses drew the sample, recruited schools, facilitated treatment delivery, obtained administrative data, and cleaned and merged data. Data were processed blind to treatment status.

…A random sample of schools, rather than a convenience sample, meant that it represented the full array of the U.S. public educational contexts.

Data were analyzed following a pre-registered analysis plan (the so-called “preregistration challenge,” that was developed by an interdisciplinary team, including one external evaluator. All analyses were “intent to treat” (ITT); data were analyzed as long as students saw the first page of the randomized materials.

independent statisticians reproduced the key moderation findings by estimating a hierarchical, nonlinear Bayesian model using a blinded dataset that masked the identities of the variables, to further reduce the possibility of chance findings.

Ok, so what were the results?

Based on administrative records, 9th grade adolescents assigned to the growth mindset
intervention, as compared to the control activity, earned slightly higher GPAs in core classes at
the end of 9
th grade. On a 4-point grade metric (“A” = 4.0, “B” = 3.0, etc.), the average treatment
effect was 0.03 grade points,
SE = .01, N = 12,542 students, k = 65 schools, t = 3.09, P = .003.

In other words, a small, positive effect. But this small effect is coming from a small intervention, two online survey/interventions of 25 minutes each that could be easily scaled to the entire country or even worldwide. We have come a long way from the “mindset revolution” but who am I to discount a marginal revolution? Moreover, the average effect hides heterogeneity, the effect was bigger on the students who needed it most.

as expected, average effects were small because many students
are already doing well, do not have motivational issues, or are not in environments that
encourage or support growth-mindset behaviors. When we take account of such factors, more
noteworthy effects emerge. The improvements in the gateway outcome of 9
th grade GPA were
concentrated among adolescents who are at significant risk for compromised well-being and
economic welfare: those with lower levels of prior achievement attending relatively lower achieving schools. The finding that an intervention can redirect this adolescent outcome in this
sub-group, in under an hour, without training of teachers, and at scale (i.e. in a random sample
of nation’s schools), represents a significant advance.

Overall, this is a very impressive study and one that I suspect will be used to mark the beginning of the post-replication-crisis era.

The ending of the post-replication-crisis era also makes another trend clear–the future of social science will be even more hierarchical and unequal–future social science will be done by large, well-funded teams, run by superstar researchers at top universities. This study, for example, had 10 co-authors from multiple universities and probably cost well over a million dollars. The smaller the effect the bigger the team that will be needed to find it.

Addendum: A big meta-analysis out today also finds very small effects for growth mindset (correlation of growth mindset with achievement=.01) but the effects are probably real especially for academically high-risk students and low-SES students and perhaps they could be magnified by better interventions.

Hat tip: Stuart Richie.

The Push on Netflix is a deeply disturbing replication of the Milgram Experiment. The question it asks is whether someone can quickly be convinced to commit a murder? Spoiler alert: yes. British mentalist Derren Brown and a cast of confederates create an evil version of the Truman Show. By taking an individual from one seemingly minor moral deviation–labeling meat canapes as vegetarian–to another, to another, Brown puts people in a situation where by the end of one hour they are so emotionally disoriented and stressed that they will try to commit a murder to relieve their tension.

If you had asked me yesterday whether I thought it would be ok to run the Milgram experiment again, I would have said yes, as science. Today, I am not sure. What Brown does to these people for our entertainment (?) is disgusting. I feel complicit in having watched. Yes, I know, I am writing about it. I’m not sure what to make of that either.

As far as I can tell, the experiment is real. I’d be happier if it were fake but the results are consistent with previous Milgram replications. But if it is real did we then watch attempted murder? I am reminded of Leo Katz’s, Bad Acts and Guilty Minds. If a man fires a gun aiming to kill but the gun is defective is it attempted murder? Surely, yes. If thinking it a deadly poison a man adulterates a drink with sugar is it attempted murder? What if a sincere believer in voodoo tries to kill by sticking pins in a doll?

Aside from the legal issues, what Brown does to the participants is awful. How will they live the rest of their lives? Jordan Peterson says that you cannot be a good person until you know how much evil you contain within you. Well the people Brown experiments on know the evil that they contain but will they become better people? Or will they break? Brown doesn’t seem to care.

In some sense, the subjects have consented. Months earlier they applied to be on a show but they were told that they had been rejected. Perhaps you think the participants figured it out. You will have to judge for yourself but it all happens so quickly that I don’t think that is plausible. Moreover, if you figured it out wouldn’t you want to be the hero rather than the prison guard directing the Jews to the ovens?

Does The Push have any socially redeeming value? I hope so. Phillip Zimbardo of the famous Stanford Prison Experiment was so upset by his research that he started the Heroic Imagination Project, (I wrote about it here). The Heroic Imagination Project attempts to turn the issue around by asking what helps people to resist authority? And how can we train people under stress to draw on their heroic reserves? Netflix has shown us that the Heroic Imagination Project is sorely needed. Maybe next time Netflix can devote some of their considerable resources to helping us resist the push.

“There is not an epidemic of school shootings,” he said, adding that more kids are killed each year from pool drownings or bicycle accidents.

James Alan Fox, the Lipman Family Professor of Criminology, Law, and Public Policy at Northeastern.

School shootings are actually down since the 1990s (with a lot of variability). Fewer students are carrying weapons to school and fewer students report having easy access to guns (data here).

It’s been said that we live in an increasingly divided media universe but on many issues I think we live in an increasingly uniform media universe. Social media is so ubiquitous and the same things sell so widely that I suspect the collective consciousness is less fragmentary than in the past. Does anyone not know about Parkland? Contrary to common wisdom, mass shootings also occur in European countries. I suspect, however, that the Finnish media don’t cover German shootings as frequently as shootings in Florida are covered in Nebraska–as a result the larger the media-market the greater the extent of availability bias. In other words, the larger the media market the greater the over-estimation of rare but vivid events. (Someone should test this theory.)

I worry about turning schools into prisons and what kinds of citizens this will create. My letter to my son’s high school principal was sent before the recent shootings but I stand by it now more than ever:

Dear Principal _____,

Thank you for requesting feedback about the installation of interior cameras at the high school. I am against the use of cameras. I visited the school recently to pick up my son and it was like visiting a prison. A police car often sits outside the school and upon entry a security guard directs visitors to the main office where the visitor’s drivers license is scanned and information including date of birth is collected (is this information checked against other records and kept in a database for future reference? It’s unclear). The visitor is then photographed and issued a photo pass. I found the experience oppressive. Adding cameras will only add to the prison-like atmosphere. The response, of course, will be that these measures are necessary for “safety.” As with security measures at the airports I doubt that these measures increase actual safety, instead they are security theater, a play that we put on that looks like security but really is not.

Moreover, the truth is that American children have never been safer than they are today. Overall youth mortality (ages 5-14) has fallen from 60 per 100,000 in 1950 to 13.1 per 100,000 today (CDC, Vital Statistics). Yet we hide in gated communities, homes and schools as never before.

When we surround our students with security we are implicitly telling them that the world is dangerous; we are whispering in their ear, ‘be afraid, do not venture out, take no risks.’ When going to school requires police, security guards and cameras how can I encourage my child to travel to foreign countries, to seek new experiences, to meet people of different faiths, beliefs and backgrounds? When my child leaves school how will the atmosphere of fear that he has grown up in affect his view of the world and the choices he will make as a citizen in our democracy? School teaches more than words in books.

Yours sincerely,

Alex Tabarrok

An article in Wired has sparked controversy with its claim that Trump paid lower prices for its Facebook ads than Clinton:

During the run-up to the election, the Trump and Clinton campaigns bid ruthlessly for the same online real estate in front of the same swing-state voters. But because Trump used provocative content to stoke social media buzz, and he was better able to drive likes, comments, and shares than Clinton, his bids received a boost from Facebook’s click model, effectively winning him more media for less money. In essence, Clinton was paying Manhattan prices for the square footage on your smartphone’s screen, while Trump was paying Detroit prices.

The claim is plausible but although written by a Facebook expert it never really explains why Google and Facebook prices their ads in this way. The reason is what I call the “mesothelioma lawyer” problem. A click on an ad for a “mesothelioma lawyer” is extremely valuable because people who aren’t interested in hiring a mesothelioma lawyer are unlikely to click and those who do click are likely to become profitable clients. Thus, anyone searching for mesothelioma is likely to see an ad for a mesothelioma lawyer.

But suppose that Google or Facebook simply charge for ads by the click. Someone who searches for “funny hat video” isn’t likely to click on an ad for a mesothelioma lawyer but the people who do click are still likely to be very profitable to a mesothelioma lawyer. As a result, the mesothelioma lawyer can outbid the seller of funny hats for ads connected to “funny hat video” even though the search has nothing to do with mesothelioma.  If Google or Facebook only charged by the click it would be mesothelioma lawyer ads everywhere, all the time.

To avoid this problem, Google and Facebook calculate how many clicks or interactions your ad is likely to receive and they charge lower prices the greater the predicted number of clicks. As a result, sellers of funny hats get lower prices than mesothelioma lawyers for ads that pop up after the user watches a funny hat video and mesothelioma lawyers get lower prices than sellers of funny hats for ads that pop up after the user searches for information on mesothelioma. In the long run this system better targets ads to customers and thus maximizes the value of the platform to both advertisers and customers.

As the Wired piece eventually states this isn’t even new:

“I always wonder why people in politics act like this stuff is so mystical,” Brad Parscale, the leader of the Trump data effort, told reporters in late 2016. “It’s the same shit we use in commercial, just has fancier names.”

He’s absolutely right. None of this is even novel: It’s merely best practice for any smart Facebook advertiser.

Addendum: See also Hal Varian’s discussion of the underlying issues in the Online Advertising section of this paper.

I enjoyed Brian Wansink’s book Mindless Eating–it was well written and filled with creative experiments like the ever filling soup bowl. In the ten years since that time Wansink became not just a media start but an academic star with an h-index of 75 and over 24 thousand citations. In recent years, however, he has had to retract papers in the light of inconsistencies and questions about his data and statistics.

A Buzzfeed article, based in part on emails, now reveals that Wansink was running a brazen p-hacking factory:

The correspondence shows, for example, how Wansink coached Siğirci to knead the pizza data.

First, he wrote, she should break up the diners into all kinds of groups: “males, females, lunch goers, dinner goers, people sitting alone, people eating with groups of 2, people eating in groups of 2+, people who order alcohol, people who order soft drinks, people who sit close to buffet, people who sit far away, and so on…”

Then she should dig for statistical relationships between those groups and the rest of the data: “# pieces of pizza, # trips, fill level of plate, did they get dessert, did they order a drink, and so on…”

…“Work hard, squeeze some blood out of this rock, and we’ll see you soon.”…All four of the pizza papers were eventually retracted or corrected.

In essence, Wansink all but published a study finding green jelly beans cause acne. All hail XKCD.

India Lacks Formal, Salaried Jobs

by on February 24, 2018 at 7:25 am in Economics | Permalink

In the United States a majority of workers work for very large firms and only 10% are self-employed. In India hardly any workers work for large firms and 80-85% are self-employed, as I pointed out in my post India is a much more Entrepreneurial Society than the United States (and that’s a problem).

Data from a draft report by the World Bank summarized by LiveMint show that India has few formal, salaried jobs not just compared to a developed country like the United States but also compared to many other less developed countries around the world.

Solving this problem is one of India’s biggest challenges as the number of workers is increasing rapidly. Land reform and deregulation of labor markets including stopping own goals that discourage firms from growing and formalizing such as requiring absurdly “generous” maternity leave benefits is a first step.

Hat tip: Urbanomics.

Regulations that prevent land from being fully developed raise the price of housing. That’s true but land use regulations can also make some types of housing less expensive. In particular, Jaap Weel has a good post explaining how land regulations subsidize mansions.

Consider the buildings below: a mansion on a 1 acre lot in Atherton, and a 350 unit mixed use condo on a 1.6 acre lot 2 miles further up the peninsula in Redwood City. The mansion just sold for $6m. The condo building, when finished, will probably fetch hundreds of millions.

If it weren’t for Atherton’s zoning code, you’d never be able to buy that mansion for a mere $6m. A developer that wanted to tear it down and build condos could bid far more than that. But the zoning code mandates single-unit buildings with a floor area ratio below 18% on lots of at least 1 acre, so $6m it is. Quite the bargain.

In a market economy bidding tends to move resources from low-valued uses to high-valued uses. Regulations that prevent bidding freeze resources into low-valued uses–that’s bad for the resource owners and bad for society as the total value of production is reduced but it can be good for the consumers of low-valued uses.

Addendum: For more on floor area ratio regulations, see my video on skyscrapers and slums in Mumbai.

Spock’s Brain

by on February 18, 2018 at 7:32 am in Economics | Permalink

I take an inordinate amount of pleasure in this note from the Wikipedia entry on Spock’s Brain under Reception and Influence:

“The episode was referenced in Modern Principles: Microeconomics by Tyler Cowen and Alex Tabarrok of George Mason University as an example of how it is virtually impossible to have a command economy; in that not even Spock’s brain could run an economy.”

At left is the picture from Modern Principles; we also snuck in an oblique Simpson’s reference.

From Wikipedia I also learned that Phish has a song called Spock’s Brain, alas it is not about the difficulties of running a command economy.

Bulgaria has the fastest declining population in the world. From a peak of nearly 9 million around the time of the communist fall in 1990 Bulgaria’s population is 7 million today and projected to fall to around 5 million over the next generation. Entire villages have been depopulated, especially in the poorer Northern region.

A correspondent wrote me asking what to do. I responded what’s the problem? Of course, there are plenty of things one could do to make Bulgaria a richer and better place to live, some of which Bulgaria has been doing and some of which they have not. The more fundamental question, however, is why the number of a particular type of people located in a particular geographically proscribed area should be a measure of welfare?

Instead of focusing on Bulgaria let’s focus on Bulgarians. One of the reasons the population of Bulgaria has been falling is that Bulgarians have been leaving for better lives elsewhere in the European Union. Over one million Bulgarians live abroad. It is not always easy to move nor to stay in a village that is bereft of young people. But how fortunate is that those young people could move elsewhere. Instead of thinking of them just as Bulgarians lets think of them as citizens of the European Union. Problem solved. The EU population is increasing!

Is that a facile answer? Perhaps but note that in the United States great swaths of the country have seen declining populations since the 1930s or even earlier. We tend not to regard this as a big deal. In part because many of the areas with declining populations were small to begin with but also because we regard it as a good thing that Americans can move about the country. Indeed, because people have been free to move to opportunity the people remaining have not seen big declines in their standard of living. Ghosts are better than zombies.

Addendum: Bulgaria has some great beaches and historic sites at very reasonable prices!

What if I told you that there is a method of education which significantly raises achievement, has been shown to work for students of a wide range of abilities, races, and socio-economic levels and has been shown to be superior to other methods of instruction in hundreds of tests? Well, the method is Direct Instruction and I first told you about it in Heroes are Not Replicable. I am reminded of this by the just-published, The Effectiveness of Direct Instruction Curricula: A Meta-Analysis of a Half Century of Research which, based on an analysis of 328 studies using 413 study designs examining outcomes in reading, math, language, other academic subjects, and affective measures (such as self-esteem), concludes:

…Our results support earlier reviews of the DI effectiveness literature. The estimated effects were consistently positive. Most estimates would be considered medium to large using the criteria generally used in the psychological literature and substantially larger than the criterion of .25 typically used in education research (Tallmadge, 1977). Using the criteria recently suggested by Lipsey et al. (2012), 6 of the 10 baseline estimates and 8 of the 10 adjusted estimates in the reduced models would be considered huge. All but one of the remaining six estimates would be considered large. Only 1 of the 20 estimates, although positive, might be seen as educationally insignificant.

…The strong positive results were similar across the 50 years of data; in articles, dissertations, and gray literature; across different types of research designs, assessments, outcome measures, and methods of calculating effects; across different types of samples and locales, student poverty status, race-ethnicity, at-risk status, and grade; across subjects and programs; after the intervention ceased; with researchers or teachers delivering the intervention; with experimental or usual comparison programs; and when other analytic methods, a broader sample, or other control variables were used.

It is very unusual to see an educational method successfully replicate across such a long period of time and across so many different margins.

Direct Instruction was pioneered by Siegfried Engelmann in the 1960s and is a scientific approach to teaching. First, a skill such as reading or subtraction is broken down into simple components, then a method to teach that component is developed and tested in lab and field. The method must be explicitly codified and when used must be free of vagueness so students are reliably led to the correct interpretation. Materials, methods and scripts are then produced for teachers to follow very closely. Students are ability not age-grouped and no student advances before mastery. The lessons are fast-paced and feedback and assessment are quick. You can get an idea of how it works in the classroom in this Thales Academy promotional video. Here is a math lesson on counting. It looks odd but it works.

Even though Direct Instruction has been shown to work in hundreds of tests it is not widely used. It’s almost as if education is not about educating.

Some people object that DI is like mass-production. This is a feature not a bug. Mass-production is one of the few ways yet discovered to produce quality on a mass scale. Any method will probably work if a heroic teacher puts in enough blood, sweat and tears but those methods don’t scale. DI scales when used by mortals which is why it consistently beats other methods in large scale tests.

Many teachers don’t like DI when first exposed to it because it requires teacher training and discipline. Teachers are not free to make up their own lesson plans. But why should they be? Lesson plans should be developed by teams of cognitive psychologists, educational researchers and other experts who test them using randomized controlled trials; not made up by amateurs who are subject to small-sample and confirmation bias. Contrary to the critics, however, DI does leave room for teachers to be creative. Actors also follow a script but some are much better than others. Instructors who use DI enjoy being effective.

Quoting the authors of the meta-analysis:

Many current curriculum recommendations, such as those included within the Common Core, promote student-led and inquiry-based approaches with substantial ambiguity in instructional practices. The strong pattern of results presented in this article, appearing across all subject matters, student populations, settings, and age levels, should, at the least, imply a need for serious examination and reconsideration of these recommendations (see also Engelmann, 2014a; Morgan, Farkas, & Maczuga, 2015; Zhang, 2016). It is clear that students make sense of and interpret the information that they are given—but their learning is enhanced only when the information presented is explicit, logically organized, and clearly sequenced. To do anything less shirks the responsibility of effective instruction.
Hat tip: Robert Pondiscio at Education Next.