Results for “Tests” 775 found
How random are RCTs?
From Hunt Allcott and Sendhil Mullainathan, “External Validity and Partner Selection Bias”:
Program evaluation often involves generalizing internally-valid site-specific estimates to a different target population. While many analyses have tested the assumptions required for internal validity (e.g. LaLonde 1986), there has been little empirical assessment of external validity in any context, because identical treatments are rarely evaluated in multiple sites. This paper examines a remarkable series of 14 energy conservation field experiments run by a company called Opower, involving 550,000 households in different cities across the U.S. We first show that the site-specific treatment effect heterogeneity is both statistically and economically significant. We then show that Opower partners are selected on partner-level characteristics that are correlated with the treatment effect. This “partner selection bias” implies that replications with additional partners have not given an unbiased estimate of the distribution of treatment effects in non-partner sites. We augment these results in a different context by showing that partner microfinance institutions (MFIs) that carry out randomized experiments are selected on observable characteristics from the global pool of MFIs. Finally, we propose two simple suggestive tests of external validity that can be used in the absence of data from many sites: comparison of observable sample and target site characteristics and an F-test of heterogeneous treatment effects across “sub-sites” within a site.
The gated paper is here, ungated here. For the pointer I thank @JustinSandefur.
Fox and Mitchum on the Flynn Effect and how it works
James R. Flynn recommends this paper, by Fox and Mitchum, in his new book:
Secular gains in intelligence test scores have perplexed researchers since they were documented by Flynn (1984, 1987). Gains are most pronounced on abstract, so-called culture-free tests, prompting Flynn (2007) to attribute them to problem solving skills availed by scientifically advanced cultures. We propose that recent-born individuals have adopted an approach to analogy that enables them to infer higher-level relations requiring roles that are not intrinsic to the objects that constitute initial representations of items. This proposal is translated into item-specific predictions about differences between cohorts in pass rates and item-response patterns on the Raven’s Matrices, a seemingly culture-free test that registers the largest Flynn effect. Consistent with predictions, archival data reveal that individuals born around 1940 are less able to map objects at higher levels of relational abstraction than individuals born around 1990. Polytomous Rasch models verify predicted violations of measurement invariance as raw scores are found to underestimate the number of analogical rules inferred by members of the earlier cohort relative to members of the later cohort who achieve the same overall score. The work provides a plausible cognitive account of the Flynn effect, furthers understanding of the cognition of matrix reasoning, and underscores the need to consider how test-takers select item responses.
The paper is here (pdf).
Introducing MRUniversity (spread the word)
That’s Marginal Revolution University, MRU, or I suppose to some “Mister” University.
We think education should be better, cheaper, and easier to access. So we decided to take matters into our own hands and create a new online education platform toward those ends. We have decided to do more to communicate our personal vision of economics to you and to the broader world.
You can visit www.MRUniversity.com here. There you can sign up for information about our first course, Development Economics, which is described by Alex below.
Here are a few of the principles behind MR University:
1. The product is free (like this blog), and we offer more material in less time.
2. Most of our videos are short, so you can view and listen between tasks, rather than needing to schedule time for them. The average video is five minutes, twenty-eight seconds long. When needed, more videos are used to explain complex topics.
3. No talking heads and no long, boring lectures. We have tried to reconceptualize every aspect of the educational experience to be friendly to the on-line world.
4. It is low bandwidth and mobile-friendly. No ads.
5. We offer tests and quizzes.
6. We have plans to subtitle the videos in major languages. Our reach will be global, and in doing so we are building upon the global emphasis of our home institution, George Mason University.
7. We invite users to submit content.
8. It is a flexible learning module. It is not a “MOOC” per se, although it can be used to create a MOOC, namely a massive, open on-line course.
9. It is designed to grow rapidly and flexibly, absorbing new content in modular fashion — note the beehive structure to our logo. But we are starting with plenty of material.
10. We are pleased to announce that our first course will begin on October 1.
Please help spread the word via tweet, facebook and post and of course please join us at MRU.
Assorted links
Real Real Wages
The workers at Brazil’s central bank are on strike today, demanding a 23 percent pay increase to compensate for inflation that’s occurred since June 2008.
As reported by Matt Yglesias based on this article.
Assorted links
1. Tim Worstall has a Kickstarter book project.
2. How restaurants talk about their customers.
3. Can an entire Japanese firm switch to the English language? (also a story of flexible nominal wages)
4. The influence of Latin America on Tagalog (article is in Spanish), and I was wondering where all that Manila rain was going to go.
A password so secret even you don’t know it
Researchers from Stanford, Northwestern, and SRI have a new paper laying out a crafty little solution: Make people memorize passwords that they don’t know. If you don’t know your password, you can’t tell it to anyone.
Huh? How can you have a password you don’t know?
Perhaps the best analogy is the playing of a song on a musical instrument. If you’ve ever memorized a piece of music, you know that, given the instrument, you could play it no problem. The piece is in your muscle memory. But given a staff sheet and a pencil, writing it out would be a challenge. You would need to thump it out with your fingers and write down your observations, but you don’t just know the order of the notes.
…The catch is that unlike a piece of music, for which you’ve memorized a sequence that someone could, perhaps, record if they watched you tap it out enough times, this system then “tests” whether you are the real you, by having you “play” all sorts of strands, with the ones you’ve practiced mixed in. Only someone who has received the training will play their own sequences more smoothly and rapidly. “A performance gap that is substantially different from the one obtained after training indicates an attack,” the authors explain.
In effect, the software is creating a code by which you can say, I am who I say I am, and the computer recognizes it. Of course, all passwords convey that to some extent, this one is just much, much harder to pass on.
Assorted links
Barriers to entry cocoa bean counter edition
Becoming a cocoa beans grader is about four times harder than passing the New York State Bar exam, judging by a comparison of the two tests’ pass rates.
Candidates must correctly identify defects in beans such as mold, infestation from insects and cocoa that is “smoky or hammy”—a sign that the beans have been dried over a fire, not in the sun. Since beans easily absorb odors, the fire can give the beans a smoky flavor. In another section, they must identify the origin of various beans, most of which look identical to the layperson’s eye.
David Morales, one of the three men huddled over the beans in the ICE’s grading room, says he failed twice before he passed the exam two years ago. “I was studying for it, but not enough,” said the 37-year-old Bronx native, who noted the origin section tripped him up.
Is it a public sector monopoly, private sector monopoly, or some combination of both? Or is it just really hard to do well, requiring individuals of truly specialized abilities and with a lot of value at stake?:
The problem is that there are only 24 certified graders for ICE and the exchange is concerned that retirement and old age will deprive it of a crucial cog in its commodity-trading machine.
In an effort to “keep the talent pure and fresh,” the exchange is offering its licensing exam in October for the first time in two years, said Valerie Colaizzo, managing director of commodities operations for the exchange.
The full article is here, hat tip goes to the excellent Daniel Lippman, note by the way that Daniel’s most recent piece is here.
A fourth and hybrid perspective on the future of on-line education
In a very good blog post, Bryan Caplan lays out three competing perspectives. But he leaves out a fourth:
Select groups, such as adult continuing education, military officers on ships, precocious 12-year-olds, or perhaps middle class students in Kenya who can’t get the real product, will follow an exclusively on-line model. But most students will not, at least not in the United States. College still has considerable consumption value, fraternities improve your job prospects, instructors help motivate, and face-to-face contact imprints a lot of learning on our minds. Still, there is far too much duplication of lectures and universities are being squeezed by personnel costs. State governments face rising Medicaid costs and 78 percent or so of students are in state systems. Lecture duplication will be significantly reduced, and instructional time will be spent…instructing…rather than repeating canned lectures ad nauseum. Imagine that ten years from now one-third of all lectures are delivered on-line in one manner or another, perhaps with some later in person commentary. Students may watch those lectures with an instructional aide present to address questions or to show them how to press the “Play” button. There will be no need for employers to fundamentally change which sources they respect for personnel certification, although possibly some upstarts will arise in corners of the market where quality can be measured by tests.
You will find two critiques of my views on on-line education here, and here, but neither represents my views correctly. They all take on-line education to be an all-or-nothing prospect.
At the end of his post Bryan writes:
* When I talk about “online education,” I don’t just mean students at existing brick-and-mortar colleges taking some classes from their dorm rooms. I mean students enrolling in virtual colleges instead of physical colleges.
I would say he is defining away the most likely model, namely a hybrid model which has a significant on-line component.
Libertarianism and the Workplace II
Over at Crooked Timber Chris Bertram, Corey Robin and Alex Gourevitch are upset about what they call lack of freedom in the workplace. (Tyler offers his excellent comments immediately below.) They give a grab bag of peculiar examples such as workers can be fired for donating a kidney to their boss, carrying on extramarital affairs, participating in group sex at home, cross-dressing, and more. They seem especially incensed that workers can be commanded to pee or be forbidden to pee. (I will put aside that mandatory drug tests are greatly encouraged by the war on drugs.)
In other words, the CTrs have discovered that the most basic US employment law is at-will employment which means that workers can be fired for just about any reason, outside of a few protected categories such as race, sex, and age. Simply put, an employer can fire you if she doesn’t like you. This is a surprise? No one has a right to a job, a job is an agreement to exchange services for cash and compensation and both parties must agree to the exchange to make it legitimate.
The CTrs do not adequately acknowledge that workers have the same rights as employers. Workers can quit for any reason and they can refuse to work for any employer. If you don’t like the politics of Monsanto, or the NRA or Georgia-Pacific you don’t have to work or even apply for those jobs. Indeed, workers have more rights than employers since workers are not subject to anti-discrimination law; that is, employers are prohibited from discriminating against African American workers but workers are not prohibited from discriminating against African American employers.
If you think that the freedom to quit is without value bear in mind that under feudalism and into the early 19th century in the U.S. and a bit later in Britain employers and even potential employers could prevent workers from quitting and from moving. The freedom to quit was hard won. We should not disparage the liberation brought by a free market in labor.
Turning to the economics, the CTrs are so outraged by an employer’s legal possibilities that they fail to notice that most employers do not in fact fire their workers for having extra-marital affairs. Why not? The reason is that these rights are often more valuable to the employee than to the employer and thus both employee and employer can be made better off if the employee keeps the rights. If the employer values the right more than employee then the employer buys the right with a higher wage. If the employee values the right more than the employer then the employee retains the right at an otherwise lower wage. The employer gets the right only when the employer pays. The same thing is true of control rights, residual rights not explicitly noted in the contract (because of complexity and unforeseen events). The employer buys these rights from the worker when doing so maximizes the total value of the exchange. This is not to say that abuses do not occur, they do, as in all relationships and on both sides, but the CTrs lump abuses and mutually profitable exchanges together–that’s dangerous because in regulating abuses it is very easy to do away with mutually profitable exchanges.
The greater the productivity of workers and the higher their incomes the less workers will be willing to sell rights for higher wages (i.e. the more willing they will be to pay for better working conditions with lower wages). Workers gain more autonomy as they and their society become more productive. Thus, the best protector of worker autonomy is high productivity and economic growth. (The best protector not the only protector, unions can also serve a useful purpose in this regard as can shareholders and human resource departments.)
If the CTrs were merely arguing for greater economic growth there would be little with which to argue –who doesn’t want bigger televisions and better working conditions? The CTrs, however, confuse wealth and political freedom. Bigger televisions don’t make you more free and neither do better working conditions, even though both goods are desirable.
A job is an exchange with mutual consent and benefits on both sides of the bargain. The freedom is in the right to exchange not in the price at which the exchange occurs. A worker who is paid for 8 hours of work is not a serf 1/3rd of the day. We all sell our labor and we would all like to sell at a higher price but that does not make any of us serfs. From the minimum wage waiter to the highly-paid sports superstar there is dignity in work freely chosen.
To understand freedom and true coercion let us remember that American workers have the freedom to bargain and exchange with American employers, a freedom that gun, barbed wire and electrified fence deny to many millions of less fortunate workers from around the world.
“Star Wars: The Empirics Strike Back”
That is a new paper by Abel Brodeur, Mathias Lé, Marc Sangnier, and Yanos Zylberberg:
Abstract:
Journals favor rejections of the null hypothesis. This selection upon results may distort the behavior of researchers. Using 50,000 tests published between 2005 and 2011 in the AER, JPE and QJE, we identify a residual in the distribution of tests that cannot be explained by selection. The distribution of p-values exhibits a camel shape with abundant p-values above .25, a valley between .25 and .10 and a bump slightly under .05. Missing tests are those which would have been accepted but close to being rejected (p-values between .25 and .10). We show that this pattern corresponds to a shift in the distribution of p-values: between 10% and 20% of marginally rejected tests are misallocated. Our interpretation is that researchers might be tempted to inflate the value of their tests by choosing the specification that provides the highest statistics. Note that Inflation is larger in articles where stars are used in order to highlight statistical significance and lower in articles with theoretical models.
For the pointer I thank Michelle Dawson.
Addendum: Here is related commentary from Mark Thoma.
Assorted links
Did the debt ceiling dispute hurt the economy?
Matt offers a summary with links:
I was hoping to do a followup piece on that point, but Betsey Stevenson and Justin Wolfers did a great one for Bloomberg yesterday. The basic point is that we had three of the worst months for job growth of the entire recovery while the debate played out and consumer confidence took a sharp but temporary tumble. This was, in other words, just about the only time from the stress tests until the present when you seemed to see a real sense of panic and uncertainty afflicting the economy. The whole rest of the period has been pretty bad anyway, but the debt ceiling debate really dealt us a blow.
Why is unemployment so high in South Africa?
Ryan Cooper writes to me:
Hey Tyler, possible blog post topic: I’m wondering how you would explain the situation in South Africa (or other similar countries) with stupendous persistent unemployment–SA has been above 20% since 2000: http://www.tradingeconomics.com/south-africa/unemployment-rate
A few factors I imagine are important:
1) The education system is totally broken in a lot of places. As in, 12th graders can neither read nor write in any language nor figure out 3×3 in their heads.
2) Unions are crazy strong, and have been driving up wages like gangbusters, particularly in the public sector.
3) Minimum wage laws are stringent and have actually led to worker protests: http://www.nytimes.com/2010/09/27/world/africa/27safrica.html?pagewanted=all
4) Inflation hasn’t been TOO bad recently (~6%), but has seen spikes to almost 14 percent not long ago: http://www.tradingeconomics.com/south-africa/inflation-cpi
5) There’s a highly developed sector. On average, whites are far richer than blacks.6) Crime and inequality are incredibly bad.
7) The ANC has won every election in a landslide and is strongly allied with the unions.So how does it tie together? Lots of poorly-educated ZMP layabouts? Wages too high to start sweatshop-style development? Razor wire + electric fence + security guard costs deterring investment? The results of generations of systematic oppression and denial of education? All of the above, plus some?
Just trying to iron out a coherent story. I was a Peace Corps volunteer for two years there and I’m slowly building up my economics knowledge; this question has always fascinated me.