Improving But Not Learning by Doing

In the excellent The Secret of Our Success Joe Henrich gives many examples of complex technological products and practices which were not the product of intelligence but rather of many, small, poorly understood improvements that were transmitted culturally down the generations. Derex et al. offer an ingenious experimental test of the cultural generation hypothesis.

Participants in the experiment were presented with a wheel with some weights that could be moved along four axis and they were asked to place the weights to maximize the speed at which the wheel moved down a track. The problem isn’t trivial since an optimal solution requires placing the weights in different spots to take advantage of both inertial and potential energy. Participants were organized into chains of five. Each participant was given 5 trials. The weight configuration and the results of the last two trials were passed on to the next person in the chain. Thus, people farther down the chain potentially “inherit” more cultural knowledge.

What were the results? First, the average wheel speed increased down the generations from an average of 123.6 m/h for the final trial of the first generation/participant to 145.7 m/h by the last trial of the fifth generation. The researchers also tested whether participants improved their understanding of the causes of wheel speed by asking them to predict which of a series of wheel configurations would spin the fastest. If the faster speed of the fifth generation reflected learning by doing we would expect the fifth generation to make better predictions. In fact, there was no learning over time. Technology improved, understanding did not.

The authors then did an especially clever test. They allowed each generation/participant to leave the next generation a “theory” of wheel speed. Did this “book learning” speed up the evolution of technology? It did not. Moreover, theory transmission didn’t even result in much learning! Indeed, in some respects theories actually reduced learning because people who inherited a theory tended to believe it to the exclusion of other theories and, as a result, they reduced their exploration of the design space.

Of the 56 participants who received a theory… 15 received an inertia-related theory, 17 received an energy-related theory, 6 received a full theory and 18 received diverse, irrelevant theories

…inherited theories strongly affected participant’s understanding of the wheel system. Participants who did not inherit any theory (“Configurations” treatment) scored similarly (and better than chance) on questions about inertia and questions about energy (Fig. 3I). In comparison, participants who inherited an inertia- or energy- related theory showed skewed understanding patterns. Inheriting an inertia-related theory increased their understanding of inertia, but decreased their understanding of energy; symmetrically, inheriting an energy-related theory increased their understanding of energy, but decreased their understanding about inertia. One explanation for this pattern is that inheriting a unidimensional theory makes individuals focus on the effect of one parameter while blinding them to the effects of others. However, participants’ understanding may also result from different exploration patterns. For instance, participants who received an inertia-related theory mainly produced balanced wheels (Fig. 3F), which could have prevented them from observing the effect of varying the position of the wheel’s center of mass.

…These results suggest that the understanding patterns observed in participants who received unidimensional theories is likely the result of the canalizing effect of theory transmission on exploration. Note that in the present case, this canalizing effect is performance-neutral: with our 2-dimensional problem, better understanding of one dimension and worse understanding of one dimension simply compensate each other. For a many-dimensional problem, though, better understanding of one dimension is unlikely to compensate for worse understanding of all the others.

One aspect of knowledge transmission which is more difficult to study is the role of the genius. Cultural generation can get stuck in local optima. Only the genius can see over the valley to the mountain. The occasional genius may have been important even in knowledge generation in the pre-science era. In addition, these kinds of cultural evolution processes work best when feedback is quick and clear. Lengthen the time between input and output and all bets are off. Still this peculiar experiment illustrates how much cultural transmission can achieve and how theory can so dominate our thinking that it reduces vital experimentation.


What Tabarrok is describing is the phenomenon explored by Nobel winner Edmund Phelps. Here is the excellent Tim Taylor describing the Phelps' argument (with a link to the Phelps' book and article):

I think it's more the Shewhart Cycle that Deming pushed.

Stewhart's work led to Plan-Do-Study-Act (PDSA)., which Deming opposed as a misunderstanding of Shewhart.

In my experience, "applying theory" means applying one theory to a system dominated by a hundred theories.

For example, the theory is a faster CPU will make your program run faster. Then, parallel computers and a new coding algorith to use 10 parallel cpus will make the program 10 times faster. Cray demolished those theories, pointing out I/O is the biggest problem, but then listing at least 3 distinct kinds of I/O.

Cray listed rules of thumb. A theory might start you down a computer design path, but you must at every fork in the decision tree balance tradeoffs between the effects of each rule of thumb.

It's like finding the perfect chess game "theory" that always wins.

I remembered the Taylor blog post from almost a year and a half ago because the Phelps' argument comports with my view that progress and prosperity won't be found by the boy wonders ensconced in a laboratory in Silicon Valley in search of the next big thing. What Tabarrok likes are phenomena that can be expressed mathematically, with data and data mining (i.e., the Silicon Valley way to progress and prosperity).

small steps....

great issues altogether, you just received a brand new reader.

What may you recommend about your post that you just made some days ago?
Any sure?

"progress and prosperity won't be found by the boy wonders ensconced in a laboratory in Silicon Valley"

lol, another beautiful "he said on the internet" moment!

Sounds like the Cubs Scouts pinewood derby. You can add weight to bring the car up to 3 oz (If I remember right). I always front loaded it so it would transition from the flat surface to downhill the soonest. Not sure if it ever helped.

The key is actually to backload it to give you the highest potential energy

It starts out flat and then rolls down a hill. I'd guess it has the same potential energy either way. My thought was getting the weight onto the hill sooner.

I think you guys are talking about different styles of tracks. The most popular style starts with a downhill portion, then transitions to a flat surface for the finish (So opposite of what you seem to describe). In that style, you always want the weight as far back as you can without getting the center of gravity past the rear axles. That give you the highest potential energy because of the added height of the weight distribution.

(P.S. Yes, my sons took first place in their pinewood derbies, why do you ask? Of course, we then donated knowledge to everyone else once they were no longer eligible, which fits the study a little bit.)

Ummm... Exactly what makes your car move on the flat surface?

It's flat at the top, but there a lever that raises the track to dump them off. Really it just raises the back so it's not longer level. They roll along a short flat part, then hit downhill. Ours anyways. Google images says it must have been uncommon.

I did the Pinewood Derby once as a Cub Scout, and the thing I noticed more than anything else was that the track used was not uniform. One lane had a significant advantage - though I didn’t have the opportunity to investigate how/why.

... so knowledge/experience can actually be communicated among humans... and some humans are smarter than others. yawn

Maybe a genius is just someone who is really lucky his pet theory is actually correct?

Perhaps genius is the trait of not letting, even subconsciously, oneself become constrained by preconceived theories. We damage a lot of students by teaching simplified models as truth and not emphasizing how they quickly fall apart in the real world.

Or perhaps it is just working harder for longer without most noticing:

"One day about three or four years after I joined, I discovered that John Tukey was slightly younger than I was. John was a genius and I clearly was not. Well I went storming into Bode's office and said, "How can anybody my age know as much as John Tukey does?" He leaned back in his chair, put his hands behind his head, grinned slightly, and said, "You would be surprised Hamming, how much you would know if you worked as hard as he did that many years." I simply slunk out of the office!"

--Richard Hamming: You and Your Research
Talk at Bellcore, 7 March 1986

Most people are simply bad at theory; i.e. humans are not good at this. Thus, setting up successive filters of theory all but guarantees that the process will generate decreasing returns. Consider it this way: the filter constrains your parameter space on potential paths and potential configurations both. Each constraint will tend to shave away another piece left in by the last without necessarily substituting anything useful. The most likely result of such a chain is a lowest common denominator technique at the end.

Furthermore, decision trees are a lousy way to either perform pattern recognition or functionally model anything. This is why machines are dumb: they are presently designed to von Neumann process actual data to entropic death rather than capture the life in it. You may get a robust keyhole view that is accurate for what it registers, but don't count on it being accurate for the context of what it registers.

Competence is finding the established pattern in the data or parameter space. Intelligence is finding a substantive pattern in the data or parameter space. Talent is finding a representative pattern in the data or parameter space. Genius is finding THE representative pattern in the data or parameter space (and ideally a beautiful representative pattern at that).

This is actually quite simple if one does theory or studies creativity. How many people actually do?

Could they run the experiment with groups of different inteligence and knowledge?

Eg very clever stem but non physicists, physicists, very clever arts/social

Unlikely to prove anything but results might be interesting

So what discount rate should we put on theory? Seems like now that is often either 0 or 100%.

@dan in philly, that is probably a true statement for some historical and current genius. But I think the test there is the "are they one hit wonders" to borrow from the music world.

Another view is perhaps genius, in this context, is more about not taking the handed down wisdom as fact but always keeping a critical and questioning mind.

"Only the genius can see over the valley to the mountain": personally I've always found it easy to look over a valley to see a mountain.

Maybe I've missed the point of the metaphor. Or maybe it's a lousy metaphor.

Well, no reason to help Prof. Tabarrok out with any theories - it would likely just get in the way of his doing.

I read it as "only a genius can see over the valley to the mountain to a deeper valley". With valleys being the local optima

Geniuses tend to understand an analogy even if its worded poorly.

Non-genius can only see the sight in front of them and cannot see the bigger picture across the valley.

If we use the term genius to refer to those with some rare insight into a process that the rest simply didn't understand, then genius was certainly important historically. The Chinese for instance invented rag paper centuries before Europe and even after its introduction there, the new paper could not be replicated or backwards engineered in the West for a couple of centuries.
The leading Western Universities insisted on Roman numerals for mathematics even though Italian merchants were showing superior benefits of using the new fangled Indian/Arabic numerals.
And of course, Archimedes was far ahead of his time.

A successive approximation process can sometimes work?! Why, who'd a thunk it?

BUT, successive approximation fails if the path to optimization requires an incremental step that is larger than the process can handle.

“.. examples of complex technological products and practices which were not the product of intelligence but rather of many, small, poorly understood improvements that were transmitted culturally down the generations. “

I don’t think this means what you say it is. Take the humans out of the experiment and change the weights randomly and see if it improves over time. I don’t think it will.

But the individuals involved in later trials had no more insight into what was going on than those involved in the first trials, according to the post.

That would appear to be easily explainable by the fact that they prevented each generation from learning anything useful about the insight gained by the previous generation, other than the previous generation's ending configuration. That limitation means that future generations could not learn about the direction of optimization, or any features learned about the optimization landscape by the previous generations.

Each generation had to independently learn what the local optimization landscape looked like (with far too few trials to really learn anything), so the fact that each generation learned roughly the same thing about the process is hardly surprising.

Genius is probably not the right word.

There are certainly intellectual sluggers, long ball hitters, but defining them by outcome is kind of circular.

Imo the macaque is a datum for the 'genius' theory!

Good data point and fun story. Here's to Imo the macaque genius.

Or in another idiom, Imo got a brainwave.

So what's the optimal wheel configuration?!

Well, when I get new tires, they always balance the wheel on a single point in the center. If an unbalanced wheel were better, then why wouldn't they make it unbalanced?

If the wheel has a standing start there might be some trick with initial acceleration ..

Joshua, I think that from Alex's summary we don't have enough input to solve the problem, We would need to know the weight of the weights relative to the weight of the wheel, and also probably the radius of the wheel relative to the total descent of the slope.

Anyway, it is a very interesting experiment. The point is not that we can more or less solve a problem by successive approximation, which would be trivial indeed, but that it pays for the humans of the next generation to respect what have done their elders and to try to improve on it by marginal changes only, even if they don't understand why the elders acted this way (and they didn't understand it either). So it seems to me as an experience supporting the motto of this blog, "small changes to a much better world", and more generally a relatively conservative approach to problems. It makes it even more interesting that it comes from Alex, not Tyler, because Alex seems in general less "marginalist", more revolutionary, than Tyler.

Of course, we can argue that this experience is not representative of the real optimizations problems arising in real life.

Hooray for economically redundant amounts of doing! Just as deadweight loss from taxation may be a bargain when paying for public schooling, some inefficiency may be worth accepting to increase the cognitive surface area exposed to promising frontiers

In computer science, the progression of programming languages seems a lot like this progression of wheel balancing. In the beginning no one had any idea what a programming language should look like. Someone took a stab at it. People used that language, and then tried again. And that has been cycle ever since. With no end in sight.

It's kind of sad that an infrastructure problem locks us in on operating systems and we don't have the same level of iteration.

As Rob Pike called it in 2000

Interesting. But is it true? I don't know much about the history of programmatic, but from what I know it seems that it is a more chaotic evolution, with strong revolutions. For instance when I was a kind language became more and more structured, until the object-oriented programmatic became very important and have ad C++ where the fashionable language, and then suddenly Python came, very un-structured, and became the talk of the street.

There are certainly trends in any decade, but at the same time it is interesting to me that so many of these things claim to be "general purpose programming languages."

You get the same sort of libraries rewritten in both C++ and Python.

If say you wanted to access and manipulate a corporate database you could do that in either C++ or Python, or many many others.

Very interesting if it holds up. Something mysterious about where knowledge resides.

Clever experiment. Because the result is surprising we expect the later groups to inherit a better understanding with each iteration. Once the actual result is told to us we may say it’s obvious but no it was not.

This story is B.S. There's nothing you can do to make the wheel go faster, unless the wheel is unbalanced and creating friction.

AlexT doesn't know his physics. The closest story I could find is this one:

Which simply states adding weight to a wheel makes it harder to accelerate (inertia), as is well known. But making it "go faster" by sliding weights up and down the spokes is like perpetual motion, not going to happen.

Any schoolboy who studied Holiday & Resnick's "Physics 101" textbook knows this...

By the way, if you study technology, or read any of the excellent books by economic historian Vaclav Smil, you'll see that indeed to discover a new phenomena, like flying a heavier-than-air machine (Wright brothers) can be done without theory (bicycle mechanics)--though I'll point out the Wright brothers actually studied theory including how birds fly--but to perfect the invention, you need to turn it over to engineers. Numerous examples. Wright Flyer vs Boeing 747 is the easiest to visualize but every technology is the same. Tinkering does not cut it. Tinkering in a wind tunnel and understanding Reynold's number does.

Bonus trivia: creating friction in front of a round speeding bullet makes it go faster. Laminar flow is your enemy. Counter-intuitive but that's what theory predicts.

The objective is to "maximize the speed at which the wheel moved down a track", so I understand this as the wheel starting at the top of a ramp. Putting the masses closest to the centre of rotation minimises rotational inertia, so the wheel will accelerate at a greater rate; but moving the weights upwards maximises potential energy with respect to the bottom of the ramp.

If the image in AT's post is the starting configuration, my gut says that moving the two masses on the vertical spokes as high as possible while putting the two masses on the horizontal spokes as close to the hub as possible gives the optimal solution.

I concur on your proposed solution.

It doesn't take physics 101 to understand this, I actually did this experiment in my elementary school science class nearly forty years ago. Rather than calling "BS" you might consider reading the paper and trying to understand how the wheel is set up. You might even try it out yourself! It's not hard to do.

On the plus side, there's a certain pleasure in seeing someone expose themselves as a fool in such an obvious fashion.

Of course, participants increased their understanding if the wheel system improved. The improvements were not random. This result hinges on what the researcher is going to count as "theory" and "theory transmission." So, what does this prove? That people can do tasks competently that they cannot adequately describe. We knew that already. That certain types of descriptions about how to do a task (call those theories if you want), do not transmit practical knowledge very well. We already knew that. Come on, training in an introductory physics course will not make a student into a competent engineer even if the student really understood everything that they were taught. This study proves nothing useful.

After reading this I can't help but ask this tongue in cheek question -

Are you suggesting that if economists are less exposed to theory they are more likely to come up better economic ideas in terms of how they actually work or the results they produce in real life?

Actually the Stanford-Binet was the original IQ test but the WISC was developed to capture multidimensional aspects of cognition, expanding to concepts of sensation, perception and attention. It's totally possible that these areas could be culturally inherited or defined and could confound any experimental analysis designed to detect "genius", whatever that means.

This seems to validate something I keep in mind from the Conversations with Tyler discussion between Nassim Taleb and Bryan Caplan.

Taleb: "But the idea — I started writing papers — the idea of having to start by theory and ending up with practice doesn’t work.

"You should try practice, then theory."

"The secret of our success" is a great book. The main point is that the advantage of our bigger brain is not intelligence/enhanced problem solving but rather the ability to learn from each other, teach and cultural transmission ( the large body of practices, techniques,heuristics, tools motivations, values and beliefs acquired while growing up through learning). It seems that apes unlike humans don't teach each other hardly anything and every generation starts anew , there is no cultural transmission.

There's another explanation besides "inventions come from the slow accumulation of improvements without anyone understanding it." Suppose that some fixed fraction of "see the right answer." That is, they get to the trial, and on the first or second try, they realize the exact positions for where the weights should be, given some understanding of the theory. They leave that solution in their last trial, and no one thereafter will do any worse. (If they fail to come up with something better, they'll just revert to that solution). This is consistent with people performing equivalently on the theory questions regardless of at which number they are. This is also consistent with runtimes being better at the end, since it's more likely you get someone who "sees it" in any one of the five episodes than it is that you get this person in the first episode.

Now where have I heard something like that recently?
"Rule-swarm attacks can outdo deep reasoning"

Comments for this post are closed