In front of you is a sheet of paper with a list of five-word sets. I want you to make a grammatical four-word sentence as quickly as possible out of each set. It’s called a scrambled-sentence test. Ready?
- him was worried she always
- are from Florida oranges temperature
- ball the throw toss silently
- shoes give replace old the
- he observes occasionally people watches
- be will sweat lonely they
- sky the seamless gray is
- should not withdraw forgetful we
- us bingo sing play let
- sunlight makes temperature wrinkle raisins
The students were then sent to do another test in an office down the hall. Unbeknownst to them, walking the hall was the real experiment. Scattered in the sentences above are words like ”worried,” “Florida,” “old,” “lonely,” “gray,” “bingo,” and “wrinkle.” Bargh reported that students who had been primed with these words took significantly longer to walk down the hall than those not primed with the “old” words.
In the original study there were only 60 participants and the subjects were timed with a stopwatch. A new paper doubles the sample size and uses more accurate infrared sensors. You will probably not be surprised to learn that the new paper fails to replicate the priming effect. As we know from Why Most Published Research Findings are False (also here), failure to replicate is common, especially when sample sizes are small. I haven’t yet described the real surprise, however.
The authors of the new paper, Doyen et al., then took the experiment meta; they ran the experiment again but this time they told half the people supposedly “running” the experiment that they expected the participants to walk slower and the other half they told that they expected the participants to walk faster. (A confederate provided evidence for this effect.) In the second experiment they again used the infrared sensors but they also asked the nominal experimenters to use a stopwatch as the sensors were said to be new and sometimes unreliable.
In the second experiment Doyen et al. were able to replicate the Bargh results. Namely, when using the stopwatch, the nominal experimenters reported that the group primed to walk slow did walk slow and they reported that the group primed to walk fast did walk fast. The results, however, were not entirely due to subtle experimenter bias because in the slow prime case the infrared sensors also found that the slow-primed group walked slow. The infrared sensors, however, did not report an increase in speed when the nominal experimenters expected an increase in speed.
Thus, the old-slow priming results appear to be due to a subtle mix of experimenter bias and standard priming which is cued or amplified via experimenter signaling. Given what are still relatively small sample sizes (50-60) the last should also be taken provisionally.
Important Addendum: Bargh has written a nasty attack on the new paper, the journal that published the paper, and Ed Yong who blogged the new paper for Discover Magazine. Bargh’s attack is a model of how not to respond to
criticism new information. Ed Yong discusses Bargh’s response here. Like Yong, I am dismayed that Bargh quotes the new paper inaccurately. In his attack, Bargh also says things such as the overuse of elderly-related items reduces the effect of the prime. Yet in the methods paper he cites (and wrote) he says more prime stimuli generally results in bigger effects (p.11, effects can vary if the subjects consciously recognize the prime, a factor that the new paper tests). Bargh also entirely glosses over the main point which is that the authors did find priming effects when the experimenter knew and expected the effect to occur. Note that given the subtlety of the effects any experimenter bias appears to be entirely unintentional and Doyen never argue otherwise.