Seeing is Hearing: The McGurk Effect

by on September 20, 2011 at 6:30 am in Science | Permalink

Very cool auditory illusion created by visual dominance.

Hat tip to Carpe Diem.

Michael G Heller September 20, 2011 at 6:55 am

This conclusively demonstrates the Great Disadvantage of face to face social relations.

Michael G Heller September 20, 2011 at 9:03 am

This was partly tongue-in-cheek by the way (but you can’t see that).

Wayne September 20, 2011 at 12:42 pm

+1 for both comments. lol.

J Storrs Hall September 20, 2011 at 7:50 am

Aha! Holonic (expectation-influenced) interpretation at work. For a 15-minute talk about an AI architecture that works this way (by yours truly), see http://vimeo.com/7260012

Merijn Knibbe September 20, 2011 at 8:14 am

Last year, I’ve visited quite some anglo-saxon economic blogs. To my surprise, many fo them still cling to a rather outdated and refuted concept of rational, optimizing, atomistic man (a concept ridiculed by, for instance, Hayek). This is a very welcome antidote. We’re not independent atomicons. NIce to see this on a blog like this, we’re making progress.

Right Wing-nut September 20, 2011 at 8:52 am

Now I have to wonder if I’m the only person in the world who is (mostly) immune to the effect…. I _do_ see the “fa”, and I _do_ start to hear it. But something kicks in (very rapidly) & overwrites my perception to “ba”. I have to pay careful attention, or I can miss the “fa” entirely. My initial response was “why is his mouth going ‘fa’ while he’s saying ‘ba’?”

IVV September 20, 2011 at 10:58 am

For me, I ended up thinking, “That sounded like ba, but he said fa. His lip must be flapping unnecessarily, and he’s actually trying to say fa.”

When the two side by side are together is when it gets the most interesting. If I don’t focus on either person, my brain says it’s the right-hand ba speaker that’s talking.

Irwin Chusid September 23, 2011 at 9:20 pm

You’re not alone. Throughout the last 2 minutes I wondered: “What’s the illusion?” I heard “fa” immediately. FWIW, I’m a radio junkie.

Kenny October 8, 2011 at 5:10 pm

I couldn’t tell the difference between the two ‘sounds’ either. It _almost_ sounded like /pa/ or /da/ or /fa/ at various points, but I still ‘concluded’ I was hearing /ba/. Maybe this is related to my having a hard time ‘hearing’ song lyrics; to me they’re more of another (musical) instrument, though I still perceive them as voices.

Daniel Klein September 20, 2011 at 10:31 am

Very cool.

gpc31 September 20, 2011 at 10:44 am

So now I know why all those live concerts sounded so great in person but so crappy on tape. It wasn’t just the, uh, aromatic enhancers. Musicians have always known that you hear a concert with your eyes.

msgkings September 20, 2011 at 12:50 pm

Outstanding comment.

emerson September 20, 2011 at 1:07 pm

Terrible comment.

msgkings September 20, 2011 at 1:19 pm

Spiteful comment.

Dude Man September 20, 2011 at 5:34 pm

Commencing comment…

Yancey Ward September 20, 2011 at 12:18 pm

I have read and had this demonstrated before, but it was interesting to, first, close your eyes and listen to the entire segment. Without question, it was ba throughout. Then it was interesting to watch the fa motion for a few seconds, then close the eyes. At first, I kept hearing fa, but after a couple times, it reverted back to ba. Very strange how the brain works.

Pavel Kohout September 20, 2011 at 1:07 pm

Doesn’t work with me. I’ve heard “pa” throughout.

Rahul September 20, 2011 at 1:18 pm

What’s your native language?

Dude Man September 20, 2011 at 5:35 pm

Dolphin.

Roy September 20, 2011 at 5:41 pm

Exactly the right question, seeing this I remembered taking a Chinese class with two Japanese girls many years ago, their complete inability to distinguish between ba and pa, combined with their utter commitment to the program actually contributed to the wrecking of an academic department at a major midwestern university.

Rahul September 21, 2011 at 8:01 am

Not Wisconsin, by any chance?

Rahul September 20, 2011 at 1:18 pm

Coolest thing I’ve seen on MR this month!

Duh September 20, 2011 at 11:07 pm

The problem with this is that the subject is mispronouncing “Va.”

We are interpreting “Ba” when we view an abbreviated “Va.” But if the subject were stressing the “V” of “Va,” we could discern the problem, visually.

The problem is the structure of the experiment; not our interpretation.

Jim September 20, 2011 at 11:19 pm

As an old school perceptual realist, I have to say that’s an amazing illusion. I wonder if JJ Gibson knew about this one…

Andrew' September 21, 2011 at 7:50 am

A similar example here

Bill September 21, 2011 at 11:24 am

I found the url buried somewhere. cancel previous comment

Brian G. September 22, 2011 at 12:47 am

When I closed my eyes, I was able to make myself hear either one by recalling the image.

KS September 26, 2011 at 10:31 am

Exactly Brian G, this is the best way to beleivei it;s happening.

Rex Bovee September 22, 2011 at 6:38 am

This is very culturally dependent. I am not a linguist, but I know that our pronunciations of “b” and “v” are tied to our northwestern European antecedents.

In Russian (and maybe other Slavic languages), the b and v sounds are almost identical. The Russian word for “you” sounds like “bvwee.”

And in the Mexican iteration of Spanish, “b” and “p” are very similar. (Um, my colloquial example is rude, but very commonplace. Let’s just say it is sphincter-oriented.)

I know speech therapists are very educated in all this, as are linguists. Something about “aspiration,” about whether air is moving in or out of our mouths or is static as a sound is uttered. Certainly, the positioning of lips is critical, but that is seldom the only factor in cognition, just an enhancement.

I suspect that, as in so many social interactive events, it’s all about context. We (or “our brains” as the video explains) rarely take in clues from only one sense as we process information. Besides auditory input, we gather visual clues, we evaluate environmental, experiential and social settings, olfactory cues, distractions, social settings, all so quickly and without conscious thought (unless trained to employ conscious evaluative techniques.).

So “our brain” decides whether the consonant is “b” or “f” or “v” or “p” based on not only the sound we hear and the lip motion we see, but also the context of the words expressed, the social or physical environment (amusement park, hospital or courtroom?), possibly a perceived accent, the “noise” factor (Whether actual environmental noise or the cultural “noise” of an unfamiliar country, etc.).

Billions of neurons are furiously calculating, trying to make sense of disparate inputs, to answer our (our brains’) constant questions: What’s happening now? Am I threatened? What do I need to do/know to survive/thrive/conquer? Can I relax yet? Should I fight/flee/surrender?

In the context of all these factors, the McGurk effect is interesting to Academia, but the real question is how often it actually affects or confuses our decision-making process regarding any individual’s survival.

In any instance of confusion of a spoken statement, how frequently does the confusion of “b” and “f” actually result in misunderstanding vs. how frequently do additional clues and inputs result in a correct interpretation of the statement? That’s the key.

If in large part we understand correctly the message being expressed, the communication was successful despite some contrary visual clues.

(For more context, I am amdues by teh expamlse of mseasges whit srcbamlde ltestre oru bniasr cna dceode!)

Anthony September 22, 2011 at 1:51 pm

Rex – while Mexicans may be merging ‘b’ and ‘p’, pretty much all Spanish-speakers have merged ‘b’ and ‘v’. So much so that the Spanish Academy decided that the name of the letter ‘v’ would henceforth be “uve”, which was not universal among Spanish-speakers, to eliminate confusion between “be” and “ve” (the more widespread name of the letter ‘v’). Also notice that Spanish-language maps often spell the name of Cuba’s capital city as “Habana”, which is not phonetically distinct from “Havana”.

gamesliga October 3, 2011 at 4:44 pm

gamesliga bahis sitesi.
gamesliga yüksek oranlı iddaa oyna.
gamesliga gamesliga referans kodu servisi.
gamesliga gamesligaya üyelik üye olma
gamesliga referansı bahis sitesi.
gamesliga referans yüksek oranlı iddaa oyna.
gamesliga üyelik gamesliga referans kodu servisi.
gamesliga referans gamesligaya üyelik üye olma…

Comments on this entry are closed.

Previous post:

Next post: