The McGurk effect is an illusion in which what you see a speaker’s lips do changes the sound you hear. Harry McGurk and John MacDonald reported it in Nature in 1976: when the sound [ba] was dubbed onto a film of lips saying [ga], adults reported hearing [da], a syllable that was in neither the sound nor the picture. It shows that speech perception is not purely auditory; the brain fuses sight and sound.
A video of a face saying one syllable is dubbed with audio of a different syllable. In McGurk and MacDonald's study, audio [ba] on lips saying [ga] was heard as [da]; the reverse dub was often heard as [bagba] or [gaba].
02
The brain settles on a compromise
Speech perception draws on both hearing and lip movements. When they conflict, the brain often settles on a sound that fits both partly, rather than picking one.
03
It happens in a specific place
Disrupting the superior temporal sulcus with magnetic stimulation made people less likely to hear the fused syllable, pointing to that region as a hub for combining sight and sound (Beauchamp et al., 2010).
04
Not everyone falls for it
In 165 English-speaking adults, how often people perceived the illusion ranged from 0% to 100%, and individual rates were stable when 40 of them were retested a year later (Mallick et al., 2015).
Where you see it today
Machines that read lips
AI speech systems now copy the brain’s trick. AV-HuBERT learns from lip movements and sound together, and the combined representation helps both lip-reading and speech recognition (Shi et al., 2022).
AI dubbing and lip-sync
Models such as Wav2Lip generate lip movements to match a new soundtrack in ordinary video, with lip-sync accuracy close to real footage on the authors’ benchmarks (Prajwal et al., 2020). They aim to remove the visible mismatch that makes dubbed video feel wrong.
Synthetic video and trust
The McGurk effect is a reminder that people do not judge audio on its own: a face that moves in sync with a voice shapes what viewers think they heard. That makes lip-synced synthetic video more persuasive than a voice clip alone, and checks on video need to treat sight and sound together.
Calls, glasses and headsets
Video calls, smart glasses and headsets all deliver speech with a face attached. When picture and sound drift apart, the brain’s fusion works against the listener, so product teams treat audio-visual sync as part of speech clarity.
The McGurk effect and related perception effects
McGurk effect
What happensSeen lips change the syllable heard
What it showsSpeech perception fuses vision and hearing
Ventriloquism effect
What happensSound seems to come from a moving mouth or object
What it showsVision pulls where we locate a sound
Illusory truth effect
What happensRepeated claims feel more true
What it showsFamiliarity is mistaken for accuracy
Framing effect
What happensThe same facts, worded differently, change choices
What it showsPresentation shapes judgment
Mandela effect
What happensMany people share the same false memory
What it showsMemory is rebuilt, not replayed
How to use it
When a message matters, show your face as well as your voice: lip movements help listeners decode speech.
Check audio-visual sync in video calls, recorded talks and dubbed content before you publish.
Judge suspicious clips with the sound off and then with eyes closed; if the two tell different stories, dig further.
When testing voice or video products, expect large differences between people; do not rely on an average response.
In localisation, weigh lip-synced AI dubbing against subtitles: the first feels natural, the second keeps the original performance.
The Business Engineer lens
As AI makes it cheap to generate faces and voices that move together, the McGurk effect explains why that pairing is so persuasive. The Business Engineer’s view is that the interface battle moves to devices that combine sight and sound, and trust will go to the platforms that can prove what is real.
It is an illusion in which seeing a speaker's lips form one syllable while hearing another makes you perceive a third, such as hearing [da] when the sound is [ba] and the lips say [ga].
Who discovered the McGurk effect?
Harry McGurk and John MacDonald described it in the paper Hearing lips and seeing voices, published in Nature in 1976.
Does everyone experience the McGurk effect?
No. A 2015 study of 165 adults found that how often people perceived it ranged from never to always, and each person's rate was stable over a year.
Why does the McGurk effect matter for AI?
AI lip-reading systems use the same mix of sight and sound, and lip-sync models can make dubbed or synthetic video look natural, which affects how people judge what they hear.
The complete explainer, cleaned up and reformatted for mobile.
The McGurk Effect is a fascinating perceptual phenomenon demonstrating that visual input can override auditory perception, causing a fusion of what’s seen and heard. It uncovers insights into speech perception and the complexity of multisensory interactions, revealing how vision influences the way we perceive speech sounds.
Key Concepts of the McGurk Effect:
Audio-Visual Integration:
The McGurk Effect demonstrates the brain’s tendency to integrate auditory and visual cues during speech perception, even when the two streams of information conflict.
When presented with incongruent audio and visual inputs, individuals often perceive a combined percept that blends elements of both modalities.
Phonetic Fusion:
In the McGurk Effect, listeners may perceive a phoneme that is different from both the auditory and visual stimuli presented.
For example, hearing the sound “ba” while seeing the speaker’s lips forming the shape for “ga” may result in perceiving the syllable “da.”
Cross-Modal Influence:
The McGurk Effect illustrates how visual information can influence auditory perception and vice versa, highlighting the multisensory nature of speech perception.
The brain integrates information from multiple sensory modalities to construct a coherent perceptual experience, even in the presence of conflicting signals.
Implications of the McGurk Effect:
Speech Perception:
The McGurk Effect has significant implications for understanding how humans perceive and interpret speech in everyday communication.
It underscores the importance of visual cues, such as lip movements and facial expressions, in complementing auditory information during speech processing.
Language Development:
Studies of the McGurk Effect suggest that infants and children may rely on both auditory and visual cues to learn language and develop phonemic awareness.
Understanding the interplay between sensory modalities in speech perception can inform interventions for individuals with speech and language disorders.
Neuroscience of Perception:
Neuroimaging studies have elucidated the neural mechanisms underlying the McGurk Effect, implicating regions of the brain involved in multisensory integration, such as the superior temporal sulcus and auditory cortex.
Investigating the neural basis of audio-visual integration sheds light on broader principles of sensory processing and perception.
Applications and Research Directions:
Speech Therapy and Rehabilitation:
The McGurk Effect has practical applications in speech therapy and rehabilitation for individuals with communication disorders.
Therapeutic interventions that leverage multisensory cues can improve speech perception and production in clinical populations.
Virtual Reality and Augmented Reality:
Understanding the mechanisms of audio-visual integration underlying the McGurk Effect is relevant for the design of immersive technologies such as virtual reality (VR) and augmented reality (AR).
These technologies can leverage multisensory cues to enhance user experiences in interactive environments and simulations.
Cognitive Neuroscience Studies:
Ongoing research on the McGurk Effect contributes to our understanding of higher-level cognitive processes, including attention, memory, and decision-making.
Studying individual differences in susceptibility to the McGurk Effect provides insights into the variability of perceptual experiences across populations.
Future Directions and Challenges:
Individual Variability:
Exploring the factors that influence susceptibility to the McGurk Effect, such as linguistic background, cognitive abilities, and sensory processing differences, remains a key area of investigation.
Understanding individual variability in audio-visual integration can inform personalized approaches to intervention and rehabilitation.
Neural Mechanisms:
Further research is needed to elucidate the neural mechanisms underlying the McGurk Effect at the cellular and circuit levels.
Advances in neuroimaging techniques and computational modeling offer opportunities to unravel the complex interactions between brain regions involved in multisensory integration.
Real-World Applications:
Translating insights from laboratory studies of the McGurk Effect into real-world applications requires interdisciplinary collaboration between researchers, clinicians, engineers, and educators.
Integrating knowledge from basic research on audio-visual perception with applied domains can lead to innovations in communication technologies, assistive devices, and educational interventions.
Case Studies and Variations of the McGurk Effect
To illustrate the nuances and impact of the McGurk Effect, let’s explore a few case studies and variations:
2. Multisensory Integration in Autism
Research has shown that individuals with autism spectrum disorder may experience altered multisensory integration, affecting their susceptibility to the McGurk Effect. This highlights the role of sensory processing differences in speech perception.
3. Language Learning
The McGurk Effect is relevant in language learning scenarios. When individuals learn a new language, they may rely on visual cues from lip movements to aid in speech comprehension, demonstrating the significance of audiovisual integration in language acquisition.
The McGurk Effect is a captivating phenomenon that underscores the intricate interplay between auditory and visual information in speech perception. It challenges our understanding of how we perceive the spoken word and highlights the brain’s remarkable ability to seamlessly integrate sensory input from different modalities. Beyond its scientific significance, the McGurk Effect has practical applications in speech and language research, communication disorders, technology development, and education. It serves as a reminder of the complexity and versatility of human perception, continually deepening our appreciation of how we make sense of the world around us.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
