How does AI lip sync work, and how do you get a clean result?

How audio-driven lip sync works

Speech sounds map to a small set of visible mouth shapes, often called visemes. The sounds p, b and m close the lips; f and v put the lower lip against the teeth; an open "ah" drops the jaw. Lip sync AI learns this mapping from many hours of video of people talking, so that given new audio it can predict which shape the mouth should take at each frame.

A well-known example is Wav2Lip, published in 2020 by Prajwal and colleagues. Its generator receives a face frame with the lower half hidden, a reference frame of the same person, and a short window of audio, and it paints a new lower face to match the sound. A separate network, a pretrained lip-sync "expert", judges whether the mouth matches the audio, and the generator is trained to satisfy it. The paper reports that this expert is the main reason the model keeps sync on arbitrary faces in unconstrained video.

Newer systems go further. Diffusion-based and avatar models generate the whole face or head, with blinks, head motion and expressions driven by the audio, which is how a single photo becomes a talking video. Some video generators now produce speech and lip movement together from a prompt. The core idea is the same: sound in, matching mouth motion out, and the quality depends heavily on what you feed in.

Three uses: talking photo, dubbing, singing

Talking photos are the easiest to make and the hardest to make convincing, because the model invents every frame of motion. Dubbing real footage keeps real motion and only replaces the mouth, so it looks more natural but is less forgiving of bad source video. Singing is the most demanding: vowels are long, the mouth opens wider than in speech, and the timing has to match the music, not just the words. For full song workflows, see our guide to AI music videos.

Input requirements for the face and the audio

Most failures trace back to the input. Check these before you generate.

• Face angle: frontal or up to roughly a three-quarter view. Side profiles fail often, because the model sees too little of the mouth.
• Mouth visible: no hand, microphone, hair or mask over the lips, and no heavy motion blur.
• Light: even, soft light on the face. Hard shadows across the mouth make the redrawn area stand out.
• Face size: the face should fill a good part of the frame. Small faces in wide shots give the model few pixels to work with, and the result looks soft.
• Audio: one voice, dry (little room echo), no music underneath, no clipping.
• Length: check the tool's limits. Many tools cap clip length or total audio length, and long clips drift more. Shorter segments are easier to check and redo.

Preparing the audio and a worked example

Treat the audio as the master and the video as something that follows it. Here is a complete example: a 12-second English line from a talking-head clip, dubbed into Spanish with lip sync for a vertical short.

1. Export the original speech and transcribe it. A Whisper-based transcriber, for example cv.cm's speech-to-text tool, gives you timestamps for each line.
2. Translate the line and shorten it to fit about 12 seconds when spoken. Spanish usually needs more time than English, so cut words rather than speed up the voice.
3. Generate the Spanish voice, or record it. See our guide to AI voiceover for voice choice and pronunciation fixes.
4. Clean it: remove background noise, keep the voice dry, and trim silence at the start so the first sound lands where the speaker's first mouth movement is.
5. Cut the audio to the exact segment the tool will use, with an audio cutter, instead of letting the tool choose a slice of a longer file.
6. Convert it to a plain format the tool accepts, then run lip sync on the 12-second video clip.
7. Put the clean audio back on the result and check sync at full screen, frame by frame on the p, b and m sounds.

# 1. Extract a mono 48 kHz WAV from the source clip
ffmpeg -i source.mp4 -vn -ac 1 -ar 48000 speech.wav

# 2. Cut the dubbed voice to exactly 12 seconds, starting at 0.25 s
ffmpeg -ss 0.25 -t 12 -i dub_es.wav -ac 1 -ar 48000 dub_es_12s.wav

# 3. Replace whatever audio the tool returned with your own clean track
ffmpeg -i lipsync.mp4 -i dub_es_12s.wav -map 0:v:0 -map 1:a:0 -c:v copy -c:a aac -shortest synced.mp4

Step 3 matters more than it looks. When a tool regenerates or re-encodes the soundtrack, the audio can start a little late while the picture still follows the original timing. With video models that render audio themselves, we have measured shifts of around one second. Before you decide the sync is wrong, put your own audio back on the clip with the command above and check again.

For a talking photo, the same audio steps apply. Some tools also take a voice sample to set the voice; a clean 3-second sample is usually enough. Take the loudest, most continuous sentence, and keep in mind that some tools cap the total length of all reference audio combined.

Common artifacts and how to fix them

Upscaling helps soft results, but do it once, after the edit, and keep one master export. Every extra encode, including the one a social app does on upload, adds artifacts around the mouth, where viewers look first.

Consent, deepfake rules and labeling

Lip sync changes what a person appears to say. That is exactly what deepfake rules address, so treat consent and labels as part of the workflow, not an afterthought.

• Use only faces and voices you own or have written permission to use, with the purpose stated. Dubbing a client's video into other languages is a clear case; making a public figure say new words is not.
• YouTube asks creators to disclose realistic altered or synthetic content, including content that makes a real person appear to say or do something they did not. YouTube also offers its own experimental lip sync for auto-dubbed videos on selected channels.
• The EU AI Act (Article 50) requires deployers to disclose AI-generated or manipulated audio and video that constitutes a deep fake. For evidently artistic, creative, satirical or fictional works, the duty is limited to disclosing that the content exists in a way that does not spoil the work.
• China's Measures for Labeling AI-Generated Synthetic Content, in force since 1 September 2025, require visible labels on synthetic video, including at the start, and require users to declare AI content when they publish it on platforms such as Douyin, Kuaishou, Xiaohongshu or WeChat Channels.
• Avoid minors' faces, and never use lip sync to create sexual, defamatory or fraudulent content. Those uses are prohibited on major platforms and illegal in many places, with or without a label.

Our guide to labeling AI content explains how each platform's AI label works.