Lip-sync that holds (Plan A)
Fixing mouths after generation is slow and rarely works: a model that generated a wide-open shouting mouth cannot be "edited" into words. Drive the mouth from the start.
- Isolate the lead vocal. Separate the song into lead vocal and the rest; remove backing vocals and harmonies. The mouth should follow one voice.
- Cut it into lines. 3–8 seconds per segment, starting and ending in breaths.
- Front-facing close-ups only. Chest-up, whole head in frame, mouth closed or slightly open in the start frame, locked camera.
- Voice-driven generation. Feed the start frame and the vocal segment to an audio-driven avatar model. Prompt: the scene, plus "mouth closed when there is no voice".
- Everything else is not a mouth shot. Wides, backs, dancing, hands, props. Mark in the shot list which lines show the singing face; design the rest to avoid it.
- Measure, do not eyeball. Track mouth opening per frame and compare with the vocal loudness curve. Low correlation or an open mouth during silence = redo that segment only.
Plan B for the hardest fast rap: a person performs the lines to camera and a motion/expression transfer tool maps the performance onto the character (the performer's face never appears).