Stage 4 · engines/audio2face/rigfit and identity

Voice to face motion

This stage makes two things: a 3D face of the creator that can be moved like a puppet, and a small model that listens to speech and works the puppet's strings the way this creator's face moves. Its output is a moving 3D face with no colour or skin; the next stage paints it.

The parts, in plain words

partwhat it iswhere it comes from
a MetaHuman rigEpic Games' professional face puppet: a detailed face mesh of 24,049 points, moved by 263 named controls (jaw open, left lip corner up, brow lower, ...). Setting the controls poses the faceyou supply one (setup); it is someone else's face
their rigthat same puppet, its resting face reshaped to the creator's head. Same 263 controlsbuilt here from the head shape tracked in face clips
Audio2Face-3DNVIDIA's model that turns speech into those 263 control values. Trained on eight other peopleyou supply its two model files
the correctiona small network trained here: it adjusts Audio2Face's controls to fit this creatortrained here, about 40 minutes

How speech becomes a moving face

speech16 kHz audio Audio2Face-3D263 control values,30 times a second,for a generic face the correctionlooks at a few secondsof sound around eachmoment; nudges each their rigcontrols →24,049 points a moving3D faceto stage 5 Only the correction is trained per creator. The rig is built once, from their tracked head. + 16 extra shapesfor what the puppetcannot reach
The 16 extra shapes (the corrective layer) cover the part of this creator's face the stock puppet cannot form however its controls are set. They travel with the model, so the next stage poses exactly the face the model was trained on.

How the correction is trained

From stage 3 we know what the creator's face actually did in every frame of every clip. That is the answer key. The model is scored on where the face's points end up, never on the control values themselves (many control settings give the same face, so the values are not a fair target).

a clip's soundshifted by the measured lag Audio2Face + correction→ controls → their rig predicted face24,049 points, each frame the same clip, trackedstage 3's 3D head, movedonto the rig's points how far apart?point by point, plus apenalty for lips left open the rig is maths a computer can runbackwards, so the error flows back and tells the correction which way to nudge

The clips are split once, for every stage

train: 110 of Dr K's 162 chunks validate: 26 test: 26 learned frompicks the runscored once, last
Whole clips, never single frames: a frame next to one the model trained on tells you little. The renderer reads the same split, so no stage learns from a clip another stage is tested on.

How good it is

The score is how much of the creator's real face movement the predicted face reproduces on test clips it never saw: 0% is their plain resting face, 100% is exact. Dr K's deployed model, beside what it is compared with:

0%255075100% Audio2Face alone11.3 average face, held still20.6 with the correction32.2 best the stock puppet can do61.0 ... with the 16 extra shapes87.0 ceilings: setting the controls by hand, frame by frame, to fit the tracked face
Out of the box, Audio2Face is worse than not moving at all on this creator; the correction is what makes it usable. The dashed bars are ceilings: no model driving this puppet can beat them, so a score is read against them. Huberman's model does not beat his average face held still on this score (39.0% against 39.1%): most of what it measures for him is a fixed difference in his resting face, not movement. His mouth movement is learned about as well as Dr K's, and that is what a viewer reads. Training on one video is the likely limit.

How it is built

1 their rigreshape the puppet'sresting face to theirtracked head a person: lookis it them, beside thestock face? 2 Audio2Facerun over each video'swhole soundtrack, once;about a minute 3 measure the laghow late the sound isagainst the mouth, pervideo a person: timinga sheet of mouths: video,tracker and rig in thesame state, row by row? 4 targets and splitthe tracked face on therig's points; train /validate / test 5 ceiling and shapesbest fit per frame; fitthe 16 extra shapes towhat is left over 6 traintwo runs, differentrandom starts, to seehow much is luck 7 mouth checkopening, width, closure 8 promote → checkpoints/<creator>/face/motion/ and face/rig/only the run that won on validation; a person must write what is still wrong with it
About two hours of machine time after face clips. The timing sheet matters: once the lag was applied backwards, and the model trained without complaint and scored five points low.

Commands

./alive build <creator> --stage motion runs it. By hand, from engines/audio2face/:

~/miniconda3/envs/stavatar/bin/python rigfit/recipes/audio-to-mesh/tools/status.py --profile <name>

It prints what is done and the next command, with the right Python in it.

A new creator starts as a copy of rigfit/recipes/audio-to-mesh/profiles/_template.py: their name, and which tracked video goes with which soundtrack. The full recipe is engines/audio2face/rigfit/recipes/audio-to-mesh/RECIPE.md; how a rig is reshaped to a person is in engines/audio2face/identity/README.md.

Where the decisions are kept

decisionstored in
the timing sheet was looked at, and what the reader saidcreators/<id>/build/motion/joins_acked.json (Dr K only; Huberman's was never signed off)
which run to keep, and what is still wrong with itcreators/<id>/build.json, motion.run and motion.wrong

Things that go wrong

you seewhy, and the fix
a decent score but the mouth moves early or latethe lag. Look at the timing sheet; never type a lag in by hand, it differs per video (Dr K +110 ms, Huberman −105 ms)
every score is low, the mouth looks finethe rig is still the stock face; the identity check flags it. Rebuild step 1
the lips never quite shutknown on both creators (Dr K about 4 mm open). The closure penalty helps a little; more clips help more
the face upstream looks fine, the next stage's mesh is torna model handed on without its 16 extra shapes; promote copies them, never copy a model by hand