Stage 5 · engines/audio2face/gauss, built on STAvatar

Face motion to picture

Stage 4 gives a moving 3D face with no skin, hair or light. This stage learns to paint it as the creator, from their own video frames, so that whatever the mesh does, the picture looks like them doing it.

How a face is drawn: blobs stuck to a mesh

The renderer is STAvatar (a published method; credits). It draws the head as a cloud of about 80,000 tiny soft coloured blobs (3D Gaussians: each a fuzzy ellipsoid with a colour and a transparency). Each blob is attached to a triangle of the face mesh, so when the mesh moves, the blobs ride along. A small network adds a nudge per blob for each expression: the crease that appears in a smile, the inside of the mouth.

mesh at rest each blob sits on one triangle mesh moves mesh posed (mouth opening) blobs ride along; the network nudges each for this expression seen fromthe cameraa photo-like512 × 512 frame

How it learns

the mesh for one frameas stage 4 predicts it fromthat frame's sound draw the blobsfrom that frame's cameraand head position a drawn frame the real framefrom the tracked clip,background removed compare pixelsmove, recolour, add orremove blobs to close the gap 150,000-170,000 rounds, 3-4 hours
It trains on the mesh predicted from sound, not the tracked one, so it learns to paint the face it will actually be given when live: where the predicted mouth is a little wrong, it learns to make it look right anyway.

What was changed from STAvatar as published

changewhy
the mesh is the MetaHuman rig, with teeth and eyeballs added, not FLAMEso stage 4's output plugs straight in; the teeth make an open mouth read as a mouth
many short clips merged into one training setSTAvatar expects one long, steady video; Dr K's training set is 162 chunks
only frames where they face the cameraturned heads smeared, and the live app shows them facing forward
blink amount measured from the real eyesthe sound cannot say when someone blinks

Our changes are kept as one patch (env/patches/stavatar.patch) applied on a fixed STAvatar commit by env/setup_externals.sh.

Putting the face on their real body

A floating head looks wrong. So the live app pastes the drawn head into real footage of the creator, and only the head is new: shoulders, hands and room are the original video.

a real clip of themfacing camera, still draw the new headwith that clip's owncamera and head track,so it lands wheretheir head was twice: on white andon blackthe difference gives theexact edge, hair included pasted backonly the part that moveswith the skull; edge pulled in The body footage is chosen automatically from frames where they face the camera, one recording at a time. Known gap: a hand or microphone in front of the face is painted over.

How it is built

stepdoes
exporteach tracked chunk into the renderer's input format: frames, masks, cameras
head placementswhere the skull is in every frame, so the rig's face can be put there
render settingsthree values worked out from rules, not guessed (derive.py)
blinkshow open the real eyes are, every frame
cook meshesstage 4's model run over every chunk's sound: the predicted mesh per frame, with teeth
merge, front-facing onlyone training set, holding back the validation and test clips
a person: the joinsa sheet with the mesh drawn on the photos, and a two-second video with sound. Is the mesh on the face, and in time?
trainSTAvatar, 3-4 hours
render and looka held-back clip, with sound
a person: promoteto checkpoints/<creator>/face/render/, writing down what is still wrong

Commands

./alive build <creator> --stage render runs it. By hand, from engines/audio2face/:

~/miniconda3/envs/stavatar/bin/python gauss/recipes/mesh-to-render/tools/status.py --profile <name>

# render a held-back clip, on its real body, with sound:
~/miniconda3/envs/stavatar/bin/python gauss/recipes/mesh-to-render/tools/infer.py --profile <name> --run <run> \
    --clip <chunk> --camera clip --pose replay:<chunk> --composite

The profile has the same name as the creator's motion profile and reads most of what it needs from it. The full recipe is engines/audio2face/gauss/recipes/mesh-to-render/RECIPE.md.

Where the decisions are kept

decisionstored in
the joins sheet was looked atcreators/<id>/build/render/joins_render_acked.json (Dr K only)
release the deployed run, and what is still wrong with itcreators/<id>/build.json, render.deployed and render.wrong

The profile is gauss/recipes/mesh-to-render/profiles/<id>.py, copied from _template.py in the same folder.

Things that go wrong

you seewhy, and the fix
dark specks or lines beside the eyes or moutha render path that let the network nudge the teeth and eyeball blobs. Every path must freeze them, as training does
the face smears when they turntrained on front-facing frames on purpose; the body footage is front-facing too
a hand or microphone vanishes behind the faceknown; separating the face from what is in front of it needs a face-parsing model not yet added
the mouth runs ahead of the voiceHuberman: about 0.2 s, because his source video's sound is late. Open
training will not startthe live app holds the card. ./alive down