Stage 3 · engines/audio2face/corpus

Clean, tracked clips of their face

Everything the face stages learn comes from this: short stretches of video where we know, frame by frame, the 3D shape and position of the creator's head. Getting there from an edited YouTube video is mostly about throwing the right footage away.

What "tracked" means

The tracker is VHAP. It fits FLAME, a standard 3D model of a human head, to every frame. FLAME is a mesh whose shape is set by a few hundred numbers: some fixed for the person (how wide the jaw is), some changing every frame (mouth open, brow raised), plus where the head is and which way it points.

a clip of them 512 × 512, face centred VHAP moves FLAME until itsoutline and features landon the face in the frame their head shapeone set of numbers per video expression, every framejaw, lips, brows, eyes head position, every frameturn, tilt, distance

Why a YouTube video cannot go straight in

VHAP expects one unbroken shot from one fixed camera. An edited video has cuts, side panels, graphics and screenshots with other people's faces. None of these makes the tracker fail; each makes it return a confident wrong answer.

one video, as edited him, centredadcentredside panelgraphicscreenshotcentredtiny inset what each does to the tracker graphic or ad · no face found; the tracker is pushed away from the face and the head spins. Measured: it turned 138° over an 18 s graphic and stayed wrong for the next 28 s of perfectly good footage. side panel · the tracker assumes the face is near the middle of the picture; a face at the edge is read as a head turned 30°. Fixed by cropping every frame so the face is centred. screenshot · the face finder locks onto someone else, at full confidence. Dr K's second video: 7% of frames. kept, as chunks a camera move splits one shot
Each kept stretch is a chunk: one camera, one layout, cropped to a fixed box around the face, a few seconds to a minute long. A shot can split into two chunks when the camera re-frames. Side panels are kept, because after the crop the face is centred again.

How it is built

prepareevery frame as an image,and a person/background mask landmarks68 face points per frame,or "no face" thumbnailsa tiny copy of each frame,and where the cuts are group framesframes that look alike gotogether: about 30 groups a person: labellook at 8 frames of eachgroup; keep or drop, andcentred or panel face checkevery face in every frame,compared with thecreator's face cut chunkskept groups, minus fades,camera moves and strayfaces; crop each a person: watcha few chunk clips: onecamera, no cut, nographic, face centred track each chunkVHAP, a few minutesper chunk; most of the10-20 hours pick the donorthe longest chunk thatfits well; its head shapebecomes theirs track againevery chunk with that onehead shape held fixed a person: overlaythe mesh drawn on eachchunk's worst frame: is iton the face? reclaimdelete the tracker's 50 MB textures per chunk, keep its0.2 MB of numbers. Without it the disk fills
Orange boxes are the three looks a person must take. Each caught something no number reported: a bodybuilder in screenshots boxed as the creator, a membership promo inside a kept group, a title card left inside a chunk after a filter silently stopped running.

Why the crop fixes two problems at once

the full frame, a side-panel layout screenshot,another face picture centre the crop: 3.2 × theface, face in the middle no false head turn,and no room for asecond face (0.15% ofkept frames at most)

Commands

./alive build <creator> --stage clips runs it, stopping where a person must look. By hand, from engines/audio2face/, with the tracker's environment:

~/miniconda3/envs/vhap/bin/python corpus/status.py --takes takes/<folder>   # every video: done so far, next command
~/miniconda3/envs/vhap/bin/python corpus/status.py --seq <take>             # one video

A new creator starts as a copy of recipes/face-clips/profiles/_template.py. Its numbers (how transparent a graphic is, where a side panel sits, how short a shot can be) are read off the creator's own footage, because they differ between channels: corpus/derive_constants.py --seq <take> prints each measured distribution with a suggested cut-off, and you type the value you choose into the profile. A suggestion is a starting point: on Dr K's second video it suggested 0.325 for one value where 0.62 was right. The full recipe, with every measurement from Dr K's three videos and Huberman's one, is engines/audio2face/corpus/recipes/face-clips/RECIPE.md.

Labelling the groups

The grouping step writes one sheet per group to data/face/corpus/chunks/<take>/clusters_full/: 8 frames, with the face detector's box in red. Open them in any image viewer and write, by hand, one line per group into creators/<id>/build/clips/<take>/verdicts.json:

{
 "0":  {"verdict": "keep", "kind": "centred", "note": "clean full-frame talking head"},
 "3":  {"verdict": "drop", "kind": "graphic", "note": "title card"},
 "7":  {"verdict": "keep", "kind": "panel",   "note": "him on the right, screenshot left"},
 "11": {"verdict": "drop", "kind": "wrong_face", "note": "red box on a man in a screenshot"}
}

Every group gets a line. kind is centred or panel for kept groups.

Where the decisions are kept

decisionstored in
which videos to trackcreators/<id>/build.json, clips.takes
keep / drop per group, with the reasoncreators/<id>/build/clips/<take>/verdicts.json
how the groups were made, so the labels point at the same frames next timecreators/<id>/build/clips/<take>/cluster_config.json
When unsure, dropA good frame thrown away costs a fraction of a percent of hours of footage. A bad frame kept is fitted into the head shape that every other chunk then inherits. Dr K's third video, a livestream, lost 36% to this rule, mostly a webcam inset too small to use.

Things that go wrong

you seewhy, and the fix
a group's 8 frames show two different thingssplit just that group: cluster_frames.py --subcluster <id> --sub-k 3. Re-grouping everything renumbers the groups and breaks every label
the red box in a group sits on someone elsedrop the group as wrong_face
"no donor candidate" after hours of trackingthe donor is the chunk whose head shape every other chunk borrows, chosen by its fit score (residual: how far the fitted head drifts, in mm). The scores were not written; look in the tracking log for residual extraction FAILED
the machine runs out of memory while trackingeach chunk starts several face-point workers (2 by default). Run with LMK_JOBS=1 in front of the command
the disk fillsreclaim was not run. About 4 GB per tracked video after reclaim, more while it is being tracked; only the tracked videos count