Reference

What this is built on

Most of the hard parts are other people's work. This page says whose, where each one sits in the system, and what was changed. Each has its own licence; read it before you use this for anything public.

Where each one sits

1 · answers Anthropic Claudewrites every answer inmystyletried, not used (below) 2 · voice F5-TTSthe voice model, finetuned faster-whisper · DemucsWavLM speaker check 3 · face clips VHAP + FLAMEthe tracker and its head model STAR face points · RobustVideoMattingInsightFace identity check 4 · voice to face motion NVIDIA Audio2Face-3Dspeech → face controls, which we correct Epic MetaHuman + OpenRigLogicthe face puppet, and the code that runs it 5 · face motion to picture STAvatarblobs bound to a mesh, trained from video;our changes are one patch on a fixed commit

The main pieces

STAvatar: the renderer

Zhao et al., STAvatar: Soft Binding and Temporal Density Control for Monocular 3D Head Avatars Reconstruction, arXiv 2511.19854. Code: github.com/JiankuoZhao/STAvatar.

Learns a photoreal head from a single video, as soft coloured blobs bound to a tracked mesh. We drive it with the MetaHuman mesh (plus teeth and eyeballs) instead of FLAME, train on many short clips merged, train on meshes predicted from sound, and fix several schedule and render-path issues. All of it is env/patches/stavatar.patch, applied by env/setup_externals.sh. Stage 5 explains how it is used.

VHAP and FLAME: the face tracker

Shenhan Qian, VHAP: Versatile Head Alignment with Adaptive Appearance Priors. Code: github.com/ShenhanQian/VHAP. It fits FLAME, the head model from the Max Planck Institute for Intelligent Systems (flame.is.tue.mpg.de; free registration, research licence), to every frame. Our changes are env/patches/vhap.patch. Stage 3.

F5-TTS: the voice

Chen et al., F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching, arXiv 2410.06885. Code: github.com/SWivid/F5-TTS, used unchanged at a fixed commit and finetuned per creator. Stage 2.

NVIDIA Audio2Face-3D: speech to face controls

Two model files (a speech encoder and an animation decoder) that turn speech into MetaHuman face controls. You supply them under NVIDIA's licence. We do not change them; stage 4 trains a correction on top.

Epic Games MetaHuman and OpenRigLogic: the face puppet

A MetaHuman face rig (DNA file) from Epic's MetaHuman Creator, under Epic's licence; you supply one. OpenRigLogic is Epic's open code that evaluates a rig; it is used once, to build the shared face assets, and the rig then runs in plain numpy and torch.

Anthropic Claude: the answers

Every answer is written by Claude through the Anthropic API, searching the creator's transcripts. Stage 1. Claude Code was also used to build and run the recipes, and picked both creators' resting poses by looking at frames.

inmystyle: tried for speaking style, not used

github.com/achak1987/inmystyle, arXiv 2607.29238. A toolkit that trains a small local model to rewrite a paragraph in one person's style with no instructions, by learning to undo machine-style paraphrases of their own writing.

tried LLM writes the content small model rewrites in style a judge spotted the realspeaker 60 times of 60 ships one LLM, which searches their own words and readsa note of 18 passages showing how they talk every claim cited toa real passage
The rewrite also changed meaning: in 1 of 5 answers it reversed a fact. The style is now carried by the persona note instead (stage 1). The design and the measurements are in problems.md (entries N8-N11) and engines/knowledge_style/blocks/manner/.

Smaller tools

toolused forstage
faster-whisper (Whisper large-v3)word-timed transcripts of their speech; reading back the cloned voice to count errorsvoice
Demucsmeasuring how much music is under the voicevoice
WavLM speaker model (microsoft/wavlm-base-plus-sv)keeping only clips that sound like the creatorvoice
STAR (VHAP's fork)68 face points per frameface clips
RobustVideoMatting (through VHAP)separating the person from the backgroundface clips
InsightFacechecking each face in a frame is the creatorface clips
PyTorch3D, nvdiffrast3D maths and drawing, inside the tracker and renderer3-5
yt-dlp, ffmpegdownloading the videos, cutting audio and framesall

The creators

The two example creators are Dr Alok Kanojia (HealthyGamerGG) and Dr Andrew Huberman (Huberman Lab). Their videos, transcripts and trained models are not in this repository; the recipes rebuild them from public uploads. Ask a creator before you build or show a model of them.