Ask a creator a question, and watch them answer
alive takes a creator's public YouTube videos and builds four things from them: a way to answer questions in their words, their voice, a model of how their face moves when they talk, and a renderer that draws their face. Put together, you type a question and see them answer it on their own body.
The whole system in one picture
The two hardest parts stand on published work: the face is drawn by STAvatar, a method that learns a photoreal head from video, and the voice is F5-TTS. Credits lists every outside piece, where it sits, and what was changed.
What is a recipe?
A recipe is the written procedure for building one of those models, for any creator. The same recipe builds Dr K's voice and Huberman's voice; only the settings differ. Each recipe has four parts:
Because the status command only reads what is on disk, you can stop at any point, reboot, and pick up where you were. Nothing remembers anything except the files.
You will rarely run the recipes by hand. ./alive build <creator> runs all of them in
order, asking each one's status command what is next. The pages under Build a creator explain
what each recipe does, so that when one stops and asks for you, you know what it wants.
Words this guide uses
| word | means |
|---|---|
| creator, id | the person being modelled. Each has a short id: creators/<id>/, checkpoints/<id>/. Today: drk (Dr K) and huberman |
| stage names for the same person | some stages know a creator by another name. Dr K is drk to the voice and face stages but healthygamer (his channel) to the answer and face-clip stages. creators/<id>/build.json lists the name each stage uses; for a new creator, use the id everywhere |
| take | one downloaded YouTube video, in data/takes/<folder>/<take>/; <folder> is set in build.json |
| LLM | a large language model: here Anthropic's Claude, called over the internet with your API key. It writes the answers |
| persona note | a short file of 18 passages of the creator's own speech, picked by a person, that shows the LLM how they talk |
| rig, face puppet | a 3D face mesh moved by named sliders (controls: jaw open, lip corner up ...). The one used is Epic Games' MetaHuman rig, reshaped to each creator |
| profile | a small settings file for one creator in one recipe: profiles/<name>.py |
| checkpoint | a trained model. All of a creator's are in checkpoints/<id>/ |
| promote | copy the chosen trained model into checkpoints/<id>/, with a record of how it was made and what is still wrong with it. Only promoted models are used by the app |
| stop (gate) | a step where the build halts because a person must look and decide |
| joins sheet | an image, and sometimes a two-second video, showing sound, video and 3D face side by side, so a person can confirm they line up. The motion and render stages each have one |
data/, externals/ | folders that are not in git: your data, and outside code (the face tracker, the renderer, the voice model) |
Where do you want to start?
Whichever you pick, set up the machine first.
What is not in the repository
- The trained models. Not released. You build them with the recipes.
- Licensed inputs each user must get themselves: the FLAME head model, NVIDIA's Audio2Face-3D speech models, and a MetaHuman face rig. Setup says where each comes from.
- The creators' videos and words. Downloaded from YouTube by the build; never committed.
What you need
| to run the app | to build a creator | |
|---|---|---|
| GPU | NVIDIA, 12 GB | NVIDIA, 16 GB (built on an RTX 5080) |
| RAM | 16 GB | 32 GB |
| disk | ~3 GB per creator | ~150 GB per creator while building |
| time | seconds to start | about a day per tracked video, mostly face tracking |
| also | Linux, an Anthropic API key | the same, plus ffmpeg and yt-dlp |