Stage 2 · engines/text2audio

Their voice

The answer is spoken by F5-TTS, an open text-to-speech model that copies a voice from a ten-second sample. Out of the box it already sounds like them. This stage cleans four or so hours of their speech, trains F5 further on it (a finetune) so it also picks up how they pace and pitch a sentence, and picks the sample it copies from. About an hour and a half on one card.

How it speaks

the reference clip ~10 s of them, and exactlythe words they say in it the new text the answer, a fewsentences at a time F5-TTS, finetuned continues the clip as ifthey kept talking and saidthe new text next audio, streamed first sound in ~1 s;the rest follows whileit is still being made
Because it continues the clip, it inherits the clip's pitch and speed. The clip is chosen as carefully as the model, and shipped with it: checkpoints/<creator>/voice/ holds the weights, the clip, its words, and settings.

Why the clip matters: one measured case

For Dr K the first clip was picked because it sounded the most like him to a speaker-matching model. It was also higher than his usual pitch, and every sentence the voice spoke came out higher too. The speaker-matching score did not move, because that model is built to ignore pitch.

130 Hz150170190210 his median, 165 clip picked by match192 voice it produced177 clip picked by pitchat his median voice it produced21 Hz closer to him; sentence length error down 27%
No retraining, only a different clip. For Huberman a clip at the right pitch was 22% slower than his normal pace, and every sentence came out slow; the clip is now also matched on speed.

How it is built

01 inventorywhich videos train,which are held back 02 music probeis there a music bed?reports only 03 transcribeWhisper: every wordwith its time 04 plan cuts3-14 s clips, cut wherethey actually pause 05 speaker filterhow much each clipsounds like the main voice a person: thresholdread 05's table, dropguests and quoted voices 06 build datasetcut, one loudness levelper video, 48 kHz 15 pick referencea clean clip at theirusual pitch and pace a person: listenis it them, and arethe words right? trainF5 finetune; keeps acopy every 2,000 steps sweepeach saved model speaks40 held-back sentences a person: pickwhich saved step;listen beside the real 11 promote → checkpoints/<creator>/voice/weights without training state, the clip, its words, settings
The numbers are the script names in engines/text2audio/blocks/voice/scripts/. The orange boxes are the three places a person decides; for Dr K and Huberman each decision is recorded.

Why each cleaning step exists

stepwhat would go wrong without it
held-back videosScored on sentences from a video it trained on, a voice looks better than it is. Whole videos are held back, the same ones the face is scored on, so no stage learns from what another is tested on.
own transcript, not YouTube'sYouTube captions are tidied. A model trained on words that differ from the audio learns the difference.
cut on pausesCutting every 10 s slices words in half and teaches the model to start mid-breath.
speaker filterOne clip of a guest or a quoted voice shifts the voice the model settles on, and nothing measures it later. On their own channel the most common voice is them, so each clip is compared to that.
one loudness per videoVideos recorded months apart differ by ~10 dB. Levelling each whole video removes that; levelling each clip would also flatten how they get louder when they mean it.

How a saved model is judged

Three measurements per saved step, each read against what the creator's own real held-back speech scores, because even real speech does not score perfectly.

measurequestionwhat it can and cannot tell
similaritydoes it sound like them?Always about 99% of their own score, even before training: F5 copies the sound of a voice from the clip alone. Confirms the voice is them; cannot choose a step.
word errorscan Whisper read back the words?Catches slurring. Huberman: 0.4% for the voice, 1.3% for his own real speech.
pace and pitchsentence length, pitch range, pauses vs. the real sentenceThe only one that sees what training adds (14_prosody.py).

Commands

./alive build <creator> --stage voice runs the lot, stopping at the decisions. By hand, from engines/text2audio/blocks/voice/recipes/text-to-voice/:

tools/py tools/status.py --profile <name>      # what is done, and the next command to run

tools/py is a two-line wrapper that runs Python from the voice data-preparation environment (~/.venvs/vc-prep); the commands status prints already name the right Python.

Run what it prints, then ask again. A new creator starts as a copy of profiles/_template.py with three things filled in: the creator's name, their takes folder, and which videos to hold back. The full recipe, with every number from both creators, is RECIPE.md in that folder.

Where the decisions are kept

decisionstored in
held-back videos, speaker thresholdthe profile, profiles/<name>.py
the reference clipcreators/<id>/build/voice/reference.json
which saved stepcreators/<id>/build.json, voice.release_step

Things that go wrong

you hear or seewhy, and the fix
the voice is too high or too low throughoutthe reference clip's pitch. Re-run 15 and listen
every sentence drags or rushesthe reference clip's pace; same fix
an odd word or stumble at the start of answersthe clip's transcript is wrong. Check it word by word against the audio
training dies at a savedisk full: each saved step is 5.4 GB with its training state. Clear old steps in externals/F5-TTS/ckpts/
a hint of someone else's voicespeaker threshold too low; 05's table shows where the other voices sit