Standalone experiment cloning Kevin/Marge/Terry/Priya into a single
VibeVoice-1.5B multi-speaker generation, for comparison against the
current per-line AuK+MOSS pipeline. See vibevoice-spike/README.md.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SN15Vxn4svKuhYxFsVtMJf
Type any text and hear the selected character say it, in their base
voice or any approved/pending emotion. Generalizes two things at once:
testing an emotion candidate with real dialogue instead of the fixed
"Testing the X emotion for Y" line, and just answering "how would
they say this?" for the approved voice directly from the browser
without switching to the Pipeline Studio tab.
list_character_voices() enumerates every voice available for a
character (base + approved emotion_library entries + pending
emotion-cache candidates) - the voice dropdown refreshes whenever the
character selection changes, and also right after generating a new
emotion candidate so it is immediately pickable without reselecting
the character. Every clone gets the same Whisper transcript check as
everywhere else in this app.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Real bug, not user error: Gradio refuses to serve a file back to the
browser unless it lives under the app's own directory, the system
temp dir, or an explicit allowlist. Every character's reference audio
lives in the-service-station repo, a completely different path, so
selecting *any* character in the Characters tab threw
gradio.exceptions.InvalidPathError server-side (visible in the server
log, not surfaced usefully in the browser's generic error toast).
Fixed with launch(allowed_paths=[SERVICE_STATION_DIR]), the documented
fix for this exact error.
Also: the character dropdowns choices were fixed at server-start time,
so a character created earlier in the same running session did not
show up without manually clicking Refresh. Added a demo.load() hook to
refresh both dropdowns automatically on page load, in addition to
keeping the manual Refresh button for mid-session use without a full
reload.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Real workflow gap: the transcript boxes were read-only, so correcting a
mis-hearing meant manually retyping into the separate Text to speak
field. Reported case: uploading a clip with casual speech ("its
gonna") transcribed as "it is going to be" - if that formalized text
then gets used for generation (by copying it over by hand, the only
option available), the cloned voice ends up speaking noticeably more
formally than intended.
Both transcript boxes are now editable, and thereas a direct "Use this
transcript as Text to speak" button on the reference transcript so a
corrected transcript can be sent straight into generation without
retyping. Also bumped the Whisper model from small to medium for
somewhat better fidelity on casual speech - a real but modest
improvement on the one comparison run, not a guaranteed fix, which is
why the editable box is the actual fix and the model bump is a bonus.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Four more AuK tasks exposed in the Pipeline Studio (vendor/AuK/docs/
COOKBOOK.md sections 3.3, 4.3-4.5), grouped in a collapsed accordion so
they do not crowd the main flow:
- De-accent: remove a regional accent, keep the voice.
- Volume: simple +/-5/10/15 dB.
- Whisper conversion: speech <-> a soft whisper, same content/speaker.
- Nonverbal editing: add/remove breaths, laughs, coughs - directly
relevant to this project's original brief (make a character sound
surprised or laugh). Needs an explicit --gen_seconds like Speed does,
but unlike Speed there is no way to auto-derive the target (add/remove
both change duration unpredictably per AuK's own cookbook) - defaults
to the current reference's own duration if left blank.
Also added a Flash/Base model toggle (ckpts/AuK-Flash, downloaded - same
architecture distilled to 4 fixed steps with CFG=0, ~4.6GB since it
shares the Qwen text encoder with Base rather than needing its own
copy). Verified end-to-end: 37s wall-clock including model load vs the
60-90s+ typical for Base on similar edits. Intended workflow: draft and
iterate fast with Flash, confirm the take you are actually keeping with
Base before promoting a voice or emotion - surfaced as an explicit
reminder in the emotion-candidate flow, which defaults to Flash since
that one is inherently exploratory.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Timbre is actually the most-used AuK task in this whole project - every
character base voice was built from a full free-form description
(accent, age, register, personality in one sentence), but the UI never
had a labeled control for it. It only existed as implicit plumbing
behind the emotion-word wrapping, or something a user could hand-type
into the generic custom-instruction box. Added a clearly labeled
Timbre / voice description field that builds the correct AuK template
directly.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Speed editing was a silent no-op - AuK needs --gen_seconds or it just
preserves the input duration (vendor/AuK/docs/COOKBOOK.md 3.2). Fixed
by computing the target from the current reference duration / speed
factor, with a UI warning that longer references risk a memory-swap
stall on this box (reproduced building a character voice: a 27s target
hung 20+ minutes swapping with 0%% GPU use, a 12s source completed in
seconds).
Added Whisper content verification (faster-whisper, small/CPU/int8)
after every AuK edit and MOSS generation. Motivated by a real incident
building the-service-station cast: a chained AuK edit leaked fragments
of its own instruction text into a character reference, undetected
through a full listen-and-approve round because nothing had
transcribed the reference itself, only generated lines.
History is now per-instruction rather than per-click, so Undo is
granular and a save action can derive a sensibly-scoped
generation_recipe.
New Characters tab: browses the-service-station repo's CHARACTER.md
cards (name/role/status/reference audio/emotion_library), lets you
create a new character (writes a fresh CHARACTER.md, always
status: draft) or generate an emotion candidate for an existing one
(writes to voices/<name>/emotion-cache/<emotion>.wav, same path the
automated pipeline's own fallback uses). Approving anything stays a
manual step - this tab creates candidates for a human to listen to and
promote, it never self-approves.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
--instruction passes through to MOSS-TTS's build_user_message (confirmed
to work for emotion in earlier testing) so callers can steer per-line
delivery. --seed enables reproducible generation, currently the only
way to get determinism out of this pipeline since MOSS-TTS/AuK don't
use a fixed-seed workflow the way Qwen3 did.
Needed for a downstream per-line episode-generation pipeline (character
cards + emotion-tagged script lines).
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Real reference clips are often cheap-mic/phone-call quality. Adds
AuK's dedicated Speech Enhancement task (denoise+dereverb, or a
telephone-effect-specific variant) as a new dropdown, applied first
in the edit chain - before emotion/pitch/speed/custom - since character
edits on noisy source audio just carry the noise through.
Found useful while iterating on real character voices for a sibling
project (the-service-station) using this exact tool.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
AuK's loader (librosa/soundfile) can't read compressed containers like
m4a/mp3 at all - only demucs's loader handles those. Gradio keeps
uploads in their original format, so any edit attempted directly on an
uploaded m4a failed with "Format not recognised."
Normalize to wav via ffmpeg at the single point a reference is set
(use_as_reference), so both AuK and MOSS-TTS downstream always see wav
regardless of what was uploaded.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Gradio app that orchestrates the three existing isolated venvs via
subprocess (their dependencies conflict too much to merge into one
environment - MOSS-TTS needs transformers 5.0.0, AuK needs 4.57.6).
Lets you strip a voice, iteratively edit it (emotion/pitch/speed/custom
instruction) with preview and undo/reset after each step, then generate
new speech in the resulting voice - addressing the trial-and-error
workflow the pitch-overshoot investigation needed.
Also extracts tts/generate.py as a reusable, general-purpose MOSS-TTS
cloning CLI (text/reference/language/output args) rather than one-off
personal scripts.
Verified all three stages end to end via direct function calls before
wiring up the UI.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Edits emotion on an existing recording in place (same speaker, words,
duration) rather than steering style at generation time. Demo runs
"Change the emotion to excited." on the public-domain narrator clip
already used by whisperx/tts.
Found and documented three real environment/packaging gaps on the RTX
3090: a missing audioread dependency, a huggingface_hub/transformers
version conflict from installing the hf CLI, and --cpu_offload being
mandatory (not optional) since AuK's own undoffloaded VRAM figures
exceed 24GB entirely. Also found the practical input-length ceiling is
~25-30s, not the several minutes a podcast episode would need — an 80s
clip OOM'd on attention memory even with offload enabled.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
vocals.mp3 (narrator only) and no_vocals.mp3 (background/instrumental
only), 128kbps MP3, so both are viewable/playable directly in Gitea
without needing local playback or GPU-box access.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Two-stem separation (vocals/instrumental) run on RTX 3090 against the
shared sample clip. Isolated vocals.wav is meant as the clean voice
reference for a future voice-cloning TTS step.
Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
Replaces the 418 MB 1080p source video with a 27 MB 480p derivative and
adds the audio track extracted losslessly from it (AAC stream copy, no
re-encode), which is what the whisperx/demucs/tts demos actually consume.
The full-size source is now gitignored and kept locally only, matching
the existing convention of excluding large binaries from the repo.
Co-Authored-By: Claude Opus 5 <[email protected]>
