steve-adminandClaude Sonnet 5
b5d2f2f19b
AuK's loader (librosa/soundfile) can't read compressed containers like m4a/mp3 at all - only demucs's loader handles those. Gradio keeps uploads in their original format, so any edit attempted directly on an uploaded m4a failed with "Format not recognised." Normalize to wav via ffmpeg at the single point a reference is set (use_as_reference), so both AuK and MOSS-TTS downstream always see wav regardless of what was uploaded. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
webui
A pipeline studio: strip a voice with demucs, edit it with AuK (emotion, pitch, speed, free-form instructions — with preview and undo/reset), then generate new speech in that voice with MOSS-TTS.
Why this exists
Manually chaining demucs → AuK → MOSS-TTS across three separate venvs (via SSH one-off scripts)
worked but made it slow to experiment — especially with AuK's pitch/emotion controls, where you
really want to preview a result and back out if it's wrong (see the pitch-overshoot findings in
auk/README.md) rather than committing to a specific edit blind.
Architecture
A single lightweight Gradio app (app.py) that shells out to each tool's own isolated venv
via subprocess, rather than merging their dependencies into one environment — MOSS-TTS pins
transformers==5.0.0, AuK needs 4.57.6; they can't coexist in one venv. This app's own venv
only needs gradio itself.
Each stage's output becomes the next stage's input by filesystem path; a history list backs
the undo/reset buttons in the voice editor.
Setup
python3 -m venv .venv && . .venv/bin/activate
pip install gradio
Assumes demucs/, auk/, and tts/ (siblings in this repo) already have their own working
venvs and, for auk/, downloaded model weights — see their READMEs.
Running
python3 app.py
Serves on 0.0.0.0:7860. The app automatically calls ComfyUI's /free endpoint (if its queue
is empty) before each heavy step, to reduce GPU contention on a shared box — see the DNS/routing
and GPU-contention notes earlier in this project's history for why that matters here.
Known limitations (inherited from the underlying tools)
- AuK's real usable clip length is ~25-30s, not several minutes — feeding it a long recording
will OOM even with
--cpu_offload(seeauk/README.md). This app doesn't chunk long inputs automatically; keep reference clips short. - AuK's pitch-editing instruction-following is unreliable — even "1 semitone" has measured out at ~4+ semitones of actual shift in testing. The UI surfaces a warning; preview before trusting a pitch edit.
- Each stage is a blocking subprocess call (model load + inference each time) — the UI will appear to hang for anywhere from ~10s to a couple of minutes per click, with no progress bar yet.
- No queueing/locking across browser tabs or concurrent users — this is a single-GPU box, so two simultaneous generations will contend for VRAM.