17 Commits
Author SHA1 Message Date
steve-adminandClaude Sonnet 5 57ae9deb30 Add a Mastering tab and auto-apply toggle for generated voice output
Pedalboard-based chain (warmth EQ, presence dip, compression, light
reverb, hard-ceiling limiter) fixes the spectral imbalance behind
generated TTS sounding tinny - see mastering/master.py docstring and
commit history for the before/after spectral analysis and the A/B tuning
pass against real character clips. Pipeline Studios Generate step can
auto-apply it; the standalone tab works on any clip.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-10-16 11:08:53 +00:00
steve-adminandClaude Sonnet 5 b4fef19993 Add Stable Audio 3 Small-SFX to sound effects tab, with model toggle and save-to-library
Runs entirely on CPU (no VRAM contention with ComfyUI), and sounds
noticeably better than MOSS-SoundEffect per listening test, so it is now
the default model. Toggle keeps MOSS-SoundEffect available as a fallback.
Saved clips land in the-service-station/sfx_library/ on disk, no db/
metadata system per explicit scope.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
2026-10-11 18:09:59 +00:00
steve-adminandClaude Sonnet 5 042c438b8d Move Sound Effects into its own top-level tab
Was a section tacked onto Pipeline Studio; Steve asked for it as a
separate tab instead to keep the existing tabs from getting crowded.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SN15Vxn4svKuhYxFsVtMJf
2026-10-09 15:00:48 +00:00
steve-adminandClaude Sonnet 5 f4e5b9e20a Add MOSS-SoundEffect generation (CLI + webui section)
New tts/generate_sfx.py mirrors generate.py (MOSS-TTS) but for the
MOSS-SoundEffect checkpoint: text-only ambience/SFX generation, no
reference audio or voice cloning. Needed PYTORCH_ALLOC_CONF=
expandable_segments:True baked in - this checkpoints weights alone use
~22.3GB of this boxs ~23.56GB usable VRAM, leaving generate() itself no
room without it (confirmed via OOM otherwise).

webui/app.py gets a new standalone "5. Sound effects" section in the
Pipeline Studio tab (separate from the AuK/MOSS-TTS voice flow - this is
a different generation primitive with no reference/history state).
Verified end-to-end: three real test clips (door creak, market ambience,
car pass) generated cleanly, approved by Steve as "definitely passable,"
then wired into the GUI and confirmed working through app.py itself.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SN15Vxn4svKuhYxFsVtMJf
2026-10-09 14:57:39 +00:00
steve-adminandClaude Sonnet 5 f038ba609d Add VibeVoice 7B result for the 4-person podcast spike
Same script/seed/voices as the 1.5B test, for direct comparison. Notably
more content-faithful (no unscripted insertions), though one line drifted
from a statement to a question. See vibevoice-spike/README.md.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SN15Vxn4svKuhYxFsVtMJf
2026-09-30 22:04:13 +00:00
steve-adminandClaude Sonnet 5 d1bd1ec955 Add VibeVoice 4-person podcast spike test clip
Standalone experiment cloning Kevin/Marge/Terry/Priya into a single
VibeVoice-1.5B multi-speaker generation, for comparison against the
current per-line AuK+MOSS pipeline. See vibevoice-spike/README.md.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01SN15Vxn4svKuhYxFsVtMJf
2026-09-30 21:57:25 +00:00
steve-adminandClaude Sonnet 5 5f97f983fe Add "Clone text as this character" to the Characters tab
Type any text and hear the selected character say it, in their base
voice or any approved/pending emotion. Generalizes two things at once:
testing an emotion candidate with real dialogue instead of the fixed
"Testing the X emotion for Y" line, and just answering "how would
they say this?" for the approved voice directly from the browser
without switching to the Pipeline Studio tab.

list_character_voices() enumerates every voice available for a
character (base + approved emotion_library entries + pending
emotion-cache candidates) - the voice dropdown refreshes whenever the
character selection changes, and also right after generating a new
emotion candidate so it is immediately pickable without reselecting
the character. Every clone gets the same Whisper transcript check as
everywhere else in this app.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-16 10:48:39 +00:00
steve-adminandClaude Sonnet 5 7569cdc322 Fix Characters tab: every selection errored, dropdown went stale
Real bug, not user error: Gradio refuses to serve a file back to the
browser unless it lives under the app's own directory, the system
temp dir, or an explicit allowlist. Every character's reference audio
lives in the-service-station repo, a completely different path, so
selecting *any* character in the Characters tab threw
gradio.exceptions.InvalidPathError server-side (visible in the server
log, not surfaced usefully in the browser's generic error toast).
Fixed with launch(allowed_paths=[SERVICE_STATION_DIR]), the documented
fix for this exact error.

Also: the character dropdowns choices were fixed at server-start time,
so a character created earlier in the same running session did not
show up without manually clicking Refresh. Added a demo.load() hook to
refresh both dropdowns automatically on page load, in addition to
keeping the manual Refresh button for mid-session use without a full
reload.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-16 10:39:12 +00:00
steve-adminandClaude Sonnet 5 2bc1cbf6d6 Make Whisper transcripts editable, add use-as-text-to-speak button
Real workflow gap: the transcript boxes were read-only, so correcting a
mis-hearing meant manually retyping into the separate Text to speak
field. Reported case: uploading a clip with casual speech ("its
gonna") transcribed as "it is going to be" - if that formalized text
then gets used for generation (by copying it over by hand, the only
option available), the cloned voice ends up speaking noticeably more
formally than intended.

Both transcript boxes are now editable, and thereas a direct "Use this
transcript as Text to speak" button on the reference transcript so a
corrected transcript can be sent straight into generation without
retyping. Also bumped the Whisper model from small to medium for
somewhat better fidelity on casual speech - a real but modest
improvement on the one comparison run, not a guaranteed fix, which is
why the editable box is the actual fix and the model bump is a bonus.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-16 09:04:15 +00:00
steve-adminandClaude Sonnet 5 d890736445 Add De-accent/Volume/Whisper/Nonverbal tasks and a Flash/Base model toggle
Four more AuK tasks exposed in the Pipeline Studio (vendor/AuK/docs/
COOKBOOK.md sections 3.3, 4.3-4.5), grouped in a collapsed accordion so
they do not crowd the main flow:

- De-accent: remove a regional accent, keep the voice.
- Volume: simple +/-5/10/15 dB.
- Whisper conversion: speech <-> a soft whisper, same content/speaker.
- Nonverbal editing: add/remove breaths, laughs, coughs - directly
  relevant to this project's original brief (make a character sound
  surprised or laugh). Needs an explicit --gen_seconds like Speed does,
  but unlike Speed there is no way to auto-derive the target (add/remove
  both change duration unpredictably per AuK's own cookbook) - defaults
  to the current reference's own duration if left blank.

Also added a Flash/Base model toggle (ckpts/AuK-Flash, downloaded - same
architecture distilled to 4 fixed steps with CFG=0, ~4.6GB since it
shares the Qwen text encoder with Base rather than needing its own
copy). Verified end-to-end: 37s wall-clock including model load vs the
60-90s+ typical for Base on similar edits. Intended workflow: draft and
iterate fast with Flash, confirm the take you are actually keeping with
Base before promoting a voice or emotion - surfaced as an explicit
reminder in the emotion-candidate flow, which defaults to Flash since
that one is inherently exploratory.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-16 08:02:41 +00:00
steve-adminandClaude Sonnet 5 5dfb64bb0c Add a dedicated Timbre/voice-description field to the pipeline studio
Timbre is actually the most-used AuK task in this whole project - every
character base voice was built from a full free-form description
(accent, age, register, personality in one sentence), but the UI never
had a labeled control for it. It only existed as implicit plumbing
behind the emotion-word wrapping, or something a user could hand-type
into the generic custom-instruction box. Added a clearly labeled
Timbre / voice description field that builds the correct AuK template
directly.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-16 07:51:04 +00:00
steve-adminandClaude Sonnet 5 f780fbda01 Refresh webui: fix Speed no-op, Whisper verification, Characters tab
Speed editing was a silent no-op - AuK needs --gen_seconds or it just
preserves the input duration (vendor/AuK/docs/COOKBOOK.md 3.2). Fixed
by computing the target from the current reference duration / speed
factor, with a UI warning that longer references risk a memory-swap
stall on this box (reproduced building a character voice: a 27s target
hung 20+ minutes swapping with 0%% GPU use, a 12s source completed in
seconds).

Added Whisper content verification (faster-whisper, small/CPU/int8)
after every AuK edit and MOSS generation. Motivated by a real incident
building the-service-station cast: a chained AuK edit leaked fragments
of its own instruction text into a character reference, undetected
through a full listen-and-approve round because nothing had
transcribed the reference itself, only generated lines.

History is now per-instruction rather than per-click, so Undo is
granular and a save action can derive a sensibly-scoped
generation_recipe.

New Characters tab: browses the-service-station repo's CHARACTER.md
cards (name/role/status/reference audio/emotion_library), lets you
create a new character (writes a fresh CHARACTER.md, always
status: draft) or generate an emotion candidate for an existing one
(writes to voices/<name>/emotion-cache/<emotion>.wav, same path the
automated pipeline's own fallback uses). Approving anything stays a
manual step - this tab creates candidates for a human to listen to and
promote, it never self-approves.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-15 23:30:34 +00:00
steve-adminandClaude Sonnet 5 a65532f3b2 Add --instruction and --seed to tts/generate.py
--instruction passes through to MOSS-TTS's build_user_message (confirmed
to work for emotion in earlier testing) so callers can steer per-line
delivery. --seed enables reproducible generation, currently the only
way to get determinism out of this pipeline since MOSS-TTS/AuK don't
use a fixed-seed workflow the way Qwen3 did.

Needed for a downstream per-line episode-generation pipeline (character
cards + emotion-tagged script lines).

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-15 12:21:10 +00:00
steve-adminandClaude Sonnet 5 057e728091 Add speech enhancement as a pipeline step in the voice editor
Real reference clips are often cheap-mic/phone-call quality. Adds
AuK's dedicated Speech Enhancement task (denoise+dereverb, or a
telephone-effect-specific variant) as a new dropdown, applied first
in the edit chain - before emotion/pitch/speed/custom - since character
edits on noisy source audio just carry the noise through.

Found useful while iterating on real character voices for a sibling
project (the-service-station) using this exact tool.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-15 11:55:08 +00:00
steve-adminandClaude Sonnet 5 b5d2f2f19b Fix AuK editor failing on non-wav uploads (e.g. m4a)
AuK's loader (librosa/soundfile) can't read compressed containers like
m4a/mp3 at all - only demucs's loader handles those. Gradio keeps
uploads in their original format, so any edit attempted directly on an
uploaded m4a failed with "Format not recognised."

Normalize to wav via ffmpeg at the single point a reference is set
(use_as_reference), so both AuK and MOSS-TTS downstream always see wav
regardless of what was uploaded.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-13 16:05:29 +00:00
steve-adminandClaude Sonnet 5 e5233433b5 Add pipeline studio web UI (demucs -> AuK -> MOSS-TTS)
Gradio app that orchestrates the three existing isolated venvs via
subprocess (their dependencies conflict too much to merge into one
environment - MOSS-TTS needs transformers 5.0.0, AuK needs 4.57.6).

Lets you strip a voice, iteratively edit it (emotion/pitch/speed/custom
instruction) with preview and undo/reset after each step, then generate
new speech in the resulting voice - addressing the trial-and-error
workflow the pitch-overshoot investigation needed.

Also extracts tts/generate.py as a reusable, general-purpose MOSS-TTS
cloning CLI (text/reference/language/output args) rather than one-off
personal scripts.

Verified all three stages end to end via direct function calls before
wiring up the UI.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-13 15:51:48 +00:00
steve-adminandClaude Sonnet 5 c137fbae0c Add AuK voice-editing demo
Edits emotion on an existing recording in place (same speaker, words,
duration) rather than steering style at generation time. Demo runs
"Change the emotion to excited." on the public-domain narrator clip
already used by whisperx/tts.

Found and documented three real environment/packaging gaps on the RTX
3090: a missing audioread dependency, a huggingface_hub/transformers
version conflict from installing the hf CLI, and --cpu_offload being
mandatory (not optional) since AuK's own undoffloaded VRAM figures
exceed 24GB entirely. Also found the practical input-length ceiling is
~25-30s, not the several minutes a podcast episode would need — an 80s
clip OOM'd on attention memory even with offload enabled.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
2026-09-13 15:24:31 +00:00
15 changed files with 1670 additions and 1 deletions

No files matched your search

+7
View File
@@ -13,6 +13,7 @@ venv/
*.safetensors
models/
weights/
ckpts/
# Audio I/O scratch
*.wav
@@ -33,3 +34,9 @@ media-files/video/*.mp4
# Compressed exports of demucs stems — keep committed for review without
# local playback (the raw separated/*.wav stems stay local, per *.wav above)
!demucs/*.mp3
# Small public-domain demo evidence clip for the auk voice-editing demo
!auk/*.wav
# VibeVoice 4-speaker spike test clip - committed for remote review
!vibevoice-spike/*.mp3
+4 -1
View File
@@ -6,7 +6,10 @@ Audio processing demos: WhisperX, Demucs, and TTS technology proofs-of-concept.
- `whisperx/` — speech-to-text with word-level alignment and diarization
- `demucs/` — source separation (vocals/instrumentals stem splitting)
- `tts/` — text-to-speech service demos
- `tts/` — text-to-speech service demos (voice cloning via MOSS-TTS)
- `auk/` — voice *editing* demos (emotion/pitch/speed/content editing on existing recordings,
as opposed to `tts/`'s clone-and-generate approach)
- `webui/` — interactive pipeline studio chaining demucs -> AuK -> MOSS-TTS with previews
Each subdirectory has its own README with setup and run instructions.
+67
View File
@@ -0,0 +1,67 @@
# auk
Voice-*editing* demos using [AuK](https://github.com/Tencent-Hunyuan/AuK) (Tencent-Hunyuan) — a
1.5B foundation model for speech generation **and editing**. Unlike MOSS-TTS's `instruction`
field (steers style at generation time, from a reference clip), AuK edits an *existing*
recording directly: same speaker, same words, same duration, different delivery. It also does
content editing (rewrite/insert/remove words), pitch/speed/volume editing, de-accenting, and
speech enhancement/separation — all through one natural-language instruction interface.
## Demo
`emotion_edit_demo.py` takes the same public-domain narrator clip used by the whisperx/tts demos
(`tts/reference/narrator_ref_en.wav`, 11.7s, isolated via demucs) and edits its emotion to
"excited" while preserving the voice, words, and exact duration.
### Setup
AuK is vendored outside this repo (large third-party project, editable install), same pattern as
MOSS-TTS:
```bash
git clone https://github.com/Tencent-Hunyuan/AuK.git /path/to/vendor/AuK
python3 -m venv .venv && . .venv/bin/activate
pip install -e /path/to/vendor/AuK
# Two runtime deps AuK needs but doesn't declare (see Gotchas):
pip install audioread "huggingface_hub<1.0,>=0.34.0"
# Model weights (~17GB total)
pip install -U "huggingface_hub[cli]"
hf download tencent/AuK --local-dir ./ckpts/AuK
hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B
```
### Running
```bash
python3 emotion_edit_demo.py
```
Output goes to `narrator_excited.wav` (24kHz mono, ~11.7s — exact source duration preserved).
## Gotchas found running this on an RTX 3090
- **`qwen_omni_utils` needs `audioread`, but nothing declares it as a dependency** — same class
of packaging gap as demucs's missing `numpy` earlier in this repo. Install it explicitly or
the first real inference call fails with `ModuleNotFoundError: No module named 'audioread'`
(model loading itself succeeds first, so this only surfaces once generation actually starts).
- **`pip install "huggingface_hub[cli]"` breaks `transformers`** — it pulls `huggingface_hub`
2.x-style 1.31.0, but `transformers` (installed as an AuK dependency) requires `<1.0`. Only
needed transiently to fetch the weights; downgrade back to `<1.0,>=0.34.0` before running
inference, or install the CLI in a separate throwaway environment.
- **`--cpu_offload` is not optional on a 24GB card** — AuK's own documented VRAM figures
(24.78–25.00GiB *without* offload) exceed the RTX 3090's 24GB entirely, even dedicating the
whole card. Always pass `--cpu_offload` (or `cpu_offload=True` in the Python API).
- **Real practical length limit is much shorter than it looks: ~25–30s, not 5 minutes.** AuK's
ComfyUI docs mention a "30-second source/reference-plus-target budget" as if it were a
ComfyUI-specific constraint — it isn't. A 25s clip worked fine even with `--cpu_offload`; an
80s clip OOM'd (tried to allocate another 8.9GB mid-attention on top of the ~17GB already in
use) because attention memory scales fast with sequence length. Editing a full multi-minute
podcast take means chunking it into ≤~25s segments and concatenating, not a single pass —
worth planning for prosody continuity across chunk boundaries.
- **Default checkpoint path** baked into `auk-infer`/`AukInfer` points at wherever you cloned
the vendor repo, not this directory — always pass `--ckpt`/`--qwen_path` (CLI) or
`config_path`/`ckpt_path`/`qwen_path` (Python API) explicitly.
- **Output is 24kHz mono**, notably lower fidelity than MOSS-TTS's 48kHz stereo output — worth
factoring in if AuK-edited audio needs to sit alongside MOSS-TTS output in the same pipeline.
+47
View File
@@ -0,0 +1,47 @@
"""
Voice-editing demo using Tencent's AuK (https://github.com/Tencent-Hunyuan/AuK).
Unlike MOSS-TTS's `instruction` field (steers style at generation time),
AuK edits emotion on an *existing* recording directly - same speaker, same
words, same duration, different delivery. Demonstrates on the public-domain
narrator clip already used by the whisperx/tts demos.
Run from the auk/ directory with its .venv activated.
"""
from pathlib import Path
from auk.infer.infer_auk import AukInfer, save_audio
HERE = Path(__file__).parent
CONFIG = HERE / "ckpts" / "AuK" / "config.yaml"
CKPT = HERE / "ckpts" / "AuK" / "auk_base.safetensors"
QWEN_PATH = HERE / "ckpts" / "Qwen2.5-Omni-3B"
INPUT_AUDIO = HERE.parent / "tts" / "reference" / "narrator_ref_en.wav"
OUTPUT_PATH = HERE / "narrator_excited.wav"
INSTRUCTION = "Change the emotion to excited."
def main() -> None:
engine = AukInfer(
str(CONFIG), str(CKPT), qwen_path=str(QWEN_PATH), cpu_offload=True
)
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": INSTRUCTION},
{"type": "audio", "audio": str(INPUT_AUDIO)},
],
}
]
audio, sr = engine.generate(messages)
save_audio(audio, sr, str(OUTPUT_PATH))
print(f"[INFO] Saved {OUTPUT_PATH}")
if __name__ == "__main__":
main()
Binary file not shown.
+82
View File
@@ -0,0 +1,82 @@
"""
Mastering chain for generated TTS/voice clips: fixes the "tinny" spectral
imbalance common to neural vocoder output (excess 3-6kHz presence energy,
starved low end relative to a naturally-recorded voice - see the project's
spectral comparison: raw recordings skew toward 200-1000Hz, generated clips
dump ~35% of their energy into 3-6kHz) and adds the density/space that
reads as "produced" rather than dry/synthetic.
Chain order matters: EQ before compression (so the compressor reacts to
the corrected tonal balance, not the raw tinny one), reverb after
compression (so the room tail doesn't get squashed), limiter last.
"""
import argparse
import numpy as np
import soundfile as sf
from pedalboard import (
Compressor,
HighpassFilter,
HighShelfFilter,
Limiter,
LowShelfFilter,
PeakFilter,
Pedalboard,
Reverb,
)
def build_chain(
warmth_db: float,
presence_cut_db: float,
air_db: float,
comp_ratio: float,
reverb_wet: float,
) -> Pedalboard:
return Pedalboard([
HighpassFilter(cutoff_frequency_hz=60),
LowShelfFilter(cutoff_frequency_hz=150, gain_db=warmth_db, q=0.7),
PeakFilter(cutoff_frequency_hz=3800, gain_db=presence_cut_db, q=1.0),
HighShelfFilter(cutoff_frequency_hz=6500, gain_db=air_db, q=0.7),
Compressor(threshold_db=-20, ratio=comp_ratio, attack_ms=12, release_ms=120),
Reverb(room_size=0.08, damping=0.85, wet_level=reverb_wet, dry_level=1.0 - reverb_wet),
Limiter(threshold_db=-1.0, release_ms=100),
])
def main() -> None:
parser = argparse.ArgumentParser(description="Apply a studio mastering chain to a voice clip")
parser.add_argument("--input", required=True)
parser.add_argument("--output", required=True)
parser.add_argument("--warmth-db", type=float, default=5.5, help="Low-shelf boost @150Hz")
parser.add_argument("--presence-cut-db", type=float, default=-5.0, help="Peak cut @3.8kHz (the tinny band)")
parser.add_argument("--air-db", type=float, default=-4.5, help="High-shelf @6.5kHz, for softness")
parser.add_argument("--comp-ratio", type=float, default=2.5)
parser.add_argument("--reverb-wet", type=float, default=0.025)
args = parser.parse_args()
audio, sr = sf.read(args.input, always_2d=True)
audio = audio.T.astype(np.float32) # pedalboard wants (channels, samples)
board = build_chain(
args.warmth_db, args.presence_cut_db, args.air_db, args.comp_ratio, args.reverb_wet
)
processed = board(audio, sr)
# pedalboard's Limiter is attack/release-based, not a true brick-wall
# limiter - fast transients can still punch through its threshold (seen
# in practice: in-memory peaks hit 0dBFS despite threshold_db=-1.0,
# producing audible clipping distortion on playback). Hard safety
# ceiling here guarantees the written file never clips regardless.
target_peak = 10 ** (-1.0 / 20)
peak = np.max(np.abs(processed))
if peak > target_peak:
processed = processed * (target_peak / peak)
sf.write(args.output, processed.T, sr)
print(f"[INFO] Saved {args.output}")
if __name__ == "__main__":
main()
+32
View File
@@ -0,0 +1,32 @@
"""
Stable Audio 3 Small-SFX CLI: generate a sound effect/ambience clip from a
text description. Runs on CPU by design - this checkpoint is the
"lightweight, no GPU required" model in Stable Audio 3's lineup, so it
never contends with ComfyUI/other models for VRAM (unlike generate_sfx.py's
MOSS-SoundEffect, which needs the 3090 and ComfyUI's memory freed first).
"""
import argparse
import torchaudio
from stable_audio_3 import StableAudioModel
def main() -> None:
parser = argparse.ArgumentParser(description="Stable Audio 3 Small-SFX text-to-sound-effect")
parser.add_argument("--text", required=True, help="Description of the sound effect/ambience")
parser.add_argument("--output", required=True, help="Output wav path")
parser.add_argument("--duration", type=float, default=10.0, help="Clip duration in seconds (max 120)")
parser.add_argument("--seed", type=int, default=-1, help="Random seed, or -1 for random")
args = parser.parse_args()
model = StableAudioModel.from_pretrained("small-sfx", device="cpu")
sample_rate = model.model.sample_rate
audio = model.generate(prompt=args.text, duration=args.duration, seed=args.seed)
torchaudio.save(args.output, audio[0].cpu(), sample_rate)
print(f"[INFO] Saved {args.output}")
if __name__ == "__main__":
main()
+109
View File
@@ -0,0 +1,109 @@
"""
Generic MOSS-TTS zero-shot cloning CLI: clone the voice in --reference and
say --text in it. Used directly, and as the final stage of webui/app.py.
"""
import argparse
import importlib.util
import soundfile as sf
import torch
from transformers import AutoModel, AutoProcessor
PRETRAINED = "OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5"
def resolve_attn_implementation(device: str, dtype: torch.dtype) -> str:
if (
device == "cuda"
and importlib.util.find_spec("flash_attn") is not None
and dtype in {torch.float16, torch.bfloat16}
):
major, _ = torch.cuda.get_device_capability()
if major >= 8:
return "flash_attention_2"
return "sdpa" if device == "cuda" else "eager"
def main() -> None:
parser = argparse.ArgumentParser(description="MOSS-TTS zero-shot voice cloning")
parser.add_argument("--text", required=True, help="Text to synthesize")
parser.add_argument("--reference", required=True, help="Reference audio path")
parser.add_argument("--output", required=True, help="Output wav path")
parser.add_argument("--language", default="English", help="Language tag")
parser.add_argument(
"--instruction", default=None,
help='Style/delivery instruction (e.g. "Change the emotion to excited."). '
"Undocumented MOSS-TTS field for this checkpoint - confirmed to work for "
"emotion in testing, passed through as-is.",
)
parser.add_argument(
"--seed", type=int, default=None,
help="Random seed for reproducibility. Unset = nondeterministic.",
)
parser.add_argument("--audio_temperature", type=float, default=1.7)
parser.add_argument("--audio_top_p", type=float, default=0.8)
parser.add_argument("--audio_top_k", type=int, default=25)
parser.add_argument("--audio_repetition_penalty", type=float, default=1.0)
parser.add_argument("--max_new_tokens", type=int, default=4096)
args = parser.parse_args()
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
if args.seed is not None:
torch.manual_seed(args.seed)
if device == "cuda":
torch.cuda.manual_seed_all(args.seed)
torch.backends.cuda.enable_cudnn_sdp(False)
torch.backends.cuda.enable_flash_sdp(True)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_math_sdp(True)
attn_implementation = resolve_attn_implementation(device, dtype)
print(f"[INFO] attn_implementation={attn_implementation}")
processor = AutoProcessor.from_pretrained(PRETRAINED, trust_remote_code=True)
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
model = AutoModel.from_pretrained(
PRETRAINED,
trust_remote_code=True,
attn_implementation=attn_implementation,
dtype=dtype,
).to(device)
model.eval()
conversation = [
processor.build_user_message(
text=args.text,
language=args.language,
reference=[args.reference],
instruction=args.instruction,
)
]
with torch.no_grad():
batch = processor([conversation], mode="generation")
outputs = model.generate(
input_ids=batch["input_ids"].to(device),
attention_mask=batch["attention_mask"].to(device),
max_new_tokens=args.max_new_tokens,
do_sample=True,
audio_temperature=args.audio_temperature,
audio_top_p=args.audio_top_p,
audio_top_k=args.audio_top_k,
audio_repetition_penalty=args.audio_repetition_penalty,
)
message = processor.decode(outputs)[0]
audio = message.audio_codes_list[0]
data = audio.detach().cpu().to(torch.float32).numpy()
if data.ndim == 2:
data = data.T
sf.write(args.output, data, processor.model_config.sampling_rate)
print(f"[INFO] Saved {args.output}")
if __name__ == "__main__":
main()
+99
View File
@@ -0,0 +1,99 @@
"""
MOSS-SoundEffect CLI: generate an ambient/sound-effect clip from a text
description. Sibling of generate.py (MOSS-TTS voice cloning) - same
backend setup and model-loading pattern, different checkpoint and a
text-only build_user_message call (no reference audio, no cloning).
"""
import argparse
import importlib.util
import os
# This checkpoint's weights alone use ~22.3GB of this box's ~23.56GB usable
# VRAM, leaving generate()'s own small allocations (a vocab-sized mask
# tensor, etc.) no room without this - confirmed via OOM otherwise. Must be
# set before torch's CUDA allocator initializes, so it's set here rather
# than relying on the caller's shell environment.
os.environ.setdefault("PYTORCH_ALLOC_CONF", "expandable_segments:True")
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
import soundfile as sf
import torch
from transformers import AutoModel, AutoProcessor
PRETRAINED = "OpenMOSS-Team/MOSS-SoundEffect"
def resolve_attn_implementation(device: str, dtype: torch.dtype) -> str:
if (
device == "cuda"
and importlib.util.find_spec("flash_attn") is not None
and dtype in {torch.float16, torch.bfloat16}
):
major, _ = torch.cuda.get_device_capability()
if major >= 8:
return "flash_attention_2"
return "sdpa" if device == "cuda" else "eager"
def main() -> None:
parser = argparse.ArgumentParser(description="MOSS-SoundEffect text-to-sound-effect")
parser.add_argument("--text", required=True, help="Description of the sound effect/ambience")
parser.add_argument("--output", required=True, help="Output wav path")
parser.add_argument(
"--seed", type=int, default=None,
help="Random seed for reproducibility. Unset = nondeterministic.",
)
parser.add_argument("--max_new_tokens", type=int, default=4096)
args = parser.parse_args()
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
if args.seed is not None:
torch.manual_seed(args.seed)
if device == "cuda":
torch.cuda.manual_seed_all(args.seed)
torch.backends.cuda.enable_cudnn_sdp(False)
torch.backends.cuda.enable_flash_sdp(True)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_math_sdp(True)
attn_implementation = resolve_attn_implementation(device, dtype)
print(f"[INFO] attn_implementation={attn_implementation}")
processor = AutoProcessor.from_pretrained(PRETRAINED, trust_remote_code=True)
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
model = AutoModel.from_pretrained(
PRETRAINED,
trust_remote_code=True,
attn_implementation=attn_implementation,
dtype=dtype,
).to(device)
model.eval()
conversation = [processor.build_user_message(ambient_sound=args.text)]
with torch.no_grad():
batch = processor([conversation], mode="generation")
# This checkpoint's generate() doesn't accept the TTS model's sampling
# kwargs (do_sample/audio_temperature/etc raise TypeError) - matches
# the official README example, which only passes these three.
outputs = model.generate(
input_ids=batch["input_ids"].to(device),
attention_mask=batch["attention_mask"].to(device),
max_new_tokens=args.max_new_tokens,
)
message = processor.decode(outputs)[0]
audio = message.audio_codes_list[0]
data = audio.detach().cpu().to(torch.float32).numpy()
if data.ndim == 2:
data = data.T
sf.write(args.output, data, processor.model_config.sampling_rate)
print(f"[INFO] Saved {args.output}")
if __name__ == "__main__":
main()
+52
View File
@@ -0,0 +1,52 @@
# VibeVoice (community fork) 4-person podcast spike
Standalone experiment, not integrated into the webui or the production
pipeline. Tests whether VibeVoice (https://github.com/vibevoice-community/VibeVoice)
can generate a 4-speaker podcast scene in one pass, cloning our four
established character voices (Kevin, Marge, Terry, Priya) from their
existing approved `reference_audio` in the-service-station repo.
- `scripts/trailer_promo_4speaker.txt` - the test script: the real
"sausage rolls" trailer scene from `the-service-station/scripts/trailer-promo.md`,
converted from our `[Character|tags]` format to VibeVoices required
`Speaker N: ...` plain-text format (tags dropped - VibeVoice has no
per-line emotion/timing control).
- `trailer_promo_4speaker_vibevoice-1.5b.mp3` - the generated result
(VibeVoice-1.5B, seed 42). 47s of audio generated in 14.5s (RTF 0.31x).
Setup lives outside this repo (not committed): `.venv/` here is a fresh
isolated venv (`pip install -e /root/projects/VibeVoice`, pinned
`transformers==4.51.3`), pointed at the already-downloaded
`VibeVoice-1.5B` weights under `/root/projects/ComfyUI/models/tts/VibeVoice/`.
## Findings (2026-09-30)
Whisper-transcript check against the script: mostly faithful, not
word-perfect. Two small unscripted insertions ("I am here!" before Priyas
line, "For me," before Marges reply) and one substitution ("He will." ->
"You will."). No repetition/collapse like the AuK chain-corruption bug -
just minor ad-libbing, not catastrophic.
No audible AI disclaimer or obvious background music in this run - both
are documented risks for this model family (see
`the-service-station/research/vibevoice-2026-08-21.md` and
`vibevoice-community-fork-2026-08-21.md`), so treat this one clean run as
encouraging, not conclusive.
Steves call after listening: pursue 7B (weights already downloaded, same
venv/script, swap `--model_path`) or not worth it given the limitations
the prior research already flagged.
## 7B result (2026-09-30, same script/seed/voices)
`trailer_promo_4speaker_vibevoice-7b.mp3` - VibeVoice-7B, same seed (42),
same script, same reference voices. 34.8s of audio generated in 22.2s
(RTF 0.64x - still faster than real-time, just less headroom than 1.5B).
Whisper-transcript check: cleaner than 1.5B - neither of 1.5Bs unscripted
insertions ("I am here!", "For me,") appeared, and "He will." came through
exactly right this time (1.5B had drifted it to "You will."). One real
deviation: Marges defensive line "I get one fact wrong per episode."
came out as a question, "Can I get one fact wrong per episode?" - changes
the delivery/meaning, not just wording. Still no audible disclaimer and
nothing that reads as injected background music.
@@ -0,0 +1,9 @@
Speaker 1: The Service Station. Three facts, one car park, and absolutely no reason to be in a hurry.
Speaker 2: Except for me, obviously. I'm always in a hurry.
Speaker 3: I had a Sinclair Spectrum in 1984, and I will find a way to bring it up.
Speaker 1: He will.
Speaker 4: I still don't understand why the sausage rolls are relevant, but here we are.
Speaker 2: They're always relevant, Priya.
Speaker 1: Every week: one underdog, one argument, and one fact Marge gets wrong.
Speaker 2: I get one fact wrong per episode. Tops.
Speaker 3: It's usually about football.
+56
View File
@@ -0,0 +1,56 @@
# webui
A pipeline studio: strip a voice with **demucs**, edit it with **AuK** (emotion, pitch, speed,
free-form instructions — with preview and undo/reset), then generate new speech in that voice
with **MOSS-TTS**.
## Why this exists
Manually chaining demucs → AuK → MOSS-TTS across three separate venvs (via SSH one-off scripts)
worked but made it slow to experiment — especially with AuK's pitch/emotion controls, where you
really want to preview a result and back out if it's wrong (see the pitch-overshoot findings in
`auk/README.md`) rather than committing to a specific edit blind.
## Architecture
A single lightweight Gradio app (`app.py`) that **shells out** to each tool's own isolated venv
via subprocess, rather than merging their dependencies into one environment — MOSS-TTS pins
`transformers==5.0.0`, AuK needs `4.57.6`; they can't coexist in one venv. This app's own venv
only needs `gradio` itself.
Each stage's output becomes the next stage's input by filesystem path; a `history` list backs
the undo/reset buttons in the voice editor.
## Setup
```bash
python3 -m venv .venv && . .venv/bin/activate
pip install gradio
```
Assumes `demucs/`, `auk/`, and `tts/` (siblings in this repo) already have their own working
venvs and, for `auk/`, downloaded model weights — see their READMEs.
## Running
```bash
python3 app.py
```
Serves on `0.0.0.0:7860`. The app automatically calls ComfyUI's `/free` endpoint (if its queue
is empty) before each heavy step, to reduce GPU contention on a shared box — see the DNS/routing
and GPU-contention notes earlier in this project's history for why that matters here.
## Known limitations (inherited from the underlying tools)
- **AuK's real usable clip length is ~25-30s**, not several minutes — feeding it a long recording
will OOM even with `--cpu_offload` (see `auk/README.md`). This app doesn't chunk long inputs
automatically; keep reference clips short.
- **AuK's pitch-editing instruction-following is unreliable** — even "1 semitone" has measured
out at ~4+ semitones of actual shift in testing. The UI surfaces a warning; preview before
trusting a pitch edit.
- Each stage is a blocking subprocess call (model load + inference each time) — the UI will
appear to hang for anywhere from ~10s to a couple of minutes per click, with no progress bar
yet.
- No queueing/locking across browser tabs or concurrent users — this is a single-GPU box, so two
simultaneous generations will contend for VRAM.
+1106
View File
File diff suppressed because it is too large. Load diff