Compare commits
2
Commits
5006475c8d
..
tts
| Author | SHA1 | Date | |
|---|---|---|---|
  steve-adminandClaude Sonnet 5
|
e70f517fec | ||
|
steve-adminandClaude Sonnet 5
|
7c4f226e17 |
No files matched your search
+5
-10
@@ -13,7 +13,6 @@ venv/
|
||||
*.safetensors
|
||||
models/
|
||||
weights/
|
||||
ckpts/
|
||||
|
||||
# Audio I/O scratch
|
||||
*.wav
|
||||
@@ -31,12 +30,8 @@ outputs/
|
||||
media-files/video/*.mp4
|
||||
!media-files/video/*-480p.mp4
|
||||
|
||||
# Compressed exports of demucs stems — keep committed for review without
|
||||
# local playback (the raw separated/*.wav stems stay local, per *.wav above)
|
||||
!demucs/*.mp3
|
||||
|
||||
# Small public-domain demo evidence clip for the auk voice-editing demo
|
||||
!auk/*.wav
|
||||
|
||||
# VibeVoice 4-speaker spike test clip - committed for remote review
|
||||
!vibevoice-spike/*.mp3
|
||||
# Small demo evidence clips — keep committed so results are reviewable
|
||||
# without local playback (unlike demucs's separated/ stems, which stay local)
|
||||
!tts/reference/*.wav
|
||||
!tts/output/
|
||||
!tts/output/*.wav
|
||||
@@ -6,10 +6,7 @@ Audio processing demos: WhisperX, Demucs, and TTS technology proofs-of-concept.
|
||||
|
||||
- `whisperx/` — speech-to-text with word-level alignment and diarization
|
||||
- `demucs/` — source separation (vocals/instrumentals stem splitting)
|
||||
- `tts/` — text-to-speech service demos (voice cloning via MOSS-TTS)
|
||||
- `auk/` — voice *editing* demos (emotion/pitch/speed/content editing on existing recordings,
|
||||
as opposed to `tts/`'s clone-and-generate approach)
|
||||
- `webui/` — interactive pipeline studio chaining demucs -> AuK -> MOSS-TTS with previews
|
||||
- `tts/` — text-to-speech service demos
|
||||
|
||||
Each subdirectory has its own README with setup and run instructions.
|
||||
|
||||
@@ -26,5 +23,10 @@ Chain the three demos together on `media-files/` into an end-to-end pipeline:
|
||||
3. `tts` — clone the original speaker's voice (from the isolated vocal stem) and synthesize
|
||||
the translated text in that voice, then remix with the original instrumental stem
|
||||
|
||||
Stem separation and transcription are already working end to end (see `demucs/README.md` and
|
||||
`whisperx/README.md`); translation and voice-cloning TTS are not yet implemented.
|
||||
All three stages now have a working, if narrow, proof of concept:
|
||||
|
||||
- Stem separation (`demucs/README.md`) and transcription (`whisperx/README.md`) run end to end
|
||||
on the full sample clip.
|
||||
- Translation + voice-cloning TTS (`tts/README.md`) works for a single short sentence
|
||||
(English → Spanish, cloned onto the isolated narrator voice via MOSS-TTS). Not yet wired up
|
||||
to run over the full transcript or to remix with the instrumental stem.
|
||||
@@ -1,67 +0,0 @@
|
||||
# auk
|
||||
|
||||
Voice-*editing* demos using [AuK](https://github.com/Tencent-Hunyuan/AuK) (Tencent-Hunyuan) — a
|
||||
1.5B foundation model for speech generation **and editing**. Unlike MOSS-TTS's `instruction`
|
||||
field (steers style at generation time, from a reference clip), AuK edits an *existing*
|
||||
recording directly: same speaker, same words, same duration, different delivery. It also does
|
||||
content editing (rewrite/insert/remove words), pitch/speed/volume editing, de-accenting, and
|
||||
speech enhancement/separation — all through one natural-language instruction interface.
|
||||
|
||||
## Demo
|
||||
|
||||
`emotion_edit_demo.py` takes the same public-domain narrator clip used by the whisperx/tts demos
|
||||
(`tts/reference/narrator_ref_en.wav`, 11.7s, isolated via demucs) and edits its emotion to
|
||||
"excited" while preserving the voice, words, and exact duration.
|
||||
|
||||
### Setup
|
||||
|
||||
AuK is vendored outside this repo (large third-party project, editable install), same pattern as
|
||||
MOSS-TTS:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Tencent-Hunyuan/AuK.git /path/to/vendor/AuK
|
||||
python3 -m venv .venv && . .venv/bin/activate
|
||||
pip install -e /path/to/vendor/AuK
|
||||
|
||||
# Two runtime deps AuK needs but doesn't declare (see Gotchas):
|
||||
pip install audioread "huggingface_hub<1.0,>=0.34.0"
|
||||
|
||||
# Model weights (~17GB total)
|
||||
pip install -U "huggingface_hub[cli]"
|
||||
hf download tencent/AuK --local-dir ./ckpts/AuK
|
||||
hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B
|
||||
```
|
||||
|
||||
### Running
|
||||
|
||||
```bash
|
||||
python3 emotion_edit_demo.py
|
||||
```
|
||||
|
||||
Output goes to `narrator_excited.wav` (24kHz mono, ~11.7s — exact source duration preserved).
|
||||
|
||||
## Gotchas found running this on an RTX 3090
|
||||
|
||||
- **`qwen_omni_utils` needs `audioread`, but nothing declares it as a dependency** — same class
|
||||
of packaging gap as demucs's missing `numpy` earlier in this repo. Install it explicitly or
|
||||
the first real inference call fails with `ModuleNotFoundError: No module named 'audioread'`
|
||||
(model loading itself succeeds first, so this only surfaces once generation actually starts).
|
||||
- **`pip install "huggingface_hub[cli]"` breaks `transformers`** — it pulls `huggingface_hub`
|
||||
2.x-style 1.31.0, but `transformers` (installed as an AuK dependency) requires `<1.0`. Only
|
||||
needed transiently to fetch the weights; downgrade back to `<1.0,>=0.34.0` before running
|
||||
inference, or install the CLI in a separate throwaway environment.
|
||||
- **`--cpu_offload` is not optional on a 24GB card** — AuK's own documented VRAM figures
|
||||
(24.78–25.00GiB *without* offload) exceed the RTX 3090's 24GB entirely, even dedicating the
|
||||
whole card. Always pass `--cpu_offload` (or `cpu_offload=True` in the Python API).
|
||||
- **Real practical length limit is much shorter than it looks: ~25–30s, not 5 minutes.** AuK's
|
||||
ComfyUI docs mention a "30-second source/reference-plus-target budget" as if it were a
|
||||
ComfyUI-specific constraint — it isn't. A 25s clip worked fine even with `--cpu_offload`; an
|
||||
80s clip OOM'd (tried to allocate another 8.9GB mid-attention on top of the ~17GB already in
|
||||
use) because attention memory scales fast with sequence length. Editing a full multi-minute
|
||||
podcast take means chunking it into ≤~25s segments and concatenating, not a single pass —
|
||||
worth planning for prosody continuity across chunk boundaries.
|
||||
- **Default checkpoint path** baked into `auk-infer`/`AukInfer` points at wherever you cloned
|
||||
the vendor repo, not this directory — always pass `--ckpt`/`--qwen_path` (CLI) or
|
||||
`config_path`/`ckpt_path`/`qwen_path` (Python API) explicitly.
|
||||
- **Output is 24kHz mono**, notably lower fidelity than MOSS-TTS's 48kHz stereo output — worth
|
||||
factoring in if AuK-edited audio needs to sit alongside MOSS-TTS output in the same pipeline.
|
||||
@@ -1,47 +0,0 @@
|
||||
"""
|
||||
Voice-editing demo using Tencent's AuK (https://github.com/Tencent-Hunyuan/AuK).
|
||||
|
||||
Unlike MOSS-TTS's `instruction` field (steers style at generation time),
|
||||
AuK edits emotion on an *existing* recording directly - same speaker, same
|
||||
words, same duration, different delivery. Demonstrates on the public-domain
|
||||
narrator clip already used by the whisperx/tts demos.
|
||||
|
||||
Run from the auk/ directory with its .venv activated.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from auk.infer.infer_auk import AukInfer, save_audio
|
||||
|
||||
HERE = Path(__file__).parent
|
||||
CONFIG = HERE / "ckpts" / "AuK" / "config.yaml"
|
||||
CKPT = HERE / "ckpts" / "AuK" / "auk_base.safetensors"
|
||||
QWEN_PATH = HERE / "ckpts" / "Qwen2.5-Omni-3B"
|
||||
INPUT_AUDIO = HERE.parent / "tts" / "reference" / "narrator_ref_en.wav"
|
||||
OUTPUT_PATH = HERE / "narrator_excited.wav"
|
||||
|
||||
INSTRUCTION = "Change the emotion to excited."
|
||||
|
||||
|
||||
def main() -> None:
|
||||
engine = AukInfer(
|
||||
str(CONFIG), str(CKPT), qwen_path=str(QWEN_PATH), cpu_offload=True
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": INSTRUCTION},
|
||||
{"type": "audio", "audio": str(INPUT_AUDIO)},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
audio, sr = engine.generate(messages)
|
||||
save_audio(audio, sr, str(OUTPUT_PATH))
|
||||
print(f"[INFO] Saved {OUTPUT_PATH}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Binary file not shown.
@@ -17,10 +17,6 @@ Took ~12s for 11 minutes of audio. Outputs `vocals.wav` and `no_vocals.wav` (44.
|
||||
duration matches source) under `separated/htdemucs/<track-name>/` — gitignored, left on the GPU
|
||||
box at `/root/projects/audio-demo/repo/demucs/separated/`.
|
||||
|
||||
Full-length MP3 versions (128kbps, ~10MB each) are committed as `vocals.mp3` and
|
||||
`no_vocals.mp3` in this directory so both stems are reviewable directly from Gitea without
|
||||
needing local playback or GPU-box access.
|
||||
|
||||
**Known gotcha:** demucs 4.1.0's package metadata doesn't declare `numpy` as a dependency even
|
||||
though `demucs/transformer.py` imports it directly — `pip install demucs` alone leaves the env
|
||||
broken (`ModuleNotFoundError: No module named 'numpy'` at runtime). Install `numpy` explicitly
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -1,82 +0,0 @@
|
||||
"""
|
||||
Mastering chain for generated TTS/voice clips: fixes the "tinny" spectral
|
||||
imbalance common to neural vocoder output (excess 3-6kHz presence energy,
|
||||
starved low end relative to a naturally-recorded voice - see the project's
|
||||
spectral comparison: raw recordings skew toward 200-1000Hz, generated clips
|
||||
dump ~35% of their energy into 3-6kHz) and adds the density/space that
|
||||
reads as "produced" rather than dry/synthetic.
|
||||
|
||||
Chain order matters: EQ before compression (so the compressor reacts to
|
||||
the corrected tonal balance, not the raw tinny one), reverb after
|
||||
compression (so the room tail doesn't get squashed), limiter last.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
||||
import numpy as np
|
||||
import soundfile as sf
|
||||
from pedalboard import (
|
||||
Compressor,
|
||||
HighpassFilter,
|
||||
HighShelfFilter,
|
||||
Limiter,
|
||||
LowShelfFilter,
|
||||
PeakFilter,
|
||||
Pedalboard,
|
||||
Reverb,
|
||||
)
|
||||
|
||||
|
||||
def build_chain(
|
||||
warmth_db: float,
|
||||
presence_cut_db: float,
|
||||
air_db: float,
|
||||
comp_ratio: float,
|
||||
reverb_wet: float,
|
||||
) -> Pedalboard:
|
||||
return Pedalboard([
|
||||
HighpassFilter(cutoff_frequency_hz=60),
|
||||
LowShelfFilter(cutoff_frequency_hz=150, gain_db=warmth_db, q=0.7),
|
||||
PeakFilter(cutoff_frequency_hz=3800, gain_db=presence_cut_db, q=1.0),
|
||||
HighShelfFilter(cutoff_frequency_hz=6500, gain_db=air_db, q=0.7),
|
||||
Compressor(threshold_db=-20, ratio=comp_ratio, attack_ms=12, release_ms=120),
|
||||
Reverb(room_size=0.08, damping=0.85, wet_level=reverb_wet, dry_level=1.0 - reverb_wet),
|
||||
Limiter(threshold_db=-1.0, release_ms=100),
|
||||
])
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Apply a studio mastering chain to a voice clip")
|
||||
parser.add_argument("--input", required=True)
|
||||
parser.add_argument("--output", required=True)
|
||||
parser.add_argument("--warmth-db", type=float, default=5.5, help="Low-shelf boost @150Hz")
|
||||
parser.add_argument("--presence-cut-db", type=float, default=-5.0, help="Peak cut @3.8kHz (the tinny band)")
|
||||
parser.add_argument("--air-db", type=float, default=-4.5, help="High-shelf @6.5kHz, for softness")
|
||||
parser.add_argument("--comp-ratio", type=float, default=2.5)
|
||||
parser.add_argument("--reverb-wet", type=float, default=0.025)
|
||||
args = parser.parse_args()
|
||||
|
||||
audio, sr = sf.read(args.input, always_2d=True)
|
||||
audio = audio.T.astype(np.float32) # pedalboard wants (channels, samples)
|
||||
|
||||
board = build_chain(
|
||||
args.warmth_db, args.presence_cut_db, args.air_db, args.comp_ratio, args.reverb_wet
|
||||
)
|
||||
processed = board(audio, sr)
|
||||
|
||||
# pedalboard's Limiter is attack/release-based, not a true brick-wall
|
||||
# limiter - fast transients can still punch through its threshold (seen
|
||||
# in practice: in-memory peaks hit 0dBFS despite threshold_db=-1.0,
|
||||
# producing audible clipping distortion on playback). Hard safety
|
||||
# ceiling here guarantees the written file never clips regardless.
|
||||
target_peak = 10 ** (-1.0 / 20)
|
||||
peak = np.max(np.abs(processed))
|
||||
if peak > target_peak:
|
||||
processed = processed * (target_peak / peak)
|
||||
|
||||
sf.write(args.output, processed.T, sr)
|
||||
print(f"[INFO] Saved {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,32 +0,0 @@
|
||||
"""
|
||||
Stable Audio 3 Small-SFX CLI: generate a sound effect/ambience clip from a
|
||||
text description. Runs on CPU by design - this checkpoint is the
|
||||
"lightweight, no GPU required" model in Stable Audio 3's lineup, so it
|
||||
never contends with ComfyUI/other models for VRAM (unlike generate_sfx.py's
|
||||
MOSS-SoundEffect, which needs the 3090 and ComfyUI's memory freed first).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
|
||||
import torchaudio
|
||||
from stable_audio_3 import StableAudioModel
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="Stable Audio 3 Small-SFX text-to-sound-effect")
|
||||
parser.add_argument("--text", required=True, help="Description of the sound effect/ambience")
|
||||
parser.add_argument("--output", required=True, help="Output wav path")
|
||||
parser.add_argument("--duration", type=float, default=10.0, help="Clip duration in seconds (max 120)")
|
||||
parser.add_argument("--seed", type=int, default=-1, help="Random seed, or -1 for random")
|
||||
args = parser.parse_args()
|
||||
|
||||
model = StableAudioModel.from_pretrained("small-sfx", device="cpu")
|
||||
sample_rate = model.model.sample_rate
|
||||
|
||||
audio = model.generate(prompt=args.text, duration=args.duration, seed=args.seed)
|
||||
torchaudio.save(args.output, audio[0].cpu(), sample_rate)
|
||||
print(f"[INFO] Saved {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,3 +1,55 @@
|
||||
# tts
|
||||
|
||||
Text-to-speech service demos and proofs-of-concept.
|
||||
|
||||
## Voice-clone + translate demo (MOSS-TTS)
|
||||
|
||||
`clone_and_translate_demo.py` translates a short English sentence spoken by the narrator in
|
||||
our sample clip to Spanish (via `Helsinki-NLP/opus-mt-en-es`), then uses
|
||||
[MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS) to synthesize the Spanish text in the
|
||||
narrator's cloned voice. The reference audio (`reference/narrator_ref_en.wav`) is trimmed
|
||||
from demucs's isolated vocals stem for the exact same sentence, so the English original and
|
||||
the Spanish clone can be compared directly — this is the first working slice of the full
|
||||
audio-translation pipeline sketched in the top-level README.
|
||||
|
||||
### Setup
|
||||
|
||||
MOSS-TTS ships as a full repo you install editable (not a plain `pip install`), and it isn't
|
||||
part of this git repo — it's vendored outside it since it's a large third-party project:
|
||||
|
||||
```bash
|
||||
git clone --depth 1 https://github.com/OpenMOSS/MOSS-TTS.git /path/to/vendor/MOSS-TTS
|
||||
python3 -m venv .venv && . .venv/bin/activate
|
||||
pip install --extra-index-url https://download.pytorch.org/whl/cu128 \
|
||||
-e "/path/to/vendor/MOSS-TTS[torch-runtime]"
|
||||
pip install sentencepiece # for the MarianMT translation step
|
||||
```
|
||||
|
||||
### Running
|
||||
|
||||
```bash
|
||||
SITE=$(python3 -c "import site; print(site.getsitepackages()[0])")
|
||||
export LD_LIBRARY_PATH=$(find "$SITE/nvidia" -maxdepth 2 -type d -name lib | paste -sd: -)
|
||||
python3 clone_and_translate_demo.py
|
||||
```
|
||||
|
||||
Output goes to `output/narrator_cloned_es.wav` (48kHz stereo, ~11s).
|
||||
|
||||
### Gotchas found running this on an RTX 3090
|
||||
|
||||
- **The flagship 8B `MOSS-TTS-v1.5` checkpoint OOMs on a 24GB 3090** — loading the model plus
|
||||
its audio tokenizer already uses ~22.5GB, leaving no room for generation. Use the 4B
|
||||
`MOSS-TTS-Local-Transformer-v1.5` checkpoint instead (also natively 48kHz stereo); it's
|
||||
the option the project itself recommends for smaller GPUs.
|
||||
- **`torchaudio.load()`/`.save()` fail with `Could not load libtorchcodec`** on this setup.
|
||||
The real cause (visible via `ctypes.CDLL` directly, not the wrapped exception) is a missing
|
||||
`libnppicc.so.12` — torchcodec links against NVIDIA's NPP library for its video-decode path,
|
||||
but `moss-tts`'s dependencies don't pull it in since MOSS-TTS only needs audio. Fix:
|
||||
`pip install nvidia-npp-cu12`, then add all `nvidia/*/lib` wheel directories to
|
||||
`LD_LIBRARY_PATH` before running (see the `SITE=...` line above) — the pip-installed CUDA
|
||||
libraries aren't on the default linker search path. This also affects code paths *inside*
|
||||
MOSS-TTS's own processor (it uses `torchaudio.load()` to read reference audio), so the fix
|
||||
has to be in place before the script imports it, not just around your own save calls.
|
||||
- Saving the generated audio with plain `soundfile.write()` instead of `torchaudio.save()`
|
||||
avoids depending on torchcodec at all for the output side, but doesn't help with the
|
||||
reference-audio loading path above.
|
||||
@@ -0,0 +1,120 @@
|
||||
"""
|
||||
Voice-clone + translate demo.
|
||||
|
||||
Takes a short English sentence spoken by the narrator in our sample clip
|
||||
(the same sentence used as the reference audio, for an apples-to-apples
|
||||
comparison), machine-translates it to Spanish, then uses MOSS-TTS to
|
||||
synthesize the Spanish text in the narrator's cloned voice.
|
||||
|
||||
Reference audio (tts/reference/narrator_ref_en.wav) was trimmed from
|
||||
demucs's isolated vocals stem for this same clip.
|
||||
"""
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
import soundfile as sf
|
||||
import torch
|
||||
from transformers import AutoModel, AutoProcessor, MarianMTModel, MarianTokenizer
|
||||
|
||||
HERE = Path(__file__).parent
|
||||
REFERENCE_AUDIO = HERE / "reference" / "narrator_ref_en.wav"
|
||||
OUTPUT_DIR = HERE / "output"
|
||||
|
||||
TEXT_EN = (
|
||||
"We shall show you in this film typical examples of children who have "
|
||||
"been well brought up, the kind who grow up to be strong, healthy, and "
|
||||
"happy like these young folks."
|
||||
)
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
dtype = torch.bfloat16 if device == "cuda" else torch.float32
|
||||
|
||||
|
||||
def translate_to_spanish(text: str) -> str:
|
||||
model_name = "Helsinki-NLP/opus-mt-en-es"
|
||||
tokenizer = MarianTokenizer.from_pretrained(model_name)
|
||||
model = MarianMTModel.from_pretrained(model_name).to(device)
|
||||
batch = tokenizer([text], return_tensors="pt", padding=True).to(device)
|
||||
generated = model.generate(**batch, max_new_tokens=256)
|
||||
return tokenizer.decode(generated[0], skip_special_tokens=True)
|
||||
|
||||
|
||||
def resolve_attn_implementation() -> str:
|
||||
if (
|
||||
device == "cuda"
|
||||
and importlib.util.find_spec("flash_attn") is not None
|
||||
and dtype in {torch.float16, torch.bfloat16}
|
||||
):
|
||||
major, _ = torch.cuda.get_device_capability()
|
||||
if major >= 8:
|
||||
return "flash_attention_2"
|
||||
return "sdpa" if device == "cuda" else "eager"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
text_es = translate_to_spanish(TEXT_EN)
|
||||
print(f"[EN] {TEXT_EN}")
|
||||
print(f"[ES] {text_es}")
|
||||
|
||||
# Disable the broken cuDNN SDPA backend (per MOSS-TTS README).
|
||||
torch.backends.cuda.enable_cudnn_sdp(False)
|
||||
torch.backends.cuda.enable_flash_sdp(True)
|
||||
torch.backends.cuda.enable_mem_efficient_sdp(True)
|
||||
torch.backends.cuda.enable_math_sdp(True)
|
||||
|
||||
# The flagship MOSS-TTS-v1.5 (8B) doesn't fit alongside its audio
|
||||
# tokenizer + generation overhead on a 24GB 3090 (OOMs at ~23.3GB used).
|
||||
# The 4B Local-Transformer variant is the recommended smaller-GPU option
|
||||
# and also natively outputs 48kHz stereo.
|
||||
pretrained = "OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5"
|
||||
attn_implementation = resolve_attn_implementation()
|
||||
print(f"[INFO] attn_implementation={attn_implementation}")
|
||||
|
||||
processor = AutoProcessor.from_pretrained(pretrained, trust_remote_code=True)
|
||||
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
|
||||
|
||||
model = AutoModel.from_pretrained(
|
||||
pretrained,
|
||||
trust_remote_code=True,
|
||||
attn_implementation=attn_implementation,
|
||||
dtype=dtype,
|
||||
).to(device)
|
||||
model.eval()
|
||||
|
||||
conversation = [
|
||||
processor.build_user_message(
|
||||
text=text_es,
|
||||
language="Spanish",
|
||||
reference=[str(REFERENCE_AUDIO)],
|
||||
)
|
||||
]
|
||||
|
||||
OUTPUT_DIR.mkdir(exist_ok=True, parents=True)
|
||||
with torch.no_grad():
|
||||
batch = processor([conversation], mode="generation")
|
||||
outputs = model.generate(
|
||||
input_ids=batch["input_ids"].to(device),
|
||||
attention_mask=batch["attention_mask"].to(device),
|
||||
max_new_tokens=4096,
|
||||
do_sample=True,
|
||||
audio_temperature=1.7,
|
||||
audio_top_p=0.8,
|
||||
audio_top_k=25,
|
||||
audio_repetition_penalty=1.0,
|
||||
)
|
||||
message = processor.decode(outputs)[0]
|
||||
audio = message.audio_codes_list[0]
|
||||
# torchaudio.save() depends on torchcodec, whose bundled shared libs
|
||||
# failed to load in this environment; soundfile (already a MOSS-TTS
|
||||
# dependency) sidesteps that entirely.
|
||||
data = audio.detach().cpu().to(torch.float32).numpy()
|
||||
if data.ndim == 2:
|
||||
data = data.T # soundfile wants (frames, channels), not (channels, frames)
|
||||
out_path = OUTPUT_DIR / "narrator_cloned_es.wav"
|
||||
sf.write(str(out_path), data, processor.model_config.sampling_rate)
|
||||
print(f"[INFO] Saved {out_path}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
-109
@@ -1,109 +0,0 @@
|
||||
"""
|
||||
Generic MOSS-TTS zero-shot cloning CLI: clone the voice in --reference and
|
||||
say --text in it. Used directly, and as the final stage of webui/app.py.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
|
||||
import soundfile as sf
|
||||
import torch
|
||||
from transformers import AutoModel, AutoProcessor
|
||||
|
||||
PRETRAINED = "OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5"
|
||||
|
||||
|
||||
def resolve_attn_implementation(device: str, dtype: torch.dtype) -> str:
|
||||
if (
|
||||
device == "cuda"
|
||||
and importlib.util.find_spec("flash_attn") is not None
|
||||
and dtype in {torch.float16, torch.bfloat16}
|
||||
):
|
||||
major, _ = torch.cuda.get_device_capability()
|
||||
if major >= 8:
|
||||
return "flash_attention_2"
|
||||
return "sdpa" if device == "cuda" else "eager"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="MOSS-TTS zero-shot voice cloning")
|
||||
parser.add_argument("--text", required=True, help="Text to synthesize")
|
||||
parser.add_argument("--reference", required=True, help="Reference audio path")
|
||||
parser.add_argument("--output", required=True, help="Output wav path")
|
||||
parser.add_argument("--language", default="English", help="Language tag")
|
||||
parser.add_argument(
|
||||
"--instruction", default=None,
|
||||
help='Style/delivery instruction (e.g. "Change the emotion to excited."). '
|
||||
"Undocumented MOSS-TTS field for this checkpoint - confirmed to work for "
|
||||
"emotion in testing, passed through as-is.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--seed", type=int, default=None,
|
||||
help="Random seed for reproducibility. Unset = nondeterministic.",
|
||||
)
|
||||
parser.add_argument("--audio_temperature", type=float, default=1.7)
|
||||
parser.add_argument("--audio_top_p", type=float, default=0.8)
|
||||
parser.add_argument("--audio_top_k", type=int, default=25)
|
||||
parser.add_argument("--audio_repetition_penalty", type=float, default=1.0)
|
||||
parser.add_argument("--max_new_tokens", type=int, default=4096)
|
||||
args = parser.parse_args()
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
dtype = torch.bfloat16 if device == "cuda" else torch.float32
|
||||
|
||||
if args.seed is not None:
|
||||
torch.manual_seed(args.seed)
|
||||
if device == "cuda":
|
||||
torch.cuda.manual_seed_all(args.seed)
|
||||
|
||||
torch.backends.cuda.enable_cudnn_sdp(False)
|
||||
torch.backends.cuda.enable_flash_sdp(True)
|
||||
torch.backends.cuda.enable_mem_efficient_sdp(True)
|
||||
torch.backends.cuda.enable_math_sdp(True)
|
||||
|
||||
attn_implementation = resolve_attn_implementation(device, dtype)
|
||||
print(f"[INFO] attn_implementation={attn_implementation}")
|
||||
|
||||
processor = AutoProcessor.from_pretrained(PRETRAINED, trust_remote_code=True)
|
||||
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
|
||||
|
||||
model = AutoModel.from_pretrained(
|
||||
PRETRAINED,
|
||||
trust_remote_code=True,
|
||||
attn_implementation=attn_implementation,
|
||||
dtype=dtype,
|
||||
).to(device)
|
||||
model.eval()
|
||||
|
||||
conversation = [
|
||||
processor.build_user_message(
|
||||
text=args.text,
|
||||
language=args.language,
|
||||
reference=[args.reference],
|
||||
instruction=args.instruction,
|
||||
)
|
||||
]
|
||||
|
||||
with torch.no_grad():
|
||||
batch = processor([conversation], mode="generation")
|
||||
outputs = model.generate(
|
||||
input_ids=batch["input_ids"].to(device),
|
||||
attention_mask=batch["attention_mask"].to(device),
|
||||
max_new_tokens=args.max_new_tokens,
|
||||
do_sample=True,
|
||||
audio_temperature=args.audio_temperature,
|
||||
audio_top_p=args.audio_top_p,
|
||||
audio_top_k=args.audio_top_k,
|
||||
audio_repetition_penalty=args.audio_repetition_penalty,
|
||||
)
|
||||
message = processor.decode(outputs)[0]
|
||||
audio = message.audio_codes_list[0]
|
||||
data = audio.detach().cpu().to(torch.float32).numpy()
|
||||
if data.ndim == 2:
|
||||
data = data.T
|
||||
sf.write(args.output, data, processor.model_config.sampling_rate)
|
||||
print(f"[INFO] Saved {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,99 +0,0 @@
|
||||
"""
|
||||
MOSS-SoundEffect CLI: generate an ambient/sound-effect clip from a text
|
||||
description. Sibling of generate.py (MOSS-TTS voice cloning) - same
|
||||
backend setup and model-loading pattern, different checkpoint and a
|
||||
text-only build_user_message call (no reference audio, no cloning).
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import importlib.util
|
||||
import os
|
||||
|
||||
# This checkpoint's weights alone use ~22.3GB of this box's ~23.56GB usable
|
||||
# VRAM, leaving generate()'s own small allocations (a vocab-sized mask
|
||||
# tensor, etc.) no room without this - confirmed via OOM otherwise. Must be
|
||||
# set before torch's CUDA allocator initializes, so it's set here rather
|
||||
# than relying on the caller's shell environment.
|
||||
os.environ.setdefault("PYTORCH_ALLOC_CONF", "expandable_segments:True")
|
||||
os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
|
||||
|
||||
import soundfile as sf
|
||||
import torch
|
||||
from transformers import AutoModel, AutoProcessor
|
||||
|
||||
PRETRAINED = "OpenMOSS-Team/MOSS-SoundEffect"
|
||||
|
||||
|
||||
def resolve_attn_implementation(device: str, dtype: torch.dtype) -> str:
|
||||
if (
|
||||
device == "cuda"
|
||||
and importlib.util.find_spec("flash_attn") is not None
|
||||
and dtype in {torch.float16, torch.bfloat16}
|
||||
):
|
||||
major, _ = torch.cuda.get_device_capability()
|
||||
if major >= 8:
|
||||
return "flash_attention_2"
|
||||
return "sdpa" if device == "cuda" else "eager"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description="MOSS-SoundEffect text-to-sound-effect")
|
||||
parser.add_argument("--text", required=True, help="Description of the sound effect/ambience")
|
||||
parser.add_argument("--output", required=True, help="Output wav path")
|
||||
parser.add_argument(
|
||||
"--seed", type=int, default=None,
|
||||
help="Random seed for reproducibility. Unset = nondeterministic.",
|
||||
)
|
||||
parser.add_argument("--max_new_tokens", type=int, default=4096)
|
||||
args = parser.parse_args()
|
||||
|
||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
||||
dtype = torch.bfloat16 if device == "cuda" else torch.float32
|
||||
|
||||
if args.seed is not None:
|
||||
torch.manual_seed(args.seed)
|
||||
if device == "cuda":
|
||||
torch.cuda.manual_seed_all(args.seed)
|
||||
|
||||
torch.backends.cuda.enable_cudnn_sdp(False)
|
||||
torch.backends.cuda.enable_flash_sdp(True)
|
||||
torch.backends.cuda.enable_mem_efficient_sdp(True)
|
||||
torch.backends.cuda.enable_math_sdp(True)
|
||||
|
||||
attn_implementation = resolve_attn_implementation(device, dtype)
|
||||
print(f"[INFO] attn_implementation={attn_implementation}")
|
||||
|
||||
processor = AutoProcessor.from_pretrained(PRETRAINED, trust_remote_code=True)
|
||||
processor.audio_tokenizer = processor.audio_tokenizer.to(device)
|
||||
|
||||
model = AutoModel.from_pretrained(
|
||||
PRETRAINED,
|
||||
trust_remote_code=True,
|
||||
attn_implementation=attn_implementation,
|
||||
dtype=dtype,
|
||||
).to(device)
|
||||
model.eval()
|
||||
|
||||
conversation = [processor.build_user_message(ambient_sound=args.text)]
|
||||
|
||||
with torch.no_grad():
|
||||
batch = processor([conversation], mode="generation")
|
||||
# This checkpoint's generate() doesn't accept the TTS model's sampling
|
||||
# kwargs (do_sample/audio_temperature/etc raise TypeError) - matches
|
||||
# the official README example, which only passes these three.
|
||||
outputs = model.generate(
|
||||
input_ids=batch["input_ids"].to(device),
|
||||
attention_mask=batch["attention_mask"].to(device),
|
||||
max_new_tokens=args.max_new_tokens,
|
||||
)
|
||||
message = processor.decode(outputs)[0]
|
||||
audio = message.audio_codes_list[0]
|
||||
data = audio.detach().cpu().to(torch.float32).numpy()
|
||||
if data.ndim == 2:
|
||||
data = data.T
|
||||
sf.write(args.output, data, processor.model_config.sampling_rate)
|
||||
print(f"[INFO] Saved {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Binary file not shown.
Binary file not shown.
@@ -1,52 +0,0 @@
|
||||
# VibeVoice (community fork) 4-person podcast spike
|
||||
|
||||
Standalone experiment, not integrated into the webui or the production
|
||||
pipeline. Tests whether VibeVoice (https://github.com/vibevoice-community/VibeVoice)
|
||||
can generate a 4-speaker podcast scene in one pass, cloning our four
|
||||
established character voices (Kevin, Marge, Terry, Priya) from their
|
||||
existing approved `reference_audio` in the-service-station repo.
|
||||
|
||||
- `scripts/trailer_promo_4speaker.txt` - the test script: the real
|
||||
"sausage rolls" trailer scene from `the-service-station/scripts/trailer-promo.md`,
|
||||
converted from our `[Character|tags]` format to VibeVoices required
|
||||
`Speaker N: ...` plain-text format (tags dropped - VibeVoice has no
|
||||
per-line emotion/timing control).
|
||||
- `trailer_promo_4speaker_vibevoice-1.5b.mp3` - the generated result
|
||||
(VibeVoice-1.5B, seed 42). 47s of audio generated in 14.5s (RTF 0.31x).
|
||||
|
||||
Setup lives outside this repo (not committed): `.venv/` here is a fresh
|
||||
isolated venv (`pip install -e /root/projects/VibeVoice`, pinned
|
||||
`transformers==4.51.3`), pointed at the already-downloaded
|
||||
`VibeVoice-1.5B` weights under `/root/projects/ComfyUI/models/tts/VibeVoice/`.
|
||||
|
||||
## Findings (2026-09-30)
|
||||
|
||||
Whisper-transcript check against the script: mostly faithful, not
|
||||
word-perfect. Two small unscripted insertions ("I am here!" before Priyas
|
||||
line, "For me," before Marges reply) and one substitution ("He will." ->
|
||||
"You will."). No repetition/collapse like the AuK chain-corruption bug -
|
||||
just minor ad-libbing, not catastrophic.
|
||||
|
||||
No audible AI disclaimer or obvious background music in this run - both
|
||||
are documented risks for this model family (see
|
||||
`the-service-station/research/vibevoice-2026-08-21.md` and
|
||||
`vibevoice-community-fork-2026-08-21.md`), so treat this one clean run as
|
||||
encouraging, not conclusive.
|
||||
|
||||
Steves call after listening: pursue 7B (weights already downloaded, same
|
||||
venv/script, swap `--model_path`) or not worth it given the limitations
|
||||
the prior research already flagged.
|
||||
|
||||
## 7B result (2026-09-30, same script/seed/voices)
|
||||
|
||||
`trailer_promo_4speaker_vibevoice-7b.mp3` - VibeVoice-7B, same seed (42),
|
||||
same script, same reference voices. 34.8s of audio generated in 22.2s
|
||||
(RTF 0.64x - still faster than real-time, just less headroom than 1.5B).
|
||||
|
||||
Whisper-transcript check: cleaner than 1.5B - neither of 1.5Bs unscripted
|
||||
insertions ("I am here!", "For me,") appeared, and "He will." came through
|
||||
exactly right this time (1.5B had drifted it to "You will."). One real
|
||||
deviation: Marges defensive line "I get one fact wrong per episode."
|
||||
came out as a question, "Can I get one fact wrong per episode?" - changes
|
||||
the delivery/meaning, not just wording. Still no audible disclaimer and
|
||||
nothing that reads as injected background music.
|
||||
@@ -1,9 +0,0 @@
|
||||
Speaker 1: The Service Station. Three facts, one car park, and absolutely no reason to be in a hurry.
|
||||
Speaker 2: Except for me, obviously. I'm always in a hurry.
|
||||
Speaker 3: I had a Sinclair Spectrum in 1984, and I will find a way to bring it up.
|
||||
Speaker 1: He will.
|
||||
Speaker 4: I still don't understand why the sausage rolls are relevant, but here we are.
|
||||
Speaker 2: They're always relevant, Priya.
|
||||
Speaker 1: Every week: one underdog, one argument, and one fact Marge gets wrong.
|
||||
Speaker 2: I get one fact wrong per episode. Tops.
|
||||
Speaker 3: It's usually about football.
|
||||
Binary file not shown.
Binary file not shown.
@@ -1,56 +0,0 @@
|
||||
# webui
|
||||
|
||||
A pipeline studio: strip a voice with **demucs**, edit it with **AuK** (emotion, pitch, speed,
|
||||
free-form instructions — with preview and undo/reset), then generate new speech in that voice
|
||||
with **MOSS-TTS**.
|
||||
|
||||
## Why this exists
|
||||
|
||||
Manually chaining demucs → AuK → MOSS-TTS across three separate venvs (via SSH one-off scripts)
|
||||
worked but made it slow to experiment — especially with AuK's pitch/emotion controls, where you
|
||||
really want to preview a result and back out if it's wrong (see the pitch-overshoot findings in
|
||||
`auk/README.md`) rather than committing to a specific edit blind.
|
||||
|
||||
## Architecture
|
||||
|
||||
A single lightweight Gradio app (`app.py`) that **shells out** to each tool's own isolated venv
|
||||
via subprocess, rather than merging their dependencies into one environment — MOSS-TTS pins
|
||||
`transformers==5.0.0`, AuK needs `4.57.6`; they can't coexist in one venv. This app's own venv
|
||||
only needs `gradio` itself.
|
||||
|
||||
Each stage's output becomes the next stage's input by filesystem path; a `history` list backs
|
||||
the undo/reset buttons in the voice editor.
|
||||
|
||||
## Setup
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv && . .venv/bin/activate
|
||||
pip install gradio
|
||||
```
|
||||
|
||||
Assumes `demucs/`, `auk/`, and `tts/` (siblings in this repo) already have their own working
|
||||
venvs and, for `auk/`, downloaded model weights — see their READMEs.
|
||||
|
||||
## Running
|
||||
|
||||
```bash
|
||||
python3 app.py
|
||||
```
|
||||
|
||||
Serves on `0.0.0.0:7860`. The app automatically calls ComfyUI's `/free` endpoint (if its queue
|
||||
is empty) before each heavy step, to reduce GPU contention on a shared box — see the DNS/routing
|
||||
and GPU-contention notes earlier in this project's history for why that matters here.
|
||||
|
||||
## Known limitations (inherited from the underlying tools)
|
||||
|
||||
- **AuK's real usable clip length is ~25-30s**, not several minutes — feeding it a long recording
|
||||
will OOM even with `--cpu_offload` (see `auk/README.md`). This app doesn't chunk long inputs
|
||||
automatically; keep reference clips short.
|
||||
- **AuK's pitch-editing instruction-following is unreliable** — even "1 semitone" has measured
|
||||
out at ~4+ semitones of actual shift in testing. The UI surfaces a warning; preview before
|
||||
trusting a pitch edit.
|
||||
- Each stage is a blocking subprocess call (model load + inference each time) — the UI will
|
||||
appear to hang for anywhere from ~10s to a couple of minutes per click, with no progress bar
|
||||
yet.
|
||||
- No queueing/locking across browser tabs or concurrent users — this is a single-GPU box, so two
|
||||
simultaneous generations will contend for VRAM.
|
||||
-1106
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user