steve-adminandClaude Sonnet 5
7c4f226e17
Translates a short narrator sentence to Spanish (MarianMT) and synthesizes it in the narrator's cloned voice (MOSS-TTS Local-Transformer-v1.5), using a reference clip trimmed from demucs's isolated vocals stem. First working slice of the full audio-translation pipeline. Documents two environment gotchas hit on the RTX 3090: the flagship 8B checkpoint OOMs on 24GB, and torchaudio/torchcodec fails to load without manually pulling in nvidia-npp-cu12 and fixing LD_LIBRARY_PATH. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
56 lines
2.9 KiB
Markdown
56 lines
2.9 KiB
Markdown
# tts
|
|
|
|
Text-to-speech service demos and proofs-of-concept.
|
|
|
|
## Voice-clone + translate demo (MOSS-TTS)
|
|
|
|
`clone_and_translate_demo.py` translates a short English sentence spoken by the narrator in
|
|
our sample clip to Spanish (via `Helsinki-NLP/opus-mt-en-es`), then uses
|
|
[MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS) to synthesize the Spanish text in the
|
|
narrator's cloned voice. The reference audio (`reference/narrator_ref_en.wav`) is trimmed
|
|
from demucs's isolated vocals stem for the exact same sentence, so the English original and
|
|
the Spanish clone can be compared directly — this is the first working slice of the full
|
|
audio-translation pipeline sketched in the top-level README.
|
|
|
|
### Setup
|
|
|
|
MOSS-TTS ships as a full repo you install editable (not a plain `pip install`), and it isn't
|
|
part of this git repo — it's vendored outside it since it's a large third-party project:
|
|
|
|
```bash
|
|
git clone --depth 1 https://github.com/OpenMOSS/MOSS-TTS.git /path/to/vendor/MOSS-TTS
|
|
python3 -m venv .venv && . .venv/bin/activate
|
|
pip install --extra-index-url https://download.pytorch.org/whl/cu128 \
|
|
-e "/path/to/vendor/MOSS-TTS[torch-runtime]"
|
|
pip install sentencepiece # for the MarianMT translation step
|
|
```
|
|
|
|
### Running
|
|
|
|
```bash
|
|
SITE=$(python3 -c "import site; print(site.getsitepackages()[0])")
|
|
export LD_LIBRARY_PATH=$(find "$SITE/nvidia" -maxdepth 2 -type d -name lib | paste -sd: -)
|
|
python3 clone_and_translate_demo.py
|
|
```
|
|
|
|
Output goes to `output/narrator_cloned_es.wav` (48kHz stereo, ~11s).
|
|
|
|
### Gotchas found running this on an RTX 3090
|
|
|
|
- **The flagship 8B `MOSS-TTS-v1.5` checkpoint OOMs on a 24GB 3090** — loading the model plus
|
|
its audio tokenizer already uses ~22.5GB, leaving no room for generation. Use the 4B
|
|
`MOSS-TTS-Local-Transformer-v1.5` checkpoint instead (also natively 48kHz stereo); it's
|
|
the option the project itself recommends for smaller GPUs.
|
|
- **`torchaudio.load()`/`.save()` fail with `Could not load libtorchcodec`** on this setup.
|
|
The real cause (visible via `ctypes.CDLL` directly, not the wrapped exception) is a missing
|
|
`libnppicc.so.12` — torchcodec links against NVIDIA's NPP library for its video-decode path,
|
|
but `moss-tts`'s dependencies don't pull it in since MOSS-TTS only needs audio. Fix:
|
|
`pip install nvidia-npp-cu12`, then add all `nvidia/*/lib` wheel directories to
|
|
`LD_LIBRARY_PATH` before running (see the `SITE=...` line above) — the pip-installed CUDA
|
|
libraries aren't on the default linker search path. This also affects code paths *inside*
|
|
MOSS-TTS's own processor (it uses `torchaudio.load()` to read reference audio), so the fix
|
|
has to be in place before the script imports it, not just around your own save calls.
|
|
- Saving the generated audio with plain `soundfile.write()` instead of `torchaudio.save()`
|
|
avoids depending on torchcodec at all for the output side, but doesn't help with the
|
|
reference-audio loading path above.
|