audio-demo/tts/README.md

tts

Text-to-speech service demos and proofs-of-concept.

Voice-clone + translate demo (MOSS-TTS)

clone_and_translate_demo.py translates a short English sentence spoken by the narrator in our sample clip to Spanish (via Helsinki-NLP/opus-mt-en-es), then uses MOSS-TTS to synthesize the Spanish text in the narrator's cloned voice. The reference audio (reference/narrator_ref_en.wav) is trimmed from demucs's isolated vocals stem for the exact same sentence, so the English original and the Spanish clone can be compared directly — this is the first working slice of the full audio-translation pipeline sketched in the top-level README.

Setup

MOSS-TTS ships as a full repo you install editable (not a plain pip install), and it isn't part of this git repo — it's vendored outside it since it's a large third-party project:

git clone --depth 1 https://github.com/OpenMOSS/MOSS-TTS.git /path/to/vendor/MOSS-TTS
python3 -m venv .venv && . .venv/bin/activate
pip install --extra-index-url https://download.pytorch.org/whl/cu128 \
  -e "/path/to/vendor/MOSS-TTS[torch-runtime]"
pip install sentencepiece  # for the MarianMT translation step

Running

SITE=$(python3 -c "import site; print(site.getsitepackages()[0])")
export LD_LIBRARY_PATH=$(find "$SITE/nvidia" -maxdepth 2 -type d -name lib | paste -sd: -)
python3 clone_and_translate_demo.py

Output goes to output/narrator_cloned_es.wav (48kHz stereo, ~11s).

Gotchas found running this on an RTX 3090

• The flagship 8B MOSS-TTS-v1.5 checkpoint OOMs on a 24GB 3090 — loading the model plus its audio tokenizer already uses ~22.5GB, leaving no room for generation. Use the 4B MOSS-TTS-Local-Transformer-v1.5 checkpoint instead (also natively 48kHz stereo); it's the option the project itself recommends for smaller GPUs.
• torchaudio.load()/.save() fail with Could not load libtorchcodec on this setup. The real cause (visible via ctypes.CDLL directly, not the wrapped exception) is a missing libnppicc.so.12 — torchcodec links against NVIDIA's NPP library for its video-decode path, but moss-tts's dependencies don't pull it in since MOSS-TTS only needs audio. Fix: pip install nvidia-npp-cu12, then add all nvidia/*/lib wheel directories to LD_LIBRARY_PATH before running (see the SITE=... line above) — the pip-installed CUDA libraries aren't on the default linker search path. This also affects code paths inside MOSS-TTS's own processor (it uses torchaudio.load() to read reference audio), so the fix has to be in place before the script imports it, not just around your own save calls.
• Saving the generated audio with plain soundfile.write() instead of torchaudio.save() avoids depending on torchcodec at all for the output side, but doesn't help with the reference-audio loading path above.