audio-demo/tts/README.md

# tts

Text-to-speech service demos and proofs-of-concept.

## Voice-clone + translate demo (MOSS-TTS)

`clone_and_translate_demo.py` translates a short English sentence spoken by the narrator in
our sample clip to Spanish (via `Helsinki-NLP/opus-mt-en-es`), then uses
[MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS) to synthesize the Spanish text in the
narrator's cloned voice. The reference audio (`reference/narrator_ref_en.wav`) is trimmed
from demucs's isolated vocals stem for the exact same sentence, so the English original and
the Spanish clone can be compared directly — this is the first working slice of the full
audio-translation pipeline sketched in the top-level README.

### Setup

MOSS-TTS ships as a full repo you install editable (not a plain `pip install`), and it isn't
part of this git repo — it's vendored outside it since it's a large third-party project:

```bash
git clone --depth 1 https://github.com/OpenMOSS/MOSS-TTS.git /path/to/vendor/MOSS-TTS
python3 -m venv .venv && . .venv/bin/activate
pip install --extra-index-url https://download.pytorch.org/whl/cu128 \
  -e "/path/to/vendor/MOSS-TTS[torch-runtime]"
pip install sentencepiece  # for the MarianMT translation step
```

### Running

```bash
SITE=$(python3 -c "import site; print(site.getsitepackages()[0])")
export LD_LIBRARY_PATH=$(find "$SITE/nvidia" -maxdepth 2 -type d -name lib | paste -sd: -)
python3 clone_and_translate_demo.py
```

Output goes to `output/narrator_cloned_es.wav` (48kHz stereo, ~11s).

### Gotchas found running this on an RTX 3090

- **The flagship 8B `MOSS-TTS-v1.5` checkpoint OOMs on a 24GB 3090** — loading the model plus
  its audio tokenizer already uses ~22.5GB, leaving no room for generation. Use the 4B
  `MOSS-TTS-Local-Transformer-v1.5` checkpoint instead (also natively 48kHz stereo); it's
  the option the project itself recommends for smaller GPUs.
- **`torchaudio.load()`/`.save()` fail with `Could not load libtorchcodec`** on this setup.
  The real cause (visible via `ctypes.CDLL` directly, not the wrapped exception) is a missing
  `libnppicc.so.12` — torchcodec links against NVIDIA's NPP library for its video-decode path,
  but `moss-tts`'s dependencies don't pull it in since MOSS-TTS only needs audio. Fix:
  `pip install nvidia-npp-cu12`, then add all `nvidia/*/lib` wheel directories to
  `LD_LIBRARY_PATH` before running (see the `SITE=...` line above) — the pip-installed CUDA
  libraries aren't on the default linker search path. This also affects code paths *inside*
  MOSS-TTS's own processor (it uses `torchaudio.load()` to read reference audio), so the fix
  has to be in place before the script imports it, not just around your own save calls.
- Saving the generated audio with plain `soundfile.write()` instead of `torchaudio.save()`
  avoids depending on torchcodec at all for the output side, but doesn't help with the
  reference-audio loading path above.