steve-adminandClaude Sonnet 5
7c4f226e17
Translates a short narrator sentence to Spanish (MarianMT) and synthesizes it in the narrator's cloned voice (MOSS-TTS Local-Transformer-v1.5), using a reference clip trimmed from demucs's isolated vocals stem. First working slice of the full audio-translation pipeline. Documents two environment gotchas hit on the RTX 3090: the flagship 8B checkpoint OOMs on 24GB, and torchaudio/torchcodec fails to load without manually pulling in nvidia-npp-cu12 and fixing LD_LIBRARY_PATH. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
33 lines
1.4 KiB
Markdown
33 lines
1.4 KiB
Markdown
# audio-demo
|
|
|
|
Audio processing demos: WhisperX, Demucs, and TTS technology proofs-of-concept.
|
|
|
|
## Structure
|
|
|
|
- `whisperx/` — speech-to-text with word-level alignment and diarization
|
|
- `demucs/` — source separation (vocals/instrumentals stem splitting)
|
|
- `tts/` — text-to-speech service demos
|
|
|
|
Each subdirectory has its own README with setup and run instructions.
|
|
|
|
## Requirements
|
|
|
|
Some demos are GPU-accelerated and are best run on a CUDA-capable machine (e.g. an RTX 3090).
|
|
|
|
## Planned: full audio-translation demo
|
|
|
|
Chain the three demos together on `media-files/` into an end-to-end pipeline:
|
|
|
|
1. `demucs` — separate the source clip into vocals / instrumental stems
|
|
2. `whisperx` — transcribe the isolated vocal stem, then machine-translate the transcript
|
|
3. `tts` — clone the original speaker's voice (from the isolated vocal stem) and synthesize
|
|
the translated text in that voice, then remix with the original instrumental stem
|
|
|
|
All three stages now have a working, if narrow, proof of concept:
|
|
|
|
- Stem separation (`demucs/README.md`) and transcription (`whisperx/README.md`) run end to end
|
|
on the full sample clip.
|
|
- Translation + voice-cloning TTS (`tts/README.md`) works for a single short sentence
|
|
(English → Spanish, cloned onto the isolated narrator voice via MOSS-TTS). Not yet wired up
|
|
to run over the full transcript or to remix with the instrumental stem.
|