steve-adminandClaude Sonnet 5
7c4f226e17
Translates a short narrator sentence to Spanish (MarianMT) and synthesizes it in the narrator's cloned voice (MOSS-TTS Local-Transformer-v1.5), using a reference clip trimmed from demucs's isolated vocals stem. First working slice of the full audio-translation pipeline. Documents two environment gotchas hit on the RTX 3090: the flagship 8B checkpoint OOMs on 24GB, and torchaudio/torchcodec fails to load without manually pulling in nvidia-npp-cu12 and fixing LD_LIBRARY_PATH. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
1.4 KiB
1.4 KiB
audio-demo
Audio processing demos: WhisperX, Demucs, and TTS technology proofs-of-concept.
Structure
whisperx/— speech-to-text with word-level alignment and diarizationdemucs/— source separation (vocals/instrumentals stem splitting)tts/— text-to-speech service demos
Each subdirectory has its own README with setup and run instructions.
Requirements
Some demos are GPU-accelerated and are best run on a CUDA-capable machine (e.g. an RTX 3090).
Planned: full audio-translation demo
Chain the three demos together on media-files/ into an end-to-end pipeline:
demucs— separate the source clip into vocals / instrumental stemswhisperx— transcribe the isolated vocal stem, then machine-translate the transcripttts— clone the original speaker's voice (from the isolated vocal stem) and synthesize the translated text in that voice, then remix with the original instrumental stem
All three stages now have a working, if narrow, proof of concept:
- Stem separation (
demucs/README.md) and transcription (whisperx/README.md) run end to end on the full sample clip. - Translation + voice-cloning TTS (
tts/README.md) works for a single short sentence (English → Spanish, cloned onto the isolated narrator voice via MOSS-TTS). Not yet wired up to run over the full transcript or to remix with the instrumental stem.