Add demucs stem-separation demo; sketch audio-translation pipeline
Two-stem separation (vocals/instrumental) run on RTX 3090 against the shared sample clip. Isolated vocals.wav is meant as the clean voice reference for a future voice-cloning TTS step. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
This commit is contained in:
1 parent
ff71bd5777
commit
79691b81c6
2 files changed
+35
No files matched your search
@@ -13,3 +13,15 @@ Each subdirectory has its own README with setup and run instructions.
|
|||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
Some demos are GPU-accelerated and are best run on a CUDA-capable machine (e.g. an RTX 3090).
|
Some demos are GPU-accelerated and are best run on a CUDA-capable machine (e.g. an RTX 3090).
|
||||||
|
|
||||||
|
## Planned: full audio-translation demo
|
||||||
|
|
||||||
|
Chain the three demos together on `media-files/` into an end-to-end pipeline:
|
||||||
|
|
||||||
|
1. `demucs` — separate the source clip into vocals / instrumental stems
|
||||||
|
2. `whisperx` — transcribe the isolated vocal stem, then machine-translate the transcript
|
||||||
|
3. `tts` — clone the original speaker's voice (from the isolated vocal stem) and synthesize
|
||||||
|
the translated text in that voice, then remix with the original instrumental stem
|
||||||
|
|
||||||
|
Stem separation and transcription are already working end to end (see `demucs/README.md` and
|
||||||
|
`whisperx/README.md`); translation and voice-cloning TTS are not yet implemented.
|
||||||
@@ -1,3 +1,26 @@
|
|||||||
# demucs
|
# demucs
|
||||||
|
|
||||||
Source separation demos using [Demucs](https://github.com/facebookresearch/demucs) — splitting audio into stems (vocals, drums, bass, other).
|
Source separation demos using [Demucs](https://github.com/facebookresearch/demucs) — splitting audio into stems (vocals, drums, bass, other).
|
||||||
|
|
||||||
|
## Demo run
|
||||||
|
|
||||||
|
Ran two-stem separation (vocals vs. everything else) on an RTX 3090 against
|
||||||
|
`media-files/audio/service-mbrs-ntscrm-01840194-01840194.m4a`:
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 -m demucs --two-stems=vocals -n htdemucs \
|
||||||
|
-o separated \
|
||||||
|
../media-files/audio/service-mbrs-ntscrm-01840194-01840194.m4a
|
||||||
|
```
|
||||||
|
|
||||||
|
Took ~12s for 11 minutes of audio. Outputs `vocals.wav` and `no_vocals.wav` (44.1kHz stereo PCM,
|
||||||
|
duration matches source) under `separated/htdemucs/<track-name>/` — gitignored, left on the GPU
|
||||||
|
box at `/root/projects/audio-demo/repo/demucs/separated/`.
|
||||||
|
|
||||||
|
**Known gotcha:** demucs 4.1.0's package metadata doesn't declare `numpy` as a dependency even
|
||||||
|
though `demucs/transformer.py` imports it directly — `pip install demucs` alone leaves the env
|
||||||
|
broken (`ModuleNotFoundError: No module named 'numpy'` at runtime). Install `numpy` explicitly
|
||||||
|
alongside it.
|
||||||
|
|
||||||
|
The isolated `vocals.wav` is intended as the clean reference/source audio for a voice-cloning TTS
|
||||||
|
step (see top-level README for the planned audio-translation pipeline).
|
||||||
Reference in new issue
Block a user