audio-demo

Audio processing demos: WhisperX, Demucs, and TTS technology proofs-of-concept.

Structure

  • whisperx/ — speech-to-text with word-level alignment and diarization
  • demucs/ — source separation (vocals/instrumentals stem splitting)
  • tts/ — text-to-speech service demos (voice cloning via MOSS-TTS)
  • auk/ — voice editing demos (emotion/pitch/speed/content editing on existing recordings, as opposed to tts/'s clone-and-generate approach)
  • webui/ — interactive pipeline studio chaining demucs -> AuK -> MOSS-TTS with previews

Each subdirectory has its own README with setup and run instructions.

Requirements

Some demos are GPU-accelerated and are best run on a CUDA-capable machine (e.g. an RTX 3090).

Planned: full audio-translation demo

Chain the three demos together on media-files/ into an end-to-end pipeline:

  1. demucs — separate the source clip into vocals / instrumental stems
  2. whisperx — transcribe the isolated vocal stem, then machine-translate the transcript
  3. tts — clone the original speaker's voice (from the isolated vocal stem) and synthesize the translated text in that voice, then remix with the original instrumental stem

Stem separation and transcription are already working end to end (see demucs/README.md and whisperx/README.md); translation and voice-cloning TTS are not yet implemented.

S
Description
Audio processing demos: WhisperX, Demucs, and TTS technology proofs-of-concept.
Readme
60 MiB
0 Stars 4 Watchers 0 Forks
Languages
Python 100%