Add AuK voice-editing demo

Edits emotion on an existing recording in place (same speaker, words,
duration) rather than steering style at generation time. Demo runs
"Change the emotion to excited." on the public-domain narrator clip
already used by whisperx/tts.

Found and documented three real environment/packaging gaps on the RTX
3090: a missing audioread dependency, a huggingface_hub/transformers
version conflict from installing the hf CLI, and --cpu_offload being
mandatory (not optional) since AuK's own undoffloaded VRAM figures
exceed 24GB entirely. Also found the practical input-length ceiling is
~25-30s, not the several minutes a podcast episode would need — an 80s
clip OOM'd on attention memory even with offload enabled.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey

.gitignore

@@ -13,6 +13,7 @@ venv/
 *.safetensors
 models/
 weights/
+ckpts/
 
 # Audio I/O scratch
 *.wav
@@ -25,3 +26,14 @@ outputs/
 .DS_Store
 .idea/
 .vscode/
+
+# Full-size source video — keep locally, commit only compressed derivatives
+media-files/video/*.mp4
+!media-files/video/*-480p.mp4
+
+# Compressed exports of demucs stems — keep committed for review without
+# local playback (the raw separated/*.wav stems stay local, per *.wav above)
+!demucs/*.mp3
+
+# Small public-domain demo evidence clip for the auk voice-editing demo
+!auk/*.wav

README.md

@@ -6,10 +6,24 @@ Audio processing demos: WhisperX, Demucs, and TTS technology proofs-of-concept.
 
 - `whisperx/` — speech-to-text with word-level alignment and diarization
 - `demucs/` — source separation (vocals/instrumentals stem splitting)
-- `tts/` — text-to-speech service demos
+- `tts/` — text-to-speech service demos (voice cloning via MOSS-TTS)
+- `auk/` — voice *editing* demos (emotion/pitch/speed/content editing on existing recordings,
+  as opposed to `tts/`'s clone-and-generate approach)
 
 Each subdirectory has its own README with setup and run instructions.
 
 ## Requirements
 
 Some demos are GPU-accelerated and are best run on a CUDA-capable machine (e.g. an RTX 3090).
+
+## Planned: full audio-translation demo
+
+Chain the three demos together on `media-files/` into an end-to-end pipeline:
+
+1. `demucs` — separate the source clip into vocals / instrumental stems
+2. `whisperx` — transcribe the isolated vocal stem, then machine-translate the transcript
+3. `tts` — clone the original speaker's voice (from the isolated vocal stem) and synthesize
+   the translated text in that voice, then remix with the original instrumental stem
+
+Stem separation and transcription are already working end to end (see `demucs/README.md` and
+`whisperx/README.md`); translation and voice-cloning TTS are not yet implemented.

auk/README.md

@@ -0,0 +1,67 @@
+# auk
+
+Voice-*editing* demos using [AuK](https://github.com/Tencent-Hunyuan/AuK) (Tencent-Hunyuan) — a
+1.5B foundation model for speech generation **and editing**. Unlike MOSS-TTS's `instruction`
+field (steers style at generation time, from a reference clip), AuK edits an *existing*
+recording directly: same speaker, same words, same duration, different delivery. It also does
+content editing (rewrite/insert/remove words), pitch/speed/volume editing, de-accenting, and
+speech enhancement/separation — all through one natural-language instruction interface.
+
+## Demo
+
+`emotion_edit_demo.py` takes the same public-domain narrator clip used by the whisperx/tts demos
+(`tts/reference/narrator_ref_en.wav`, 11.7s, isolated via demucs) and edits its emotion to
+"excited" while preserving the voice, words, and exact duration.
+
+### Setup
+
+AuK is vendored outside this repo (large third-party project, editable install), same pattern as
+MOSS-TTS:
+
+```bash
+git clone https://github.com/Tencent-Hunyuan/AuK.git /path/to/vendor/AuK
+python3 -m venv .venv && . .venv/bin/activate
+pip install -e /path/to/vendor/AuK
+
+# Two runtime deps AuK needs but doesn't declare (see Gotchas):
+pip install audioread "huggingface_hub<1.0,>=0.34.0"
+
+# Model weights (~17GB total)
+pip install -U "huggingface_hub[cli]"
+hf download tencent/AuK --local-dir ./ckpts/AuK
+hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B
+```
+
+### Running
+
+```bash
+python3 emotion_edit_demo.py
+```
+
+Output goes to `narrator_excited.wav` (24kHz mono, ~11.7s — exact source duration preserved).
+
+## Gotchas found running this on an RTX 3090
+
+- **`qwen_omni_utils` needs `audioread`, but nothing declares it as a dependency** — same class
+  of packaging gap as demucs's missing `numpy` earlier in this repo. Install it explicitly or
+  the first real inference call fails with `ModuleNotFoundError: No module named 'audioread'`
+  (model loading itself succeeds first, so this only surfaces once generation actually starts).
+- **`pip install "huggingface_hub[cli]"` breaks `transformers`** — it pulls `huggingface_hub`
+  2.x-style 1.31.0, but `transformers` (installed as an AuK dependency) requires `<1.0`. Only
+  needed transiently to fetch the weights; downgrade back to `<1.0,>=0.34.0` before running
+  inference, or install the CLI in a separate throwaway environment.
+- **`--cpu_offload` is not optional on a 24GB card** — AuK's own documented VRAM figures
+  (24.78–25.00GiB *without* offload) exceed the RTX 3090's 24GB entirely, even dedicating the
+  whole card. Always pass `--cpu_offload` (or `cpu_offload=True` in the Python API).
+- **Real practical length limit is much shorter than it looks: ~25–30s, not 5 minutes.** AuK's
+  ComfyUI docs mention a "30-second source/reference-plus-target budget" as if it were a
+  ComfyUI-specific constraint — it isn't. A 25s clip worked fine even with `--cpu_offload`; an
+  80s clip OOM'd (tried to allocate another 8.9GB mid-attention on top of the ~17GB already in
+  use) because attention memory scales fast with sequence length. Editing a full multi-minute
+  podcast take means chunking it into ≤~25s segments and concatenating, not a single pass —
+  worth planning for prosody continuity across chunk boundaries.
+- **Default checkpoint path** baked into `auk-infer`/`AukInfer` points at wherever you cloned
+  the vendor repo, not this directory — always pass `--ckpt`/`--qwen_path` (CLI) or
+  `config_path`/`ckpt_path`/`qwen_path` (Python API) explicitly.
+- **Output is 24kHz mono**, notably lower fidelity than MOSS-TTS's 48kHz stereo output — worth
+  factoring in if AuK-edited audio needs to sit alongside MOSS-TTS output in the same pipeline.

auk/emotion_edit_demo.py

@@ -0,0 +1,47 @@
+"""
+Voice-editing demo using Tencent's AuK (https://github.com/Tencent-Hunyuan/AuK).
+
+Unlike MOSS-TTS's `instruction` field (steers style at generation time),
+AuK edits emotion on an *existing* recording directly - same speaker, same
+words, same duration, different delivery. Demonstrates on the public-domain
+narrator clip already used by the whisperx/tts demos.
+
+Run from the auk/ directory with its .venv activated.
+"""
+
+from pathlib import Path
+
+from auk.infer.infer_auk import AukInfer, save_audio
+
+HERE = Path(__file__).parent
+CONFIG = HERE / "ckpts" / "AuK" / "config.yaml"
+CKPT = HERE / "ckpts" / "AuK" / "auk_base.safetensors"
+QWEN_PATH = HERE / "ckpts" / "Qwen2.5-Omni-3B"
+INPUT_AUDIO = HERE.parent / "tts" / "reference" / "narrator_ref_en.wav"
+OUTPUT_PATH = HERE / "narrator_excited.wav"
+
+INSTRUCTION = "Change the emotion to excited."
+
+
+def main() -> None:
+    engine = AukInfer(
+        str(CONFIG), str(CKPT), qwen_path=str(QWEN_PATH), cpu_offload=True
+    )
+
+    messages = [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": INSTRUCTION},
+                {"type": "audio", "audio": str(INPUT_AUDIO)},
+            ],
+        }
+    ]
+
+    audio, sr = engine.generate(messages)
+    save_audio(audio, sr, str(OUTPUT_PATH))
+    print(f"[INFO] Saved {OUTPUT_PATH}")
+
+
+if __name__ == "__main__":
+    main()

demucs/README.md

@@ -1,3 +1,30 @@
 # demucs
 
 Source separation demos using [Demucs](https://github.com/facebookresearch/demucs) — splitting audio into stems (vocals, drums, bass, other).
+
+## Demo run
+
+Ran two-stem separation (vocals vs. everything else) on an RTX 3090 against
+`media-files/audio/service-mbrs-ntscrm-01840194-01840194.m4a`:
+
+```
+python3 -m demucs --two-stems=vocals -n htdemucs \
+  -o separated \
+  ../media-files/audio/service-mbrs-ntscrm-01840194-01840194.m4a
+```
+
+Took ~12s for 11 minutes of audio. Outputs `vocals.wav` and `no_vocals.wav` (44.1kHz stereo PCM,
+duration matches source) under `separated/htdemucs/<track-name>/` — gitignored, left on the GPU
+box at `/root/projects/audio-demo/repo/demucs/separated/`.
+
+Full-length MP3 versions (128kbps, ~10MB each) are committed as `vocals.mp3` and
+`no_vocals.mp3` in this directory so both stems are reviewable directly from Gitea without
+needing local playback or GPU-box access.
+
+**Known gotcha:** demucs 4.1.0's package metadata doesn't declare `numpy` as a dependency even
+though `demucs/transformer.py` imports it directly — `pip install demucs` alone leaves the env
+broken (`ModuleNotFoundError: No module named 'numpy'` at runtime). Install `numpy` explicitly
+alongside it.
+
+The isolated `vocals.wav` is intended as the clean reference/source audio for a voice-cloning TTS
+step (see top-level README for the planned audio-translation pipeline).