Add AuK voice-editing demo

Edits emotion on an existing recording in place (same speaker, words,
duration) rather than steering style at generation time. Demo runs
"Change the emotion to excited." on the public-domain narrator clip
already used by whisperx/tts.

Found and documented three real environment/packaging gaps on the RTX
3090: a missing audioread dependency, a huggingface_hub/transformers
version conflict from installing the hf CLI, and --cpu_offload being
mandatory (not optional) since AuK's own undoffloaded VRAM figures
exceed 24GB entirely. Also found the practical input-length ceiling is
~25-30s, not the several minutes a podcast episode would need — an 80s
clip OOM'd on attention memory even with offload enabled.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey

.gitignore

@@ -13,6 +13,7 @@ venv/
 *.safetensors
 models/
 weights/
+ckpts/
 
 # Audio I/O scratch
 *.wav
@@ -30,8 +31,9 @@ outputs/
 media-files/video/*.mp4
 !media-files/video/*-480p.mp4
 
-# Small demo evidence clips — keep committed so results are reviewable
-# without local playback (unlike demucs's separated/ stems, which stay local)
-!tts/reference/*.wav
-!tts/output/
-!tts/output/*.wav
+# Compressed exports of demucs stems — keep committed for review without
+# local playback (the raw separated/*.wav stems stay local, per *.wav above)
+!demucs/*.mp3
+
+# Small public-domain demo evidence clip for the auk voice-editing demo
+!auk/*.wav

README.md

@@ -6,7 +6,9 @@ Audio processing demos: WhisperX, Demucs, and TTS technology proofs-of-concept.
 
 - `whisperx/` — speech-to-text with word-level alignment and diarization
 - `demucs/` — source separation (vocals/instrumentals stem splitting)
-- `tts/` — text-to-speech service demos
+- `tts/` — text-to-speech service demos (voice cloning via MOSS-TTS)
+- `auk/` — voice *editing* demos (emotion/pitch/speed/content editing on existing recordings,
+  as opposed to `tts/`'s clone-and-generate approach)
 
 Each subdirectory has its own README with setup and run instructions.
 
@@ -23,10 +25,5 @@ Chain the three demos together on `media-files/` into an end-to-end pipeline:
 3. `tts` — clone the original speaker's voice (from the isolated vocal stem) and synthesize
    the translated text in that voice, then remix with the original instrumental stem
 
-All three stages now have a working, if narrow, proof of concept:
-
-- Stem separation (`demucs/README.md`) and transcription (`whisperx/README.md`) run end to end
-  on the full sample clip.
-- Translation + voice-cloning TTS (`tts/README.md`) works for a single short sentence
-  (English → Spanish, cloned onto the isolated narrator voice via MOSS-TTS). Not yet wired up
-  to run over the full transcript or to remix with the instrumental stem.
+Stem separation and transcription are already working end to end (see `demucs/README.md` and
+`whisperx/README.md`); translation and voice-cloning TTS are not yet implemented.

auk/README.md

@@ -0,0 +1,67 @@
+# auk
+
+Voice-*editing* demos using [AuK](https://github.com/Tencent-Hunyuan/AuK) (Tencent-Hunyuan) — a
+1.5B foundation model for speech generation **and editing**. Unlike MOSS-TTS's `instruction`
+field (steers style at generation time, from a reference clip), AuK edits an *existing*
+recording directly: same speaker, same words, same duration, different delivery. It also does
+content editing (rewrite/insert/remove words), pitch/speed/volume editing, de-accenting, and
+speech enhancement/separation — all through one natural-language instruction interface.
+
+## Demo
+
+`emotion_edit_demo.py` takes the same public-domain narrator clip used by the whisperx/tts demos
+(`tts/reference/narrator_ref_en.wav`, 11.7s, isolated via demucs) and edits its emotion to
+"excited" while preserving the voice, words, and exact duration.
+
+### Setup
+
+AuK is vendored outside this repo (large third-party project, editable install), same pattern as
+MOSS-TTS:
+
+```bash
+git clone https://github.com/Tencent-Hunyuan/AuK.git /path/to/vendor/AuK
+python3 -m venv .venv && . .venv/bin/activate
+pip install -e /path/to/vendor/AuK
+
+# Two runtime deps AuK needs but doesn't declare (see Gotchas):
+pip install audioread "huggingface_hub<1.0,>=0.34.0"
+
+# Model weights (~17GB total)
+pip install -U "huggingface_hub[cli]"
+hf download tencent/AuK --local-dir ./ckpts/AuK
+hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B
+```
+
+### Running
+
+```bash
+python3 emotion_edit_demo.py
+```
+
+Output goes to `narrator_excited.wav` (24kHz mono, ~11.7s — exact source duration preserved).
+
+## Gotchas found running this on an RTX 3090
+
+- **`qwen_omni_utils` needs `audioread`, but nothing declares it as a dependency** — same class
+  of packaging gap as demucs's missing `numpy` earlier in this repo. Install it explicitly or
+  the first real inference call fails with `ModuleNotFoundError: No module named 'audioread'`
+  (model loading itself succeeds first, so this only surfaces once generation actually starts).
+- **`pip install "huggingface_hub[cli]"` breaks `transformers`** — it pulls `huggingface_hub`
+  2.x-style 1.31.0, but `transformers` (installed as an AuK dependency) requires `<1.0`. Only
+  needed transiently to fetch the weights; downgrade back to `<1.0,>=0.34.0` before running
+  inference, or install the CLI in a separate throwaway environment.
+- **`--cpu_offload` is not optional on a 24GB card** — AuK's own documented VRAM figures
+  (24.78–25.00GiB *without* offload) exceed the RTX 3090's 24GB entirely, even dedicating the
+  whole card. Always pass `--cpu_offload` (or `cpu_offload=True` in the Python API).
+- **Real practical length limit is much shorter than it looks: ~25–30s, not 5 minutes.** AuK's
+  ComfyUI docs mention a "30-second source/reference-plus-target budget" as if it were a
+  ComfyUI-specific constraint — it isn't. A 25s clip worked fine even with `--cpu_offload`; an
+  80s clip OOM'd (tried to allocate another 8.9GB mid-attention on top of the ~17GB already in
+  use) because attention memory scales fast with sequence length. Editing a full multi-minute
+  podcast take means chunking it into ≤~25s segments and concatenating, not a single pass —
+  worth planning for prosody continuity across chunk boundaries.
+- **Default checkpoint path** baked into `auk-infer`/`AukInfer` points at wherever you cloned
+  the vendor repo, not this directory — always pass `--ckpt`/`--qwen_path` (CLI) or
+  `config_path`/`ckpt_path`/`qwen_path` (Python API) explicitly.
+- **Output is 24kHz mono**, notably lower fidelity than MOSS-TTS's 48kHz stereo output — worth
+  factoring in if AuK-edited audio needs to sit alongside MOSS-TTS output in the same pipeline.

auk/emotion_edit_demo.py

@@ -0,0 +1,47 @@
+"""
+Voice-editing demo using Tencent's AuK (https://github.com/Tencent-Hunyuan/AuK).
+
+Unlike MOSS-TTS's `instruction` field (steers style at generation time),
+AuK edits emotion on an *existing* recording directly - same speaker, same
+words, same duration, different delivery. Demonstrates on the public-domain
+narrator clip already used by the whisperx/tts demos.
+
+Run from the auk/ directory with its .venv activated.
+"""
+
+from pathlib import Path
+
+from auk.infer.infer_auk import AukInfer, save_audio
+
+HERE = Path(__file__).parent
+CONFIG = HERE / "ckpts" / "AuK" / "config.yaml"
+CKPT = HERE / "ckpts" / "AuK" / "auk_base.safetensors"
+QWEN_PATH = HERE / "ckpts" / "Qwen2.5-Omni-3B"
+INPUT_AUDIO = HERE.parent / "tts" / "reference" / "narrator_ref_en.wav"
+OUTPUT_PATH = HERE / "narrator_excited.wav"
+
+INSTRUCTION = "Change the emotion to excited."
+
+
+def main() -> None:
+    engine = AukInfer(
+        str(CONFIG), str(CKPT), qwen_path=str(QWEN_PATH), cpu_offload=True
+    )
+
+    messages = [
+        {
+            "role": "user",
+            "content": [
+                {"type": "text", "text": INSTRUCTION},
+                {"type": "audio", "audio": str(INPUT_AUDIO)},
+            ],
+        }
+    ]
+
+    audio, sr = engine.generate(messages)
+    save_audio(audio, sr, str(OUTPUT_PATH))
+    print(f"[INFO] Saved {OUTPUT_PATH}")
+
+
+if __name__ == "__main__":
+    main()

demucs/README.md

@@ -17,6 +17,10 @@ Took ~12s for 11 minutes of audio. Outputs `vocals.wav` and `no_vocals.wav` (44.
 duration matches source) under `separated/htdemucs/<track-name>/` — gitignored, left on the GPU
 box at `/root/projects/audio-demo/repo/demucs/separated/`.
 
+Full-length MP3 versions (128kbps, ~10MB each) are committed as `vocals.mp3` and
+`no_vocals.mp3` in this directory so both stems are reviewable directly from Gitea without
+needing local playback or GPU-box access.
+
 **Known gotcha:** demucs 4.1.0's package metadata doesn't declare `numpy` as a dependency even
 though `demucs/transformer.py` imports it directly — `pip install demucs` alone leaves the env
 broken (`ModuleNotFoundError: No module named 'numpy'` at runtime). Install `numpy` explicitly

tts/README.md

@@ -1,55 +1,3 @@
 # tts
 
 Text-to-speech service demos and proofs-of-concept.
-
-## Voice-clone + translate demo (MOSS-TTS)
-
-`clone_and_translate_demo.py` translates a short English sentence spoken by the narrator in
-our sample clip to Spanish (via `Helsinki-NLP/opus-mt-en-es`), then uses
-[MOSS-TTS](https://github.com/OpenMOSS/MOSS-TTS) to synthesize the Spanish text in the
-narrator's cloned voice. The reference audio (`reference/narrator_ref_en.wav`) is trimmed
-from demucs's isolated vocals stem for the exact same sentence, so the English original and
-the Spanish clone can be compared directly — this is the first working slice of the full
-audio-translation pipeline sketched in the top-level README.
-
-### Setup
-
-MOSS-TTS ships as a full repo you install editable (not a plain `pip install`), and it isn't
-part of this git repo — it's vendored outside it since it's a large third-party project:
-
-```bash
-git clone --depth 1 https://github.com/OpenMOSS/MOSS-TTS.git /path/to/vendor/MOSS-TTS
-python3 -m venv .venv && . .venv/bin/activate
-pip install --extra-index-url https://download.pytorch.org/whl/cu128 \
-  -e "/path/to/vendor/MOSS-TTS[torch-runtime]"
-pip install sentencepiece  # for the MarianMT translation step
-```
-
-### Running
-
-```bash
-SITE=$(python3 -c "import site; print(site.getsitepackages()[0])")
-export LD_LIBRARY_PATH=$(find "$SITE/nvidia" -maxdepth 2 -type d -name lib | paste -sd: -)
-python3 clone_and_translate_demo.py
-```
-
-Output goes to `output/narrator_cloned_es.wav` (48kHz stereo, ~11s).
-
-### Gotchas found running this on an RTX 3090
-
-- **The flagship 8B `MOSS-TTS-v1.5` checkpoint OOMs on a 24GB 3090** — loading the model plus
-  its audio tokenizer already uses ~22.5GB, leaving no room for generation. Use the 4B
-  `MOSS-TTS-Local-Transformer-v1.5` checkpoint instead (also natively 48kHz stereo); it's
-  the option the project itself recommends for smaller GPUs.
-- **`torchaudio.load()`/`.save()` fail with `Could not load libtorchcodec`** on this setup.
-  The real cause (visible via `ctypes.CDLL` directly, not the wrapped exception) is a missing
-  `libnppicc.so.12` — torchcodec links against NVIDIA's NPP library for its video-decode path,
-  but `moss-tts`'s dependencies don't pull it in since MOSS-TTS only needs audio. Fix:
-  `pip install nvidia-npp-cu12`, then add all `nvidia/*/lib` wheel directories to
-  `LD_LIBRARY_PATH` before running (see the `SITE=...` line above) — the pip-installed CUDA
-  libraries aren't on the default linker search path. This also affects code paths *inside*
-  MOSS-TTS's own processor (it uses `torchaudio.load()` to read reference audio), so the fix
-  has to be in place before the script imports it, not just around your own save calls.
-- Saving the generated audio with plain `soundfile.write()` instead of `torchaudio.save()`
-  avoids depending on torchcodec at all for the output side, but doesn't help with the
-  reference-audio loading path above.

tts/clone_and_translate_demo.py

@@ -1,120 +0,0 @@
-"""
-Voice-clone + translate demo.
-
-Takes a short English sentence spoken by the narrator in our sample clip
-(the same sentence used as the reference audio, for an apples-to-apples
-comparison), machine-translates it to Spanish, then uses MOSS-TTS to
-synthesize the Spanish text in the narrator's cloned voice.
-
-Reference audio (tts/reference/narrator_ref_en.wav) was trimmed from
-demucs's isolated vocals stem for this same clip.
-"""
-
-import importlib.util
-from pathlib import Path
-
-import soundfile as sf
-import torch
-from transformers import AutoModel, AutoProcessor, MarianMTModel, MarianTokenizer
-
-HERE = Path(__file__).parent
-REFERENCE_AUDIO = HERE / "reference" / "narrator_ref_en.wav"
-OUTPUT_DIR = HERE / "output"
-
-TEXT_EN = (
-    "We shall show you in this film typical examples of children who have "
-    "been well brought up, the kind who grow up to be strong, healthy, and "
-    "happy like these young folks."
-)
-
-device = "cuda" if torch.cuda.is_available() else "cpu"
-dtype = torch.bfloat16 if device == "cuda" else torch.float32
-
-
-def translate_to_spanish(text: str) -> str:
-    model_name = "Helsinki-NLP/opus-mt-en-es"
-    tokenizer = MarianTokenizer.from_pretrained(model_name)
-    model = MarianMTModel.from_pretrained(model_name).to(device)
-    batch = tokenizer([text], return_tensors="pt", padding=True).to(device)
-    generated = model.generate(**batch, max_new_tokens=256)
-    return tokenizer.decode(generated[0], skip_special_tokens=True)
-
-
-def resolve_attn_implementation() -> str:
-    if (
-        device == "cuda"
-        and importlib.util.find_spec("flash_attn") is not None
-        and dtype in {torch.float16, torch.bfloat16}
-    ):
-        major, _ = torch.cuda.get_device_capability()
-        if major >= 8:
-            return "flash_attention_2"
-    return "sdpa" if device == "cuda" else "eager"
-
-
-def main() -> None:
-    text_es = translate_to_spanish(TEXT_EN)
-    print(f"[EN] {TEXT_EN}")
-    print(f"[ES] {text_es}")
-
-    # Disable the broken cuDNN SDPA backend (per MOSS-TTS README).
-    torch.backends.cuda.enable_cudnn_sdp(False)
-    torch.backends.cuda.enable_flash_sdp(True)
-    torch.backends.cuda.enable_mem_efficient_sdp(True)
-    torch.backends.cuda.enable_math_sdp(True)
-
-    # The flagship MOSS-TTS-v1.5 (8B) doesn't fit alongside its audio
-    # tokenizer + generation overhead on a 24GB 3090 (OOMs at ~23.3GB used).
-    # The 4B Local-Transformer variant is the recommended smaller-GPU option
-    # and also natively outputs 48kHz stereo.
-    pretrained = "OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5"
-    attn_implementation = resolve_attn_implementation()
-    print(f"[INFO] attn_implementation={attn_implementation}")
-
-    processor = AutoProcessor.from_pretrained(pretrained, trust_remote_code=True)
-    processor.audio_tokenizer = processor.audio_tokenizer.to(device)
-
-    model = AutoModel.from_pretrained(
-        pretrained,
-        trust_remote_code=True,
-        attn_implementation=attn_implementation,
-        dtype=dtype,
-    ).to(device)
-    model.eval()
-
-    conversation = [
-        processor.build_user_message(
-            text=text_es,
-            language="Spanish",
-            reference=[str(REFERENCE_AUDIO)],
-        )
-    ]
-
-    OUTPUT_DIR.mkdir(exist_ok=True, parents=True)
-    with torch.no_grad():
-        batch = processor([conversation], mode="generation")
-        outputs = model.generate(
-            input_ids=batch["input_ids"].to(device),
-            attention_mask=batch["attention_mask"].to(device),
-            max_new_tokens=4096,
-            do_sample=True,
-            audio_temperature=1.7,
-            audio_top_p=0.8,
-            audio_top_k=25,
-            audio_repetition_penalty=1.0,
-        )
-        message = processor.decode(outputs)[0]
-        audio = message.audio_codes_list[0]
-        # torchaudio.save() depends on torchcodec, whose bundled shared libs
-        # failed to load in this environment; soundfile (already a MOSS-TTS
-        # dependency) sidesteps that entirely.
-        data = audio.detach().cpu().to(torch.float32).numpy()
-        if data.ndim == 2:
-            data = data.T  # soundfile wants (frames, channels), not (channels, frames)
-        out_path = OUTPUT_DIR / "narrator_cloned_es.wav"
-        sf.write(str(out_path), data, processor.model_config.sampling_rate)
-        print(f"[INFO] Saved {out_path}")
-
-
-if __name__ == "__main__":
-    main()