audio-demo/auk/README.md

auk

Voice-editing demos using AuK (Tencent-Hunyuan) — a 1.5B foundation model for speech generation and editing. Unlike MOSS-TTS's instruction field (steers style at generation time, from a reference clip), AuK edits an existing recording directly: same speaker, same words, same duration, different delivery. It also does content editing (rewrite/insert/remove words), pitch/speed/volume editing, de-accenting, and speech enhancement/separation — all through one natural-language instruction interface.

Demo

emotion_edit_demo.py takes the same public-domain narrator clip used by the whisperx/tts demos (tts/reference/narrator_ref_en.wav, 11.7s, isolated via demucs) and edits its emotion to "excited" while preserving the voice, words, and exact duration.

Setup

AuK is vendored outside this repo (large third-party project, editable install), same pattern as MOSS-TTS:

git clone https://github.com/Tencent-Hunyuan/AuK.git /path/to/vendor/AuK
python3 -m venv .venv && . .venv/bin/activate
pip install -e /path/to/vendor/AuK

# Two runtime deps AuK needs but doesn't declare (see Gotchas):
pip install audioread "huggingface_hub<1.0,>=0.34.0"

# Model weights (~17GB total)
pip install -U "huggingface_hub[cli]"
hf download tencent/AuK --local-dir ./ckpts/AuK
hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B

Running

python3 emotion_edit_demo.py

Output goes to narrator_excited.wav (24kHz mono, ~11.7s — exact source duration preserved).

Gotchas found running this on an RTX 3090

• qwen_omni_utils needs audioread, but nothing declares it as a dependency — same class of packaging gap as demucs's missing numpy earlier in this repo. Install it explicitly or the first real inference call fails with ModuleNotFoundError: No module named 'audioread' (model loading itself succeeds first, so this only surfaces once generation actually starts).
• pip install "huggingface_hub[cli]" breaks transformers — it pulls huggingface_hub 2.x-style 1.31.0, but transformers (installed as an AuK dependency) requires <1.0. Only needed transiently to fetch the weights; downgrade back to <1.0,>=0.34.0 before running inference, or install the CLI in a separate throwaway environment.
• --cpu_offload is not optional on a 24GB card — AuK's own documented VRAM figures (24.78–25.00GiB without offload) exceed the RTX 3090's 24GB entirely, even dedicating the whole card. Always pass --cpu_offload (or cpu_offload=True in the Python API).
• Real practical length limit is much shorter than it looks: ~25–30s, not 5 minutes. AuK's ComfyUI docs mention a "30-second source/reference-plus-target budget" as if it were a ComfyUI-specific constraint — it isn't. A 25s clip worked fine even with --cpu_offload; an 80s clip OOM'd (tried to allocate another 8.9GB mid-attention on top of the ~17GB already in use) because attention memory scales fast with sequence length. Editing a full multi-minute podcast take means chunking it into ≤~25s segments and concatenating, not a single pass — worth planning for prosody continuity across chunk boundaries.
• Default checkpoint path baked into auk-infer/AukInfer points at wherever you cloned the vendor repo, not this directory — always pass --ckpt/--qwen_path (CLI) or config_path/ckpt_path/qwen_path (Python API) explicitly.
• Output is 24kHz mono, notably lower fidelity than MOSS-TTS's 48kHz stereo output — worth factoring in if AuK-edited audio needs to sit alongside MOSS-TTS output in the same pipeline.