steve-adminandClaude Sonnet 5
c137fbae0c
Edits emotion on an existing recording in place (same speaker, words, duration) rather than steering style at generation time. Demo runs "Change the emotion to excited." on the public-domain narrator clip already used by whisperx/tts. Found and documented three real environment/packaging gaps on the RTX 3090: a missing audioread dependency, a huggingface_hub/transformers version conflict from installing the hf CLI, and --cpu_offload being mandatory (not optional) since AuK's own undoffloaded VRAM figures exceed 24GB entirely. Also found the practical input-length ceiling is ~25-30s, not the several minutes a podcast episode would need — an 80s clip OOM'd on attention memory even with offload enabled. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
68 lines
3.6 KiB
Markdown
68 lines
3.6 KiB
Markdown
# auk
|
||
|
||
Voice-*editing* demos using [AuK](https://github.com/Tencent-Hunyuan/AuK) (Tencent-Hunyuan) — a
|
||
1.5B foundation model for speech generation **and editing**. Unlike MOSS-TTS's `instruction`
|
||
field (steers style at generation time, from a reference clip), AuK edits an *existing*
|
||
recording directly: same speaker, same words, same duration, different delivery. It also does
|
||
content editing (rewrite/insert/remove words), pitch/speed/volume editing, de-accenting, and
|
||
speech enhancement/separation — all through one natural-language instruction interface.
|
||
|
||
## Demo
|
||
|
||
`emotion_edit_demo.py` takes the same public-domain narrator clip used by the whisperx/tts demos
|
||
(`tts/reference/narrator_ref_en.wav`, 11.7s, isolated via demucs) and edits its emotion to
|
||
"excited" while preserving the voice, words, and exact duration.
|
||
|
||
### Setup
|
||
|
||
AuK is vendored outside this repo (large third-party project, editable install), same pattern as
|
||
MOSS-TTS:
|
||
|
||
```bash
|
||
git clone https://github.com/Tencent-Hunyuan/AuK.git /path/to/vendor/AuK
|
||
python3 -m venv .venv && . .venv/bin/activate
|
||
pip install -e /path/to/vendor/AuK
|
||
|
||
# Two runtime deps AuK needs but doesn't declare (see Gotchas):
|
||
pip install audioread "huggingface_hub<1.0,>=0.34.0"
|
||
|
||
# Model weights (~17GB total)
|
||
pip install -U "huggingface_hub[cli]"
|
||
hf download tencent/AuK --local-dir ./ckpts/AuK
|
||
hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B
|
||
```
|
||
|
||
### Running
|
||
|
||
```bash
|
||
python3 emotion_edit_demo.py
|
||
```
|
||
|
||
Output goes to `narrator_excited.wav` (24kHz mono, ~11.7s — exact source duration preserved).
|
||
|
||
## Gotchas found running this on an RTX 3090
|
||
|
||
- **`qwen_omni_utils` needs `audioread`, but nothing declares it as a dependency** — same class
|
||
of packaging gap as demucs's missing `numpy` earlier in this repo. Install it explicitly or
|
||
the first real inference call fails with `ModuleNotFoundError: No module named 'audioread'`
|
||
(model loading itself succeeds first, so this only surfaces once generation actually starts).
|
||
- **`pip install "huggingface_hub[cli]"` breaks `transformers`** — it pulls `huggingface_hub`
|
||
2.x-style 1.31.0, but `transformers` (installed as an AuK dependency) requires `<1.0`. Only
|
||
needed transiently to fetch the weights; downgrade back to `<1.0,>=0.34.0` before running
|
||
inference, or install the CLI in a separate throwaway environment.
|
||
- **`--cpu_offload` is not optional on a 24GB card** — AuK's own documented VRAM figures
|
||
(24.78–25.00GiB *without* offload) exceed the RTX 3090's 24GB entirely, even dedicating the
|
||
whole card. Always pass `--cpu_offload` (or `cpu_offload=True` in the Python API).
|
||
- **Real practical length limit is much shorter than it looks: ~25–30s, not 5 minutes.** AuK's
|
||
ComfyUI docs mention a "30-second source/reference-plus-target budget" as if it were a
|
||
ComfyUI-specific constraint — it isn't. A 25s clip worked fine even with `--cpu_offload`; an
|
||
80s clip OOM'd (tried to allocate another 8.9GB mid-attention on top of the ~17GB already in
|
||
use) because attention memory scales fast with sequence length. Editing a full multi-minute
|
||
podcast take means chunking it into ≤~25s segments and concatenating, not a single pass —
|
||
worth planning for prosody continuity across chunk boundaries.
|
||
- **Default checkpoint path** baked into `auk-infer`/`AukInfer` points at wherever you cloned
|
||
the vendor repo, not this directory — always pass `--ckpt`/`--qwen_path` (CLI) or
|
||
`config_path`/`ckpt_path`/`qwen_path` (Python API) explicitly.
|
||
- **Output is 24kHz mono**, notably lower fidelity than MOSS-TTS's 48kHz stereo output — worth
|
||
factoring in if AuK-edited audio needs to sit alongside MOSS-TTS output in the same pipeline.
|