67 lines
3.6 KiB
Markdown
67 lines
3.6 KiB
Markdown
 
|
# auk
|
|||
|
|
|
|||
|
|
Voice-*editing* demos using [AuK](https://github.com/Tencent-Hunyuan/AuK) (Tencent-Hunyuan) — a
|
|||
|
|
1.5B foundation model for speech generation **and editing**. Unlike MOSS-TTS's `instruction`
|
|||
|
|
field (steers style at generation time, from a reference clip), AuK edits an *existing*
|
|||
|
|
recording directly: same speaker, same words, same duration, different delivery. It also does
|
|||
|
|
content editing (rewrite/insert/remove words), pitch/speed/volume editing, de-accenting, and
|
|||
|
|
speech enhancement/separation — all through one natural-language instruction interface.
|
|||
|
|
|
|||
|
|
## Demo
|
|||
|
|
|
|||
|
|
`emotion_edit_demo.py` takes the same public-domain narrator clip used by the whisperx/tts demos
|
|||
|
|
(`tts/reference/narrator_ref_en.wav`, 11.7s, isolated via demucs) and edits its emotion to
|
|||
|
|
"excited" while preserving the voice, words, and exact duration.
|
|||
|
|
|
|||
|
|
### Setup
|
|||
|
|
|
|||
|
|
AuK is vendored outside this repo (large third-party project, editable install), same pattern as
|
|||
|
|
MOSS-TTS:
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
git clone https://github.com/Tencent-Hunyuan/AuK.git /path/to/vendor/AuK
|
|||
|
|
python3 -m venv .venv && . .venv/bin/activate
|
|||
|
|
pip install -e /path/to/vendor/AuK
|
|||
|
|
|
|||
|
|
# Two runtime deps AuK needs but doesn't declare (see Gotchas):
|
|||
|
|
pip install audioread "huggingface_hub<1.0,>=0.34.0"
|
|||
|
|
|
|||
|
|
# Model weights (~17GB total)
|
|||
|
|
pip install -U "huggingface_hub[cli]"
|
|||
|
|
hf download tencent/AuK --local-dir ./ckpts/AuK
|
|||
|
|
hf download Qwen/Qwen2.5-Omni-3B --local-dir ./ckpts/Qwen2.5-Omni-3B
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
### Running
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
python3 emotion_edit_demo.py
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Output goes to `narrator_excited.wav` (24kHz mono, ~11.7s — exact source duration preserved).
|
|||
|
|
|
|||
|
|
## Gotchas found running this on an RTX 3090
|
|||
|
|
|
|||
|
|
- **`qwen_omni_utils` needs `audioread`, but nothing declares it as a dependency** — same class
|
|||
|
|
of packaging gap as demucs's missing `numpy` earlier in this repo. Install it explicitly or
|
|||
|
|
the first real inference call fails with `ModuleNotFoundError: No module named 'audioread'`
|
|||
|
|
(model loading itself succeeds first, so this only surfaces once generation actually starts).
|
|||
|
|
- **`pip install "huggingface_hub[cli]"` breaks `transformers`** — it pulls `huggingface_hub`
|
|||
|
|
2.x-style 1.31.0, but `transformers` (installed as an AuK dependency) requires `<1.0`. Only
|
|||
|
|
needed transiently to fetch the weights; downgrade back to `<1.0,>=0.34.0` before running
|
|||
|
|
inference, or install the CLI in a separate throwaway environment.
|
|||
|
|
- **`--cpu_offload` is not optional on a 24GB card** — AuK's own documented VRAM figures
|
|||
|
|
(24.78–25.00GiB *without* offload) exceed the RTX 3090's 24GB entirely, even dedicating the
|
|||
|
|
whole card. Always pass `--cpu_offload` (or `cpu_offload=True` in the Python API).
|
|||
|
|
- **Real practical length limit is much shorter than it looks: ~25–30s, not 5 minutes.** AuK's
|
|||
|
|
ComfyUI docs mention a "30-second source/reference-plus-target budget" as if it were a
|
|||
|
|
ComfyUI-specific constraint — it isn't. A 25s clip worked fine even with `--cpu_offload`; an
|
|||
|
|
80s clip OOM'd (tried to allocate another 8.9GB mid-attention on top of the ~17GB already in
|
|||
|
|
use) because attention memory scales fast with sequence length. Editing a full multi-minute
|
|||
|
|
podcast take means chunking it into ≤~25s segments and concatenating, not a single pass —
|
|||
|
|
worth planning for prosody continuity across chunk boundaries.
|
|||
|
|
- **Default checkpoint path** baked into `auk-infer`/`AukInfer` points at wherever you cloned
|
|||
|
|
the vendor repo, not this directory — always pass `--ckpt`/`--qwen_path` (CLI) or
|
|||
|
|
`config_path`/`ckpt_path`/`qwen_path` (Python API) explicitly.
|
|||
|
|
- **Output is 24kHz mono**, notably lower fidelity than MOSS-TTS's 48kHz stereo output — worth
|
|||
|
|
factoring in if AuK-edited audio needs to sit alongside MOSS-TTS output in the same pipeline.
|