Add AuK voice-editing demo
Edits emotion on an existing recording in place (same speaker, words, duration) rather than steering style at generation time. Demo runs "Change the emotion to excited." on the public-domain narrator clip already used by whisperx/tts. Found and documented three real environment/packaging gaps on the RTX 3090: a missing audioread dependency, a huggingface_hub/transformers version conflict from installing the hf CLI, and --cpu_offload being mandatory (not optional) since AuK's own undoffloaded VRAM figures exceed 24GB entirely. Also found the practical input-length ceiling is ~25-30s, not the several minutes a podcast episode would need — an 80s clip OOM'd on attention memory even with offload enabled. Co-Authored-By: Claude Sonnet 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Y1oQ4h4WE4R9KRvMQLr5Ey
This commit is contained in:
1 parent
d072cdaa07
commit
c137fbae0c
5 files changed
+121
-1
No files matched your search
@@ -0,0 +1,47 @@
|
||||
"""
|
||||
Voice-editing demo using Tencent's AuK (https://github.com/Tencent-Hunyuan/AuK).
|
||||
|
||||
Unlike MOSS-TTS's `instruction` field (steers style at generation time),
|
||||
AuK edits emotion on an *existing* recording directly - same speaker, same
|
||||
words, same duration, different delivery. Demonstrates on the public-domain
|
||||
narrator clip already used by the whisperx/tts demos.
|
||||
|
||||
Run from the auk/ directory with its .venv activated.
|
||||
"""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from auk.infer.infer_auk import AukInfer, save_audio
|
||||
|
||||
HERE = Path(__file__).parent
|
||||
CONFIG = HERE / "ckpts" / "AuK" / "config.yaml"
|
||||
CKPT = HERE / "ckpts" / "AuK" / "auk_base.safetensors"
|
||||
QWEN_PATH = HERE / "ckpts" / "Qwen2.5-Omni-3B"
|
||||
INPUT_AUDIO = HERE.parent / "tts" / "reference" / "narrator_ref_en.wav"
|
||||
OUTPUT_PATH = HERE / "narrator_excited.wav"
|
||||
|
||||
INSTRUCTION = "Change the emotion to excited."
|
||||
|
||||
|
||||
def main() -> None:
|
||||
engine = AukInfer(
|
||||
str(CONFIG), str(CKPT), qwen_path=str(QWEN_PATH), cpu_offload=True
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": INSTRUCTION},
|
||||
{"type": "audio", "audio": str(INPUT_AUDIO)},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
audio, sr = engine.generate(messages)
|
||||
save_audio(audio, sr, str(OUTPUT_PATH))
|
||||
print(f"[INFO] Saved {OUTPUT_PATH}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in new issue
Block a user