feat: initial setup - voices, demo script, README

2026-07-18 13:43:27 +00:00
parent 298e6987bf
commit a865d76c81
6 changed files with 243 additions and 2 deletions
@@ -1,3 +1,120 @@
-# audio-gen-lab
+# Audio Gen Lab

-Multi-speaker audio generation experiments with VibeVoice, Qwen3-TTS voice design, and ComfyUI workflows
+Multi-speaker audio generation experiments using ComfyUI on an RTX 3090 (24GB VRAM).
+
+## Goal
+
+Generate multi-character audio content (podcasts, dialogues) using:
+1. **Qwen3-TTS VoiceDesign** — Design custom character voices via natural language descriptions
+2. **VibeVoice 7B** — Generate multi-speaker audio (up to 4 characters) from designed voices
+
+## Pipeline
+
+```
+Qwen3-TTS VoiceDesignerNode → generates reference voice audio → saved to voices/
+VibeVoiceEngineNode → loads reference voices as speaker references
+UnifiedTTSTextNode → feeds script with [Character] tags → multi-speaker output
+SaveAudioMP3 → final podcast audio
+```
+
+## Server
+
+- **ComfyUI:** http://10.0.0.113:8188
+- **GPU:** NVIDIA RTX 3090 (25.3GB VRAM)
+- **TTS Audio Suite:** Installed (11 engines)
+
+## Installed Models
+
+### VibeVoice
+- `vibevoice-7B` ✅ — Fully downloaded (~18GB), ready to use
+- `vibevoice-1.5B` ⚠️ — Partially downloaded (incomplete files)
+
+### Qwen3-TTS
+- `Qwen3-TTS-12Hz-1.7B-Base` ❌ — Needs download (voice cloning)
+- `Qwen3-TTS-12Hz-1.7B-CustomVoice` ❌ — Needs download (preset voices)
+- `Qwen3-TTS-12Hz-1.7B-VoiceDesign` ❌ — Needs download (voice design via description)
+
+To download, put models in: `ComfyUI/models/TTS/Qwen3-TTS/`
+Source: https://huggingface.co/collections/Qwen/qwen3-tts
+
+## Available TTS Engines (TTS Audio Suite)
+
+| Engine | Multi-Speaker | VRAM | Use Case |
+|--------|--------------|------|----------|
+| VibeVoice 7B | ✅ Up to 4 | ~18GB | Multi-character dialogue |
+| VibeVoice 1.5B | ✅ Up to 4 | ~7GB | Faster, lower quality |
+| F5-TTS | ❌ | ~4GB | Voice cloning, fast |
+| Higgs Audio 2/3 | ❌ | ~8GB | Expressive single speaker |
+| Qwen3-TTS | ❌ | ~12GB | Multilingual, voice design |
+| IndexTTS-2 | ❌ | ~4GB | Emotion control |
+| Echo-TTS | ❌ | ~6GB | Voice cloning |
+| CosyVoice 3 | ❌ | ~6GB | 23 languages |
+| ChatterBox | ❌ | ~3GB | Lightweight, reliable |
+| Step Audio EditX | ❌ | ~6GB | Audio editing |
+| RVC | ❌ | ~2GB | Pitch-based voice swap |
+
+## Workflow
+
+### Phase 1: Voice Design (Qwen3-TTS)
+1. Create `Qwen3TTSEngineNode` with model `VoiceDesign`, size `1.7B`
+2. Create `Qwen3TTSVoiceDesignerNode`
+3. Write voice descriptions (see `voices/` directory)
+4. Generate reference audio for each character
+5. Save to `models/voices/{character_name}/`
+
+### Phase 2: Multi-Speaker Generation (VibeVoice)
+1. Create `VibeVoiceEngineNode` with model `vibevoice-7B`
+2. Create `UnifiedTTSTextNode`
+3. Load script from `scripts/` with `[Character]` tags
+4. Feed character voice references as speakers
+5. Generate and save output
+
+### Phase 3: Test & Iterate
+- Test with the demo podcast script
+- Refine voice designs
+- Optimise for quality vs speed
+
+## Scripts
+
+- `scripts/demo-podcast.md` — Demo podcast script (4 characters, ~5 minutes)
+
+## Voices
+
+Character voice definitions live here:
+- `voices/alice.md` — Host, warm British female
+- `voices/bob.md` — Co-host, energetic American male
+- `voices/guest1.md` — Expert guest, calm academic
+- `voices/guest2.md` — Second guest, playful personality
+
+## ComfyUI Node References
+
+### Voice Design
+- `Qwen3TTSEngineNode` — Engine config (model: VoiceDesign, size: 1.7B)
+- `Qwen3TTSVoiceDesignerNode` — Voice designer (description → audio)
+  - `voice_description`: Natural language voice description
+  - `character_name`: Saves to `models/voices/{name}/`
+  - `reference_text`: Text to generate preview with
+
+### Multi-Speaker
+- `VibeVoiceEngineNode` — Engine config (model: vibevoice-7B, multi_speaker_mode: enabled)
+- `UnifiedTTSTextNode` — Main TTS node
+  - `text`: Script with `[Character]` speaker tags
+  - `narrator_voice`: Default speaker
+  - `speaker1_voice` through `speaker4_voice`: Character voice references
+- `SaveAudioMP3` — Export final audio
+
+## ComfyUI API
+
+Workflows can be submitted via the ComfyUI API:
+```bash
+# Submit workflow
+curl -X POST http://10.0.0.113:8188/prompt \
+  -H "Content-Type: application/json" \
+  -d @workflow.json
+
+# Check status
+curl -s http://10.0.0.113:8188/history | python3 -m json.tool
+
+# Get outputs
+curl -s http://10.0.0.113:8188/system_stats
+```