121 lines
4.1 KiB
Markdown
121 lines
4.1 KiB
Markdown
# Audio Gen Lab
|
|
|
|
Multi-speaker audio generation experiments using ComfyUI on an RTX 3090 (24GB VRAM).
|
|
|
|
## Goal
|
|
|
|
Generate multi-character audio content (podcasts, dialogues) using:
|
|
1. **Qwen3-TTS VoiceDesign** — Design custom character voices via natural language descriptions
|
|
2. **VibeVoice 7B** — Generate multi-speaker audio (up to 4 characters) from designed voices
|
|
|
|
## Pipeline
|
|
|
|
```
|
|
Qwen3-TTS VoiceDesignerNode → generates reference voice audio → saved to voices/
|
|
VibeVoiceEngineNode → loads reference voices as speaker references
|
|
UnifiedTTSTextNode → feeds script with [Character] tags → multi-speaker output
|
|
SaveAudioMP3 → final podcast audio
|
|
```
|
|
|
|
## Server
|
|
|
|
- **ComfyUI:** http://10.0.0.113:8188
|
|
- **GPU:** NVIDIA RTX 3090 (25.3GB VRAM)
|
|
- **TTS Audio Suite:** Installed (11 engines)
|
|
|
|
## Installed Models
|
|
|
|
### VibeVoice
|
|
- `vibevoice-7B` ✅ — Fully downloaded (~18GB), ready to use
|
|
- `vibevoice-1.5B` ⚠️ — Partially downloaded (incomplete files)
|
|
|
|
### Qwen3-TTS
|
|
- `Qwen3-TTS-12Hz-1.7B-Base` ❌ — Needs download (voice cloning)
|
|
- `Qwen3-TTS-12Hz-1.7B-CustomVoice` ❌ — Needs download (preset voices)
|
|
- `Qwen3-TTS-12Hz-1.7B-VoiceDesign` ❌ — Needs download (voice design via description)
|
|
|
|
To download, put models in: `ComfyUI/models/TTS/Qwen3-TTS/`
|
|
Source: https://huggingface.co/collections/Qwen/qwen3-tts
|
|
|
|
## Available TTS Engines (TTS Audio Suite)
|
|
|
|
| Engine | Multi-Speaker | VRAM | Use Case |
|
|
|--------|--------------|------|----------|
|
|
| VibeVoice 7B | ✅ Up to 4 | ~18GB | Multi-character dialogue |
|
|
| VibeVoice 1.5B | ✅ Up to 4 | ~7GB | Faster, lower quality |
|
|
| F5-TTS | ❌ | ~4GB | Voice cloning, fast |
|
|
| Higgs Audio 2/3 | ❌ | ~8GB | Expressive single speaker |
|
|
| Qwen3-TTS | ❌ | ~12GB | Multilingual, voice design |
|
|
| IndexTTS-2 | ❌ | ~4GB | Emotion control |
|
|
| Echo-TTS | ❌ | ~6GB | Voice cloning |
|
|
| CosyVoice 3 | ❌ | ~6GB | 23 languages |
|
|
| ChatterBox | ❌ | ~3GB | Lightweight, reliable |
|
|
| Step Audio EditX | ❌ | ~6GB | Audio editing |
|
|
| RVC | ❌ | ~2GB | Pitch-based voice swap |
|
|
|
|
## Workflow
|
|
|
|
### Phase 1: Voice Design (Qwen3-TTS)
|
|
1. Create `Qwen3TTSEngineNode` with model `VoiceDesign`, size `1.7B`
|
|
2. Create `Qwen3TTSVoiceDesignerNode`
|
|
3. Write voice descriptions (see `voices/` directory)
|
|
4. Generate reference audio for each character
|
|
5. Save to `models/voices/{character_name}/`
|
|
|
|
### Phase 2: Multi-Speaker Generation (VibeVoice)
|
|
1. Create `VibeVoiceEngineNode` with model `vibevoice-7B`
|
|
2. Create `UnifiedTTSTextNode`
|
|
3. Load script from `scripts/` with `[Character]` tags
|
|
4. Feed character voice references as speakers
|
|
5. Generate and save output
|
|
|
|
### Phase 3: Test & Iterate
|
|
- Test with the demo podcast script
|
|
- Refine voice designs
|
|
- Optimise for quality vs speed
|
|
|
|
## Scripts
|
|
|
|
- `scripts/demo-podcast.md` — Demo podcast script (4 characters, ~5 minutes)
|
|
|
|
## Voices
|
|
|
|
Character voice definitions live here:
|
|
- `voices/alice.md` — Host, warm British female
|
|
- `voices/bob.md` — Co-host, energetic American male
|
|
- `voices/guest1.md` — Expert guest, calm academic
|
|
- `voices/guest2.md` — Second guest, playful personality
|
|
|
|
## ComfyUI Node References
|
|
|
|
### Voice Design
|
|
- `Qwen3TTSEngineNode` — Engine config (model: VoiceDesign, size: 1.7B)
|
|
- `Qwen3TTSVoiceDesignerNode` — Voice designer (description → audio)
|
|
- `voice_description`: Natural language voice description
|
|
- `character_name`: Saves to `models/voices/{name}/`
|
|
- `reference_text`: Text to generate preview with
|
|
|
|
### Multi-Speaker
|
|
- `VibeVoiceEngineNode` — Engine config (model: vibevoice-7B, multi_speaker_mode: enabled)
|
|
- `UnifiedTTSTextNode` — Main TTS node
|
|
- `text`: Script with `[Character]` speaker tags
|
|
- `narrator_voice`: Default speaker
|
|
- `speaker1_voice` through `speaker4_voice`: Character voice references
|
|
- `SaveAudioMP3` — Export final audio
|
|
|
|
## ComfyUI API
|
|
|
|
Workflows can be submitted via the ComfyUI API:
|
|
```bash
|
|
# Submit workflow
|
|
curl -X POST http://10.0.0.113:8188/prompt \
|
|
-H "Content-Type: application/json" \
|
|
-d @workflow.json
|
|
|
|
# Check status
|
|
curl -s http://10.0.0.113:8188/history | python3 -m json.tool
|
|
|
|
# Get outputs
|
|
curl -s http://10.0.0.113:8188/system_stats
|
|
```
|