Audio Gen Lab

Multi-speaker audio generation experiments using ComfyUI on an RTX 3090 (24GB VRAM).

Goal

Generate multi-character audio content (podcasts, dialogues) using:

1. Qwen3-TTS VoiceDesign — Design custom character voices via natural language descriptions
2. VibeVoice 7B — Generate multi-speaker audio (up to 4 characters) from designed voices

Pipeline

Qwen3-TTS VoiceDesignerNode → generates reference voice audio → saved to voices/
VibeVoiceEngineNode → loads reference voices as speaker references
UnifiedTTSTextNode → feeds script with [Character] tags → multi-speaker output
SaveAudioMP3 → final podcast audio

Server

• ComfyUI: http://10.0.0.113:8188
• GPU: NVIDIA RTX 3090 (25.3GB VRAM)
• TTS Audio Suite: Installed (11 engines)

Installed Models

VibeVoice

• vibevoice-7B ✅ — Fully downloaded (~18GB), ready to use
• vibevoice-1.5B ⚠️ — Partially downloaded (incomplete files)

Qwen3-TTS

• Qwen3-TTS-12Hz-1.7B-Base ❌ — Needs download (3.9GB, voice cloning)
• Qwen3-TTS-12Hz-1.7B-CustomVoice ❌ — Needs download (3.8GB, preset voices)
• Qwen3-TTS-12Hz-1.7B-VoiceDesign ❌ — CRITICAL — Needs download (3.8GB, voice design via description)

To download on the ComfyUI server:

# SSH into ComfyUI server or run via ComfyUI Manager
# Models go in: models/TTS/Qwen3-TTS/

# VoiceDesign model (for generating character voices from descriptions)
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
  --local-dir /path/to/ComfyUI/models/TTS/Qwen3-TTS/Qwen3-TTS-12Hz-1.7B-VoiceDesign

# Base model (for voice cloning from reference audio)
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --local-dir /path/to/ComfyUI/models/TTS/Qwen3-TTS/Qwen3-TTS-12Hz-1.7B-Base

# CustomVoice model (for preset voices with instruction control)
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
  --local-dir /path/to/ComfyUI/models/TTS/Qwen3-TTS/Qwen3-TTS-12Hz-1.7B-CustomVoice

Source: https://huggingface.co/collections/Qwen/qwen3-tts

Note: The ComfyUI-Qwen3-TTS node (by DarioFT) expects models in the TTS folder. The VoiceDesign model is essential for our pipeline — it converts natural language voice descriptions into reference audio.

Available TTS Engines (TTS Audio Suite)

Engine	Multi-Speaker	VRAM	Use Case
VibeVoice 7B	✅ Up to 4	~18GB	Multi-character dialogue
VibeVoice 1.5B	✅ Up to 4	~7GB	Faster, lower quality
F5-TTS	❌	~4GB	Voice cloning, fast
Higgs Audio 2/3	❌	~8GB	Expressive single speaker
Qwen3-TTS	❌	~12GB	Multilingual, voice design
IndexTTS-2	❌	~4GB	Emotion control
Echo-TTS	❌	~6GB	Voice cloning
CosyVoice 3	❌	~6GB	23 languages
ChatterBox	❌	~3GB	Lightweight, reliable
Step Audio EditX	❌	~6GB	Audio editing
RVC	❌	~2GB	Pitch-based voice swap

Workflow

Phase 1: Voice Design (Qwen3-TTS)

1. Create Qwen3TTSEngineNode with model VoiceDesign, size 1.7B
2. Create Qwen3TTSVoiceDesignerNode
3. Write voice descriptions (see voices/ directory)
4. Generate reference audio for each character
5. Save to models/voices/{character_name}/

Phase 2: Multi-Speaker Generation (VibeVoice)

1. Create VibeVoiceEngineNode with model vibevoice-7B
2. Create UnifiedTTSTextNode
3. Load script from scripts/ with [Character] tags
4. Feed character voice references as speakers
5. Generate and save output

Phase 3: Test & Iterate

• Test with the demo podcast script
• Refine voice designs
• Optimise for quality vs speed

Scripts

• scripts/demo-podcast.md — Demo podcast script (4 characters, ~5 minutes)

Voices

Character voice definitions live here:

• voices/alice.md — Host, warm British female
• voices/bob.md — Co-host, energetic American male
• voices/guest1.md — Expert guest, calm academic
• voices/guest2.md — Second guest, playful personality

ComfyUI Node References

Voice Design

• Qwen3TTSEngineNode — Engine config (model: VoiceDesign, size: 1.7B)
• Qwen3TTSVoiceDesignerNode — Voice designer (description → audio)
  • voice_description: Natural language voice description
  • character_name: Saves to models/voices/{name}/
  • reference_text: Text to generate preview with

Multi-Speaker

• VibeVoiceEngineNode — Engine config (model: vibevoice-7B, multi_speaker_mode: enabled)
• UnifiedTTSTextNode — Main TTS node
  • text: Script with [Character] speaker tags
  • narrator_voice: Default speaker
  • speaker1_voice through speaker4_voice: Character voice references
• SaveAudioMP3 — Export final audio

ComfyUI API

Workflows can be submitted via the ComfyUI API:

# Submit workflow
curl -X POST http://10.0.0.113:8188/prompt \
  -H "Content-Type: application/json" \
  -d @workflow.json

# Check status
curl -s http://10.0.0.113:8188/history | python3 -m json.tool

# Get outputs
curl -s http://10.0.0.113:8188/system_stats