audio-gen-lab/README.md

# Audio Gen Lab

Multi-speaker audio generation experiments using ComfyUI on an RTX 3090 (24GB VRAM).

## Goal

Generate multi-character audio content (podcasts, dialogues) using:
1. **Qwen3-TTS VoiceDesign** — Design custom character voices via natural language descriptions
2. **VibeVoice 7B** — Generate multi-speaker audio (up to 4 characters) from designed voices

## Pipeline

```
Qwen3-TTS VoiceDesignerNode → generates reference voice audio → saved to voices/
VibeVoiceEngineNode → loads reference voices as speaker references
UnifiedTTSTextNode → feeds script with [Character] tags → multi-speaker output
SaveAudioMP3 → final podcast audio
```

## Server

- **ComfyUI:** http://10.0.0.113:8188
- **GPU:** NVIDIA RTX 3090 (25.3GB VRAM)
- **TTS Audio Suite:** Installed (11 engines)

## Installed Models

### VibeVoice
- `vibevoice-7B` ✅ — Fully downloaded (~18GB), ready to use
- `vibevoice-1.5B` ⚠️ — Partially downloaded (incomplete files)

### Qwen3-TTS
- `Qwen3-TTS-12Hz-1.7B-Base` ❌ — Needs download (3.9GB, voice cloning)
- `Qwen3-TTS-12Hz-1.7B-CustomVoice` ❌ — Needs download (3.8GB, preset voices)
- `Qwen3-TTS-12Hz-1.7B-VoiceDesign` ❌ — **CRITICAL** — Needs download (3.8GB, voice design via description)

**To download on the ComfyUI server:**
```bash
# SSH into ComfyUI server or run via ComfyUI Manager
# Models go in: models/TTS/Qwen3-TTS/

# VoiceDesign model (for generating character voices from descriptions)
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign \
  --local-dir /path/to/ComfyUI/models/TTS/Qwen3-TTS/Qwen3-TTS-12Hz-1.7B-VoiceDesign

# Base model (for voice cloning from reference audio)
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-Base \
  --local-dir /path/to/ComfyUI/models/TTS/Qwen3-TTS/Qwen3-TTS-12Hz-1.7B-Base

# CustomVoice model (for preset voices with instruction control)
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
  --local-dir /path/to/ComfyUI/models/TTS/Qwen3-TTS/Qwen3-TTS-12Hz-1.7B-CustomVoice
```

Source: https://huggingface.co/collections/Qwen/qwen3-tts

**Note:** The ComfyUI-Qwen3-TTS node (by DarioFT) expects models in the TTS folder. The VoiceDesign model is essential for our pipeline — it converts natural language voice descriptions into reference audio.

## Available TTS Engines (TTS Audio Suite)

| Engine | Multi-Speaker | VRAM | Use Case |
|--------|--------------|------|----------|
| VibeVoice 7B | ✅ Up to 4 | ~18GB | Multi-character dialogue |
| VibeVoice 1.5B | ✅ Up to 4 | ~7GB | Faster, lower quality |
| F5-TTS | ❌ | ~4GB | Voice cloning, fast |
| Higgs Audio 2/3 | ❌ | ~8GB | Expressive single speaker |
| Qwen3-TTS | ❌ | ~12GB | Multilingual, voice design |
| IndexTTS-2 | ❌ | ~4GB | Emotion control |
| Echo-TTS | ❌ | ~6GB | Voice cloning |
| CosyVoice 3 | ❌ | ~6GB | 23 languages |
| ChatterBox | ❌ | ~3GB | Lightweight, reliable |
| Step Audio EditX | ❌ | ~6GB | Audio editing |
| RVC | ❌ | ~2GB | Pitch-based voice swap |

## Workflow

### Phase 1: Voice Design (Qwen3-TTS)
1. Create `Qwen3TTSEngineNode` with model `VoiceDesign`, size `1.7B`
2. Create `Qwen3TTSVoiceDesignerNode`
3. Write voice descriptions (see `voices/` directory)
4. Generate reference audio for each character
5. Save to `models/voices/{character_name}/`

### Phase 2: Multi-Speaker Generation (VibeVoice)
1. Create `VibeVoiceEngineNode` with model `vibevoice-7B`
2. Create `UnifiedTTSTextNode`
3. Load script from `scripts/` with `[Character]` tags
4. Feed character voice references as speakers
5. Generate and save output

### Phase 3: Test & Iterate
- Test with the demo podcast script
- Refine voice designs
- Optimise for quality vs speed

## Scripts

- `scripts/demo-podcast.md` — Demo podcast script (4 characters, ~5 minutes)

## Voices

Character voice definitions live here:
- `voices/alice.md` — Host, warm British female
- `voices/bob.md` — Co-host, energetic American male
- `voices/guest1.md` — Expert guest, calm academic
- `voices/guest2.md` — Second guest, playful personality

## ComfyUI Node References

### Voice Design
- `Qwen3TTSEngineNode` — Engine config (model: VoiceDesign, size: 1.7B)
- `Qwen3TTSVoiceDesignerNode` — Voice designer (description → audio)
  - `voice_description`: Natural language voice description
  - `character_name`: Saves to `models/voices/{name}/`
  - `reference_text`: Text to generate preview with

### Multi-Speaker
- `VibeVoiceEngineNode` — Engine config (model: vibevoice-7B, multi_speaker_mode: enabled)
- `UnifiedTTSTextNode` — Main TTS node
  - `text`: Script with `[Character]` speaker tags
  - `narrator_voice`: Default speaker
  - `speaker1_voice` through `speaker4_voice`: Character voice references
- `SaveAudioMP3` — Export final audio

## ComfyUI API

Workflows can be submitted via the ComfyUI API:
```bash
# Submit workflow
curl -X POST http://10.0.0.113:8188/prompt \
  -H "Content-Type: application/json" \
  -d @workflow.json

# Check status
curl -s http://10.0.0.113:8188/history | python3 -m json.tool

# Get outputs
curl -s http://10.0.0.113:8188/system_stats
```