feat: initial setup - voices, demo script, README

2026-07-18 13:43:27 +00:00
parent 298e6987bf
commit a865d76c81
6 changed files with 243 additions and 2 deletions
@@ -1,3 +1,120 @@
-# audio-gen-lab
+# Audio Gen Lab
-Multi-speaker audio generation experiments with VibeVoice, Qwen3-TTS voice design, and ComfyUI workflows
+Multi-speaker audio generation experiments using ComfyUI on an RTX 3090 (24GB VRAM).
 ## Goal
 Generate multi-character audio content (podcasts, dialogues) using:
 1. **Qwen3-TTS VoiceDesign** — Design custom character voices via natural language descriptions
 2. **VibeVoice 7B** — Generate multi-speaker audio (up to 4 characters) from designed voices
 ## Pipeline
 ```
 Qwen3-TTS VoiceDesignerNode → generates reference voice audio → saved to voices/
 VibeVoiceEngineNode → loads reference voices as speaker references
 UnifiedTTSTextNode → feeds script with [Character] tags → multi-speaker output
 SaveAudioMP3 → final podcast audio
 ```
 ## Server
 - **ComfyUI:** http://10.0.0.113:8188
 - **GPU:** NVIDIA RTX 3090 (25.3GB VRAM)
 - **TTS Audio Suite:** Installed (11 engines)
 ## Installed Models
 ### VibeVoice
 - `vibevoice-7B` ✅ — Fully downloaded (~18GB), ready to use
 - `vibevoice-1.5B` ⚠️ — Partially downloaded (incomplete files)
 ### Qwen3-TTS
 - `Qwen3-TTS-12Hz-1.7B-Base` ❌ — Needs download (voice cloning)
 - `Qwen3-TTS-12Hz-1.7B-CustomVoice` ❌ — Needs download (preset voices)
 - `Qwen3-TTS-12Hz-1.7B-VoiceDesign` ❌ — Needs download (voice design via description)
 To download, put models in: `ComfyUI/models/TTS/Qwen3-TTS/`
 Source: https://huggingface.co/collections/Qwen/qwen3-tts
 ## Available TTS Engines (TTS Audio Suite)
 | Engine | Multi-Speaker | VRAM | Use Case |
 |--------|--------------|------|----------|
 | VibeVoice 7B | ✅ Up to 4 | ~18GB | Multi-character dialogue |
 | VibeVoice 1.5B | ✅ Up to 4 | ~7GB | Faster, lower quality |
 | F5-TTS | ❌ | ~4GB | Voice cloning, fast |
 | Higgs Audio 2/3 | ❌ | ~8GB | Expressive single speaker |
 | Qwen3-TTS | ❌ | ~12GB | Multilingual, voice design |
 | IndexTTS-2 | ❌ | ~4GB | Emotion control |
 | Echo-TTS | ❌ | ~6GB | Voice cloning |
 | CosyVoice 3 | ❌ | ~6GB | 23 languages |
 | ChatterBox | ❌ | ~3GB | Lightweight, reliable |
 | Step Audio EditX | ❌ | ~6GB | Audio editing |
 | RVC | ❌ | ~2GB | Pitch-based voice swap |
 ## Workflow
 ### Phase 1: Voice Design (Qwen3-TTS)
 1. Create `Qwen3TTSEngineNode` with model `VoiceDesign`, size `1.7B`
 2. Create `Qwen3TTSVoiceDesignerNode`
 3. Write voice descriptions (see `voices/` directory)
 4. Generate reference audio for each character
 5. Save to `models/voices/{character_name}/`
 ### Phase 2: Multi-Speaker Generation (VibeVoice)
 1. Create `VibeVoiceEngineNode` with model `vibevoice-7B`
 2. Create `UnifiedTTSTextNode`
 3. Load script from `scripts/` with `[Character]` tags
 4. Feed character voice references as speakers
 5. Generate and save output
 ### Phase 3: Test & Iterate
 - Test with the demo podcast script
 - Refine voice designs
 - Optimise for quality vs speed
 ## Scripts
 - `scripts/demo-podcast.md` — Demo podcast script (4 characters, ~5 minutes)
 ## Voices
 Character voice definitions live here:
 - `voices/alice.md` — Host, warm British female
 - `voices/bob.md` — Co-host, energetic American male
 - `voices/guest1.md` — Expert guest, calm academic
 - `voices/guest2.md` — Second guest, playful personality
 ## ComfyUI Node References
 ### Voice Design
 - `Qwen3TTSEngineNode` — Engine config (model: VoiceDesign, size: 1.7B)
 - `Qwen3TTSVoiceDesignerNode` — Voice designer (description → audio)
  - `voice_description`: Natural language voice description
  - `character_name`: Saves to `models/voices/{name}/`
  - `reference_text`: Text to generate preview with
 ### Multi-Speaker
 - `VibeVoiceEngineNode` — Engine config (model: vibevoice-7B, multi_speaker_mode: enabled)
 - `UnifiedTTSTextNode` — Main TTS node
  - `text`: Script with `[Character]` speaker tags
  - `narrator_voice`: Default speaker
  - `speaker1_voice` through `speaker4_voice`: Character voice references
 - `SaveAudioMP3` — Export final audio
 ## ComfyUI API
 Workflows can be submitted via the ComfyUI API:
 ```bash
 # Submit workflow
 curl -X POST http://10.0.0.113:8188/prompt \
  -H "Content-Type: application/json" \
  -d @workflow.json
 # Check status
 curl -s http://10.0.0.113:8188/history | python3 -m json.tool
 # Get outputs
 curl -s http://10.0.0.113:8188/system_stats
 ```
@@ -0,0 +1,72 @@
 # Demo Podcast: AI and Creativity
 **Podcast:** Tech Talk Thursday
 **Episode:** AI and Creative Expression
 **Duration:** ~5 minutes
 **Characters:** Alice (Host), Bob (Co-Host), Dr. Sarah Chen (Expert), Jamie (Creative)
 ---
 [Alice] Welcome back to Tech Talk Thursday, the podcast where we explore the fascinating intersection of technology and everyday life. I'm Alice, and today we have a really special episode lined up for you.
 [Bob] And I'm Bob, and Alice, I've got to say, I'm absolutely buzzing about this one. We're talking about something that's literally reshaping how we think about creativity itself — artificial intelligence as a creative tool.
 [Alice] That's right. And to help us unpack this, we've brought along two fantastic guests. First, Dr. Sarah Chen, Professor of Computer Science at MIT, whose research into creative AI has been groundbreaking. Sarah, welcome to the show.
 [Sarah] Thank you so much for having me. It's a pleasure to be here. And yes, the last eighteen months in particular have seen some truly remarkable developments in this space.
 [Alice] And joining us is Jamie, digital artist and content creator who's been working with AI tools in their creative practice for over two years now. Jamie, great to have you with us.
 [Jamie] Oh, thanks Alice! I'm really excited to be here and chat about this. It's such a rich topic, and honestly, I feel like every week there's something new and unexpected happening.
 [Bob] So let's dive right in. The question that everyone's asking — and I know I've been asking this to myself constantly — is: is AI actually creative, or is it just really good at mimicking? What do you think, Sarah?
 [Sarah] Well, that's the million dollar question, isn't it? And I think the answer depends entirely on how we define creativity. If creativity means the ability to produce novel and valuable outputs, then in a purely functional sense, yes, AI systems are becoming increasingly creative. But if creativity requires intention, emotional experience, and subjective understanding, then we're in much murkier territory.
 [Jamie] I mean, from my perspective as someone who actually uses these tools daily, I think we're asking the wrong question. It's not about whether the AI is creative — it's about how these tools expand what I, as a human creator, can do. When I work with generative AI, it's more like having a really unusual collaborator. Sometimes it gives me ideas I never would have thought of.
 [Bob] That's fascinating. So it's more like... a creative amplifier rather than a replacement?
 [Jamie] Exactly. I've had sessions where I'll generate hundreds of variations of something, and most of them are... well, mediocre. But occasionally, something comes out that's so unexpected, so completely outside my normal way of thinking, that it sends me down this completely new creative direction I never would have explored on my own.
 [Alice] Now, Sarah, there's been a lot of debate about the training data that these models use. Many are trained on massive datasets of human-created art, music, and writing. Where does that leave us ethically?
 [Sarah] This is perhaps the most important open question in our field. The current paradigm involves training models on vast corpora of human-created content, often without explicit permission or attribution from the original creators. From a technical standpoint, this is what enables these systems to learn patterns and generate novel outputs. But from an ethical and legal standpoint, we are still figuring out the implications.
 [Bob] And there are lawsuits happening right now, aren't there?
 [Sarah] Yes, multiple. The legal landscape is evolving rapidly. Some courts have argued that training on publicly available data constitutes fair use, while others have found against the companies. I think what we'll eventually see is a hybrid model — some forms of training will require licensing and compensation, while others may continue under fair use frameworks.
 [Jamie] You know, as an artist, my instinct was initially to be defensive about this. Like, oh no, here comes the technology that's going to steal my job. But the more I've worked with it, the more I've realised that the human element — the taste, the intention, the emotional resonance — that's still what makes art matter. AI can generate an image, but it can't decide why that image matters or what it's trying to communicate.
 [Alice] I love that point. It reminds me of how photography was once argued to not be art because a machine was doing the capturing. Now we think of photography as one of the most legitimate art forms.
 [Bob] True, true. But here's the thing that still bugs me a bit. When I listen to AI-generated music, and it sounds genuinely moving, part of me feels like I'm being tricked. Like, am I really connecting emotionally with something that has no emotions?
 [Sarah] That's a profound philosophical question, Bob. And I'll give you a thought experiment. When you read a novel, the characters don't exist. The emotions they express are constructed from language. Yet you still feel moved. The AI-generated content exists in a similar space — it's a construction that can evoke genuine human response, even if the creator itself has no subjective experience.
 [Jamie] And honestly? The best AI art I've seen, the stuff that actually stops me in my tracks, always has a strong human hand behind it. The AI is a tool, but the vision, the direction, the curation — that's all human. I think we'll start seeing this new wave of hybrid art forms that are genuinely their own thing.
 [Alice] Before we wrap up, let's do a quick lightning round. Bob, what's your take — will AI replace creative jobs in the next decade?
 [Bob] I don't think it'll replace jobs so much as it'll transform them. I think creative professionals who embrace these tools will be incredibly powerful, and those who don't... well, they might find themselves at a disadvantage. It's not about replacement, it's about evolution.
 [Alice] Sarah?
 [Sarah] I think we'll see a significant restructuring of creative industries, with many routine tasks automated, but genuine creative direction, curation, and emotional connection remaining firmly human domains. The net effect, I believe, will be positive — more people able to express themselves creatively than ever before.
 [Alice] And Jamie?
 [Jamie] I think we're going to see an explosion of creativity in the next few years, like nothing we've ever seen. Tools that used to require years of technical skill to master are becoming accessible to anyone with an idea. That's genuinely exciting.
 [Bob] Brilliant answers all round. Sarah, Jamie — thank you so much for joining us. This has been an absolutely cracking conversation.
 [Sarah] Thank you both. It's been a pleasure.
 [Jamie] Yeah, thanks so much! This was really fun.
 [Alice] And thanks to you, our listeners, for tuning in. Join us next week when we're exploring the future of virtual reality in education. Until then, keep curious, keep creating, and keep questioning.
 [Bob] This has been Tech Talk Thursday. I'm Bob.
 [Alice] I'm Alice. See you next time.
@@ -0,0 +1,13 @@
 # Alice - Host
 **Voice Design Description:**
 Professional British female narrator, warm and conversational tone, mid-30s, articulate with a slight Received Pronunciation accent. Engaging podcast host voice — clear, authoritative but approachable. Moderate pacing, natural enthusiasm when introducing topics.
 **Role:** Main podcast host
 **Gender:** Female
 **Age:** Mid-30s
 **Nationality:** British
 **Style:** Warm, professional, conversational
 **Reference Text for Voice Generation:**
 Welcome back to another episode of Tech Talk Thursday. Today we're diving into something that's been on my mind for weeks — the intersection of artificial intelligence and creative expression. It's a fascinating topic, and I've brought along some wonderful guests to help us explore it. Let's get started.
@@ -0,0 +1,13 @@
 # Bob - Co-Host
 **Voice Design Description:**
 Energetic American male, late 20s, upbeat and slightly informal. Enthusiastic podcast co-host with a friendly, approachable tone. Speaks with genuine excitement about technology and pop culture. Casual New York accent, fast-paced but clear, occasionally uses filler words naturally.
 **Role:** Co-host, tech enthusiast
 **Gender:** Male
 **Age:** Late 20s
 **Nationality:** American (New York)
 **Style:** Energetic, enthusiastic, casual
 **Reference Text for Voice Generation:**
 Oh man, I've been looking forward to this episode for ages. Like, I literally couldn't sleep last night thinking about how wild this stuff is getting. Every time I think AI can't get any more impressive, something completely unexpected happens and I'm just — blown away. Let's go!
@@ -0,0 +1,13 @@
 # Dr. Sarah Chen - Expert Guest
 **Voice Design Description:**
 Calm, measured female academic voice, early 50s, Chinese-American heritage. Distinguished professor of computer science. Speaks with authority and precision, moderate pace, slightly formal but warm. Clear articulation with subtle East Asian American accent. Thoughtful pauses between complex ideas.
 **Role:** Expert guest, AI researcher
 **Gender:** Female
 **Age:** Early 50s
 **Nationality:** Chinese-American
 **Style:** Academic, measured, authoritative, warm
 **Reference Text for Voice Generation:**
 Well, thank you both for having me. You know, when we talk about creative AI, I think it's important to distinguish between the technical capabilities and the philosophical implications. From a research perspective, we've seen remarkable progress in just the last eighteen months, but there are still fundamental questions about what creativity really means in a computational context.
@@ -0,0 +1,13 @@
 # Jamie - Creative Guest
 **Voice Design Description:**
 Playful, expressive non-binary voice, mid-20s, British. Digital artist and content creator with an infectious enthusiasm for technology. Speaks with genuine wonder and creativity, animated and expressive, slightly informal London accent. Uses expressive hand gestures implied through vocal energy.
 **Role:** Creative guest, digital artist
 **Gender:** Non-binary
 **Age:** Mid-20s
 **Nationality:** British (London)
 **Style:** Playful, expressive, creative, animated
 **Reference Text for Voice Generation:**
 Oh wow, this is such a brilliant question. Like, as someone who's been working with AI tools in my art practice for about two years now, I can honestly say it's completely transformed how I think about the creative process. It's not about replacing human creativity — it's about expanding what's possible. And that's genuinely exciting!