Building Cadence: Architecture of an Open-Weight Generative Voice, Music, and Media Studio

Modern digital content creation is being revolutionized by the rapid evolution of open-source artificial intelligence. Breakthrough open-weight models now match or exceed closed systems across neural speech synthesis, full vocal music composition, audio stem separation, computer vision, and real-time conversational agents. The real power of open source lies in the ability to compose these specialized foundation models into a unified, high-performance platform where every component can be inspected, optimized, and orchestrated with complete architectural freedom.

By building on open weights, developers and media creators gain total autonomy: zero black-box dependencies, customizable inference topologies, and the flexibility to deploy anywhere—from personal GPU workstations via Docker to autoscaling serverless GPU clusters.

To demonstrate what is possible when open-source foundation models are brought together under one roof, I engineered Cadence (GitHub Repository)—a self-hostable, full-spectrum generative audio, music, and media platform powered entirely by open-weight neural models.

In this architectural deep dive, I will unpack how Cadence works under the hood: the open-weights foundation model ecosystem, the decoupled serverless GPU microservice topology, the automated viral video clipper engine, the sub-200ms duplex voice agent, our dual-stream telemetry with Langfuse and Convex, and our zero-code-drift local Docker deployment.

1. Unified Open-Weight Architecture vs. Disconnected Workflows

Before examining the code and deployment topologies, let’s compare how traditional closed-source or isolated media workflows compare to Cadence’s unified open-weight platform:

2. The Open-Weights Foundation Model Ecosystem

Cadence does not rely on opaque third-party generation APIs. Every single studio capability is powered by open-weight deep learning architectures:

┌────────────────────────────────────────────────────────────────────────┐
│                   CADENCE FOUNDATION MODEL TIERS                       │
└────────────────────────────────────────────────────────────────────────┘

 [ VOICE & SPEECH ]
 ├─ Chatterbox Turbo ────► Neural TTS & Zero-Shot Cloning (< 3s reference)
 ├─ Whisper Large-v3 ───► Multilingual STT with word-level microsecond sync
 ├─ OpenVoice v2 ────────► Speech-to-Speech timbre morphing & emotion lock
 └─ Kokoro-82M ──────────► Sub-200ms streaming TTS for conversational agents

 [ MUSIC & FOLEY ]
 ├─ Meta MusicGen Melody ► Instrumental loops, synth beds, beats (up to 120s)
 ├─ ACE-Step 1.5 Turbo ──► Complete vocal songs with singing & lyrics (240s)
 ├─ Meta AudioGen ───────► Cinematic Foley & ambient sound effects (1-30s)
 ├─ Meta Demucs v4 ──────► Neural 4-stem audio splitter (Vocals/Drums/Bass)
 └─ RVC v2 ──────────────► Musical singing voice conversion with pitch transpose

 [ VISUALS & VIDEO ]
 ├─ Qwen-Image-2.1 ──────► 7B DiT + 8B Qwen3-VL: Legible typography & RGBA PNGs
 ├─ SenseNova-U1.5-8B ───► Mixture-of-Trajectories (MoT) photorealism & inpainting
 ├─ SDXL-Turbo ──────────► 1-second single-step latent diffusion cover art
 └─ Wan2.1 1.3B ─────────► Text-to-video cinematic motion visualizers

 [ AGENT & VIDEO INTELLIGENCE ]
 ├─ Silero VAD ──────────► Millisecond voice activity detection & barge-in
 ├─ Qwen2.5-1.5B ────────► Real-time agent LLM & video virality candidate scoring
 └─ Ultralytics YOLO11 ──► Active speaker detection & dynamic 9:16 reframing

By decoupling these specialized models into purpose-built engines, Cadence allows users to mix and match modalities seamlessly. For example, a user can generate a vocal song with ACE-Step 1.5, separate the stems using Demucs v4, convert the vocal lead using RVC v2, enhance the acoustics via Resemble Enhance, generate matching 2048px cover art with Qwen-Image-2.1, and render a motion visualizer with Wan2.1—all in one workspace.

3. Serverless GPU Microservice Architecture

Hosting dozens of specialized PyTorch models on a single monolithic machine leads to severe CUDA memory fragmentation, package dependency conflicts, and expensive idle GPU bills.

Cadence solves this by decomposing serverless compute into dedicated microservice engines deployed on NVIDIA A10G GPUs (24 GB VRAM) via Modal:

Client (Next.js 15 / tRPC)
         │
         ├─── REST / WebSockets
         ▼
┌────────────────────────────────────────────────────────┐
│               CONVEX REACTIVE DATA LAYER               │
│   • Multi-Tenant Organization Isolation (Clerk)        │
│   • Deterministic Credit Deduction Ledger              │
│   • Real-Time Subscriptions (Zero Client Polling)       │
└────────┬───────────────────────────────────────────────┘
         │
         ├─── Dispatches Execution Tasks
         ▼
┌────────────────────────────────────────────────────────┐
│           SERVERLESS GPU MICROSERVICE CLUSTER          │
│                                                        │
│  ┌───────────────────────┐  ┌───────────────────────┐  │
│  │  CadenceAudioEngine   │  │  CadenceSoundEngine   │  │
│  │  • Chatterbox Turbo   │  │  • Meta AudioGen      │  │
│  │  • Whisper Large-v3   │  │  • MusicGen Melody    │  │
│  │  • OpenVoice v2       │  │  • ACE-Step Fallback  │  │
│  └───────────────────────┘  └───────────────────────┘  │
│  ┌───────────────────────┐  ┌───────────────────────┐  │
│  │  CadenceSongEngine    │  │  CadenceViralClipper  │  │
│  │  • ACE-Step 1.5 Turbo │  │  • Whisper Timestamps │  │
│  │  • TorchCodec Buffer  │  │  • YOLO11 Reframing   │  │
│  └───────────────────────┘  │  • NVENC Render Short │  │
│  ┌───────────────────────┐  └───────────────────────┘  │
│  │  CadenceAgentEngine   │  ┌───────────────────────┐  │
│  │  • Silero VAD         │  │  QwenImageEngine      │  │
│  │  • Qwen2.5-1.5B       │  │  • Qwen-Image-2.1 DiT │  │
│  │  • Kokoro-82M TTS     │  │  • Multi-Image Studio │  │
│  └───────────────────────┘  └───────────────────────┘  │
└────────────────────────┬───────────────────────────────┘
                         │
                         ▼
┌────────────────────────────────────────────────────────┐
│           PERSISTENT CLOUDFLARE R2 MOUNT               │
│   • modal.CloudBucketMount mounted at /r2              │
│   • Zero-latency audio stem and video streaming        │
└────────────────────────────────────────────────────────┘

3.1 Scale-to-Zero Economics

Each microservice container is configured with scaledown_window=300 (a 5-minute warm stay-alive window). When generation requests subside, containers shut down automatically, costing $0.00 in idle compute. When traffic spikes, containers spin up in parallel across independent GPU instances without competing for VRAM.

3.2 PyTorch VRAM Fragmentation Defense

Running heterogeneous generative models in production frequently triggers CUDA Out of Memory errors caused by memory fragmentation rather than actual allocation exhaustion. Every Cadence engine container enforces:

import os
import gc
import torch

# Prevent PyTorch memory pool fragmentation
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

def release_gpu_memory():
    """Explicit post-generation memory reclamation."""
    gc.collect()
    if torch.cuda.is_available():
        torch.cuda.empty_cache()
        torch.cuda.ipc_collect()

3.3 Zero-Latency Cloudflare R2 Storage Mounting

Instead of spending hundreds of milliseconds downloading large video files or reference audio samples across HTTP boundaries, Cadence mounts Cloudflare R2 object storage directly into the Linux filesystem via modal.CloudBucketMount:

r2_mount = modal.CloudBucketMount(
    bucket_name="cadence",
    secret=modal.Secret.from_name("cloudflare-r2"),
    mount_path="/r2"
)

Input audio stems, pre-seeded system voices, and rendered video files are read and written directly as local disk files (/r2/voices/...), bypassing network transfer overhead.

4. Engineering the Viral Video Clipper Studio

Short-form vertical video (TikTok, Instagram Reels, YouTube Shorts) represents the highest-engagement format on the web today. However, turning a 60-minute landscape podcast into vertical 9:16 clips usually requires hours of tedious manual scrubbing, framing, and subtitle styling.

Cadence features an automated Viral Video Clipper Studio (/clipper) built to ingest long-form videos via YouTube URL or raw video upload and produce fully rendered vertical shorts in under 90 seconds.

Virality Evaluation Prompting

Rather than guessing where clips start and stop based on silence alone, Cadence feeds windowed transcript segments to Qwen2.5-1.5B with an explicit structured rubric:

{
  "start_time": 142.5,
  "end_time": 204.0,
  "virality_score": 92,
  "headline": "Why Multi-Agent Systems Fail at Scale",
  "reasoning": "Opens with an immediate provocative counter-intuitive hook, provides three clear technical failure modes, and ends on an unresolved insight."
}

The video engine automatically crops the exact video slice, burns the dynamic retention headline banner at the top, and applies subtle video equalization to prevent duplicate content penalties across social platforms.

5. The Real-Time Duplex Conversational Voice Agent

Interactive voice assistants frequently stumble on latency. If a human finishes speaking and has to wait 1.5 seconds for the agent to respond, the conversation feels robotic, broken, and unnatural.

Cadence implements a dedicated full-duplex conversational voice engine (/agent) designed for sub-200ms Time-To-First-Token (TTFT) speech response over WebSockets:

User Voice Input (24kHz PCM Stream)
               │
               ▼
┌──────────────────────────────────────────────┐
│        Silero VAD (Voice Activity)           │
│   • Detects speech onset in < 30ms           │
│   • Dispatches instant BARGE-IN interrupt    │
└──────────────────────┬───────────────────────┘
                       │
                       ▼ Active Utterance Stream
┌──────────────────────────────────────────────┐
│   Streaming ASR (NVIDIA Parakeet / Whisper)  │
│   • Sub-100ms real-time token transcription  │
└──────────────────────┬───────────────────────┘
                       │
                       ▼ Real-time Text Prompt
┌──────────────────────────────────────────────┐
│        Qwen2.5-1.5B Conversational LLM       │
│   • Streaming token generation               │
│   • Configurable persona system instructions │
└──────────────────────┬───────────────────────┘
                       │
                       ▼ Streaming Response Tokens
┌──────────────────────────────────────────────┐
│         Dual-Mode TTS Speech Synthesis       │
│   [Speed Mode]  ► Kokoro-82M (< 200ms TTFT)  │
│   [Clone Mode]  ► Chatterbox Turbo Proxy     │
└──────────────────────┬───────────────────────┘
                       │
                       ▼ Audio Output (24kHz PCM)
            Client Waveform Visualizer

Instant Barge-In Interruption Handling

One of the hardest challenges in duplex voice agents is handling interruptions smoothly. If the agent is speaking and the user begins talking, the agent must immediately stop transmitting, evict pending synthesis queues, and listen.

Cadence achieves this over WebSockets using a binary protocol:

1. When Silero VAD detects user vocal activity exceeding calibrated energy thresholds, the server immediately sends an interrupt packet to the frontend audio buffer.
2. The server cancels active LLM generation workers and empties Kokoro speech generation queues.
3. The client instantaneously silences audio playback, smoothly resetting the visualizer orb.

6. AI Observability & Dual-Stream Telemetry

Enterprise AI deployments cannot operate blind. When scaling generative workflows across dozens of users and multi-tenant organizations, engineering teams need instant visibility into latency spikes, token burns, and execution trace waterfalls.

Cadence implements a Dual-Stream Telemetry Architecture: every generation request across every studio streams trace telemetry simultaneously to two decoupled destinations:

Model Generation Event (TTS / STT / Music / Video / Agent)
                           │
             ┌─────────────┴─────────────┐
             ▼                           ▼
┌───────────────────────────┐ ┌───────────────────────────┐
│      LANGFUSE CLOUD       │ │  NATIVE CONVEX TRACES     │
│  • Distributed spans      │ │  • In-app reactive audits │
│  • Token accounting       │ │  • Sub-millisecond filter │
│  • Latency percentiles    │ │  • Zero-polling waterfalls│
│  • Long-term historicals  │ │  • 7-day retention prune  │
└───────────────────────────┘ └───────────────────────────┘

1. Langfuse Cloud: Captures deep distributed execution spans, model identifiers, input parameters, raw generation payloads, and comprehensive token accounting (prompt, completion, total).
2. Native Convex Trace Engine (llm_traces): Stores indexed execution traces directly within Convex. This powers Cadence’s in-app Observability Dashboard (/observability), allowing administrators to inspect step-by-step trace waterfalls, filter by status (success, error, quota_retry), and export audits to JSON or CSV with sub-millisecond query performance and zero client polling.

7. Zero Code Drift: Cloud Serverless to Local Docker

While deploying on serverless GPUs is ideal for cloud production, many AI developers, researchers, and studio creators require 100% offline local execution on personal hardware.

Cadence was built with a Zero Code Drift Architecture: the local Docker containers copy and execute the exact same *.py engine files used by the cloud serverless endpoints. A lightweight FastAPI runner simply imports each engine class and binds it directly to standard Uvicorn:

cadence_engine.py / cadence_studio_engine.py / cadence_clipper_engine.py
                    │
      ┌─────────────┴─────────────┐
      ▼                           ▼
[ Modal Serverless GPU ]    [ Local Docker Compose ]
• Scaledown to zero         • Combined Monolith (:8000)
• Serverless A10G cluster   • Modular Microservices (:8001-:8009)
• Cloudflare R2 Mount       • NVIDIA CUDA / Apple Silicon MPS

Hardware Compatibility & Topologies:

• NVIDIA Gaming PCs & Workstations (RTX 3060–4090, A10/A100): Full hardware acceleration via CUDA 12.4 and NVIDIA Container Toolkit with NVENC video encoding.
• Apple Silicon MacBooks (M1/M2/M3/M4): Native PyTorch Metal Performance Shaders (MPS) acceleration via our docker-compose.override.macbook.yml.
• Embedded MinIO S3: Automatically boots a local S3-compatible storage bucket (cadence) on port 9000 with web console access, matching Cloudflare R2 behavior with zero configuration.

8. Conclusion & The Future of Open Media Studios

Cadence proves what becomes possible when modern open-weight neural architectures are combined into a unified, production-grade media studio. By orchestrating state-of-the-art models—Chatterbox Turbo, ACE-Step 1.5, MusicGen, Whisper Large-v3, Qwen2.5, and YOLO11—with a decoupled serverless microservice topology and dual-stream observability, creators and developers can operate their own enterprise-grade studio with total autonomy and data sovereignty.

The complete codebase, Docker Compose configurations, and microservice definitions are open-source on GitHub:

• Live Demo: cadence.samuelolubukun.com
• GitHub Repository: github.com/samolubukun/Cadence