Models & Hardware

VoiceStudio runs highly optimized speech models locally. By default, it downloads weights from the Hugging Face Hub on first use. This page covers managing local speech engines, adjusting compute and performance configurations, handling disk space and sidecar virtual environments, and deploying headless remote backends.

Engine Management#

VoiceStudio features modular adaptors for text-to-speech (TTS), speech-to-text (ASR), translation, and speaker diarization. You can browse, install, and switch these models directly in the Model Catalogue or via quick-switch keys (Ctrl or Cmd + E).

Text-to-Speech Reference Constraints#

Cloning engines impose duration limits on reference voice clips. If a clip exceeds these limits, the engine trims it or rejects the request:

• VoiceStudio (OmniVoice): Accepts up to 20 seconds. If a longer clip is provided, it extracts the 15-second passage containing the most active speech (best_window). Manual transcripts sent with clips longer than 20 seconds are rejected with a [clone_ref_too_long] error.
• VoxCPM2: Accepts up to 30 seconds, taking the first 30 seconds of speech after trimming edge silence (head).
• Other Engines: Pass the reference clip directly to the engine without pre-trimming.

Diarization and Translation Adaptors#

Speaker diarization is handled locally through two main runtimes:

• pyannote 3.1: A high-accuracy option running on the Hugging Face model repository pyannote/speaker-diarization-3.1.
• Sortformer v1: A native, lightweight model running inside audio.cpp.

For translation in video dubbing, VoiceStudio uses pip-installable providers such as Argos Translate or Amazon Translate. Packaged desktop environments are read-only, but source-based developer installations allow one-click package installation directly in the UI. For Argos, you can download specific source-to-target language packs on demand.

Hugging Face Integration#

VoiceStudio downloads and validates models from Hugging Face. Setting up a Hugging Face token is recommended to bypass rate limits and access gated models.

Token Resolution Cascade#

At startup, the backend resolves the Hugging Face token by checking three sources in priority order:

1. App Database: An encrypted token stored in VoiceStudio's SQLite database, entered via Settings → API Keys. The database encrypts this token with a key derived from your hardware identifier.
2. Environment Variables: The HF_TOKEN (or legacy HUGGING_FACE_HUB_TOKEN) environment variable.
3. Hugging Face CLI: The standard token file generated by running huggingface-cli login (typically at ~/.cache/huggingface/token).

To validate the active token, click Test now in Settings → API Keys. This queries the Hugging Face whoami endpoint, displays your username, and places an Active badge next to the winning credential.

Gated Models#

Several local models require you to accept their license agreements on Hugging Face before downloading. Sign in to your Hugging Face account and accept the terms for:

• pyannote/speaker-diarization-3.1 and pyannote/segmentation-3.0 (Diarization)
• IndexTeam/IndexTTS-2.5 (IndexTTS Engine)
• Supertone/supertonic-3 (Supertonic Engine)
• kyutai/pocket-tts (PocketTTS Engine)

Segmented Downloads and Mirrors#

The app utilizes a custom segmented download accelerator by default. It fetches files in parallel byte-ranges (up to 8 streams at once, capped at 16 MB per range) for faster speeds while maintaining accurate, live progress bars. Set OMNIVOICE_SEGMENTED_DOWNLOAD=0 to revert to single-stream downloading.

For environments with restricted network access, VoiceStudio automatically probes the official site and hf-mirror.com in parallel, selecting the fastest reachable endpoint. You can bypass this check and pin a specific mirror by setting HF_ENDPOINT=https://hf-mirror.com in Settings → Models → Hugging Face mirror.

Hardware Optimization#

VoiceStudio detects and utilizes your GPU automatically. You can monitor and adjust your hardware settings under Settings → Performance & Device.

Device Detection#

• NVIDIA: Runs using CUDA with half-precision (FP16) and torch.compile enabled by default.
• Apple Silicon: Runs on the GPU via MPS and MLX. To prevent unified memory saturation on 16 GB machines, the pipeline automatically serializes tasks and unloads models when transitioning between ASR and TTS workloads.
• Windows: PyTorch GPU acceleration is NVIDIA-only. AMD and Intel GPUs fallback to CPU execution for PyTorch-based engines, though audio.cpp can run on Radeon cards via Vulkan.
• CPU Fallback: System execution falls back to CPU when compatible drivers or libraries are missing.

Pin a specific compute device by setting OMNIVOICE_DEVICE (e.g. cuda, mps, cpu) or select it in the UI.

Memory Management and Offloading#

You can reclaim GPU VRAM or system RAM without restarting the application:

• Flush Caches: Click Flush on the top toolbar to empty PyTorch's memory cache allocator. This resolves fragmentation without unloading your active models.
• Unload Models: Unload specific engines via the toolbar dropdown or the Model Catalogue. The API also exposes these controls:
  BashCopy

  curl -X POST "http://127.0.0.1:3900/system/flush-memory"                   # Flush caches
  curl -X POST "http://127.0.0.1:3900/system/flush-memory?unload_model=true" # Flush and unload TTS

• Move to System RAM: Turn on Move the voice model to system RAM after generation (or set OMNIVOICE_OFFLOAD_AFTER_GENERATION=1). This moves the TTS model off your GPU after a 3-second idle grace period, freeing VRAM for other hardware-intensive apps like local LLMs or games.

Compute Time Watchdogs#

To prevent stalled jobs from locking up system resources, VoiceStudio applies a strict compute budget:

• Accelerated Devices: 300-second base budget (OMNIVOICE_GENERATE_TIMEOUT_S).
• CPU Execution: 600-second base budget (OMNIVOICE_CPU_GENERATE_TIMEOUT_S). This scales dynamically based on text length (adding 4 seconds per character on CPU up to a 2-hour cap) to handle slower processing times.

Disk Space and Sidecar Environments#

VoiceStudio installs most engines directly into its parent Python environment. However, engines with conflicting package requirements (such as IndexTTS 2.5, MOSS-TTS-v1.5, or dots.tts) are installed in isolated sidecar virtual environments.

Reducing Disk Overhead with uv Reflinks#

Since sidecar environments require their own deep learning stacks, they can occupy several gigabytes. VoiceStudio uses uv to manage sidecars, which links identical packages from a central global cache to avoid duplicate disk usage:

• macOS & Linux: Uses copy-on-write reflinks (clone mode).
• Windows: Uses hardlinks.

To ensure maximum deduplication, engines like dots.tts pin the exact PyTorch version utilized by the main application. Deleting a sidecar directory from the app files deletes its unique scripts and weights, while shared dependencies remain in the global cache until you execute a cache prune.

Remote GPU Deployment#

You can host the VoiceStudio backend on a dedicated GPU server or cloud instance while driving the interface from a thin client or a web browser over a local network or VPN.

1. Server Configuration#

Launch the backend headlessly by setting a secure API key and enabling server mode:

export OMNIVOICE_API_KEY="your_secure_random_key"
export OMNIVOICE_SERVER_MODE=1
uv run uvicorn backend.main:app --host 0.0.0.0 --port 3900

Setting OMNIVOICE_SERVER_MODE=1 relaxes the strict loopback-only administrator constraint, allowing the key to secure sensitive system endpoints.

2. Connection and Client Setup#

• Desktop Thin Client: In Settings → Sharing → Remote backend, enter the remote URL and API key. Click Test connection to verify before saving.
• Web Browser: Access the remote instance by appending your key to the URL hash fragment:
  Copy

  https://your-gpu-server-ip:3900/#api_key=your_secure_random_key

  Using the hash fragment (#) ensures the API key remains client-side and is never sent to the server in plain-text HTTP requests or saved in reverse proxy logs.

3. Security and Trusted Networks#

All requests to a keyed backend require authentication (Authorization: Bearer <key>). WebSockets utilize short-lived, single-use path tickets that expire after 30 seconds.

If you deploy behind a trusted local network or reverse proxy, configure the trusted IP ranges:

export OMNIVOICE_TRUSTED_NETWORKS="192.168.1.0/24,10.0.0.0/8"

Clients originating from these subnets are exempted from the consumption gates (such as the share PIN and dictation WebSocket keys). However, admin routes (/system/* and /api/settings/*) remain locked and still require the master API key.