Video Dubbing

Video Dubbing in VoiceStudio allows you to translate and dub video or audio files while preserving the original speaker's voice. The system coordinates transcription, automatic translation, speaker voice cloning, and precise alignment of dubbed speech with the original video timeline.

Upload and Ingestion#

The dubbing pipeline starts with importing your media source. You can upload local video/audio files or paste a direct URL.

Media Sources#

• Local Files: VoiceStudio accepts standard audio and video container formats. The ingestion process verifies that the data drive has enough storage space to accommodate a local copy. It refuses playlists or manifest files renamed as video files.
• URL Imports: Paste an http:// or https:// link to ingest video or audio through the integrated yt-dlp pipeline.
  Live streams and upcoming premieres cannot be imported until the recording is finished and available. By default, local network addresses or loopback redirects are refused. To allow imports from a media server on your local network, set the environment variable:

  BashCopy

  OMNIVOICE_ALLOW_PRIVATE_URL_IMPORTS=1

• Sign-In Cookies: For authenticated websites, you can supply an optional cookies.txt file (limited to 1 MB) under the URL sign-in options. This cookie selection clears immediately upon ingest and is never persisted.

Hints and Captions#

• Diarization and Hints: You can guide transcription by supplying spoken-language hints and specifying expected speaker counts (ranging from 1 to 20).
• Subtitle & Caption Ingest: When importing a URL, you can automatically download available captions. VoiceStudio selects the closest source-language track, normalizes its timing, and maps the cues to the segment editor. If caption tracks are missing, the system falls back to the normal Automatic Speech Recognition (ASR) path.
• Importing Subtitles Manually: You can drop or choose an .srt or .vtt file to replace current segment text and timings. VoiceStudio retains speaker voice references where the timing overlaps with the new cues. Malformed, overlapping, or duration-clamped cues are flagged in the sidebar.

Translation Engines#

Translation is pluggable. Select your preferred engine in the Dub tab's Engine dropdown.

uv pip install deep_translator openai

Pinning the LLM Provider#

To bypass the in-app selector on shared or headless machines, pin your preferred active LLM provider using the LLM_DEFAULT_PROVIDER environment variable:

LLM_DEFAULT_PROVIDER=groq # Supported: openai, openrouter, orcarouter, cheaperinference, cerebras, iflytek, ollama, custom, etc.

Paste Translation#

If you have already translated your transcript using an external service or human translator, click Paste Translation above the segment table. The system auto-detects three input shapes:

• Timestamped: Maps to existing segments based on .srt or .vtt time overlaps.
• Numbered lines (e.g., 1. ... or [2] ...): Maps by line number.
• Plain lines: Maps sequentially by line position. Blank lines serve as separators.

Applying a pasted translation only writes to the currently selected language tab and marks the modified lines for selective regeneration.

Translation Quality and LLM Settings#

Translation quality is determined by three operational modes:

• Fast: Direct, one-shot translation from the selected engine without timing awareness.
• Cinematic: Uses an LLM to refine the literal translation into natural, in-context phrasing.
• Autofit: Cinematic translation combined with a strict fit-to-time pass. The LLM rewrites long lines so their spoken duration fits within the segment's allocated timeframe.

Cinematic and Autofit modes require a configured LLM provider under Installation & Setup.

Two-Stage LLM Quality#

When using the LLM (OpenAI-compatible) engine, you can activate two extra quality steps:

1. Auto Glossary: Runs a single LLM pass over the entire transcript to extract a theme summary and terminology map. This context is injected into every segment's translation prompt to ensure character names and domain terms remain consistent. Manual glossary entries always override auto-extracted terms.
2. Reflect Pass: Executes a critique-and-rewrite cycle for each segment to convert stiff direct translations into natural spoken dialogue. This uses three LLM calls per segment instead of one.

Custom Translation Style#

If you use Translate with agent, you can fill in a Translation style prompt (up to 5,000 characters). This allows you to describe the desired tone, target audience, formality, or idiomatic handling. The style prompt persists in your project settings and applies to all target languages.

Regional Dialects#

Specify BCP-47 dialect codes (such as es-MX for Mexican Spanish, es-AR for Rioplatense Spanish, or pt-BR for Brazilian Portuguese) to inject localized vocabulary and grammatical instructions into LLM prompts.

The Editing Workspace#

The editing interface provides full control over the timing, translation, and voices of individual segments.

+--------------------------------------------------------------+
|                         Video Preview                        |
+--------------------------------------------------------------+
| [ Timeline Segment Block ]   [ Drag / Nudge / Resize ]       |
+--------------------------------------------------------------+
| Row 1: Source Text     | Editable Translation | [Voice Chip] |
| Row 2: Source Text     | Editable Translation | [Voice Chip] |
+--------------------------------------------------------------+

Proportional Timing Lane#

• Drag segment blocks horizontally to adjust their start and end points.
• Grab edge handles to resize blocks.
• Use keyboard shortcuts to nudge or delete selected blocks.
• Overlapping segments are highlighted on the timeline with visual warning badges.

Segment Row Editing#

• Split & Merge: Split segments at your cursor or merge a segment with its neighbor. VoiceStudio preserves speaker, voice, direction, and gain attributes during these changes.
• Segment Controls: Expand the speaker header to adjust individual segment volume gain (0 to 2), set custom speed rates, or add unique direction notes.
• Bulk Operations: Check multiple segments to perform bulk voice assignments, language resets, or deletions.
• Undo/Redo: The editor retains up to 50 edit states in its history. This history resets when you load a new source file, import subtitles, or switch projects.

Advanced Speech Controls#

Mirror Source Delivery (Prosody Mirror)#

Select More → Mirror source delivery to automatically generate voice directions matching the original actor's speech.

VoiceStudio measures pitch, loudness, syllable rate, and voicing on the separated vocals track and compares each segment against the speaker's baseline. It populates empty direction fields with corresponding descriptive tags:

This analysis runs locally using NumPy and takes approximately 10 seconds per hour of dialogue.

Sound Preservation#

Background-preserving previews and exports keep original stereo elements (music, audience reactions, ambience) intact outside dialogue intervals. Inside dialogue segments, the system mixes dubbed speech over the separated background audio with a 10 ms transition crossfade.

Create Story#

When your dubbed project contains multiple speakers, click Create Story to carry all segments and cast voices directly into the Audiobooks & Stories editor. Speakers are mapped to corresponding Story characters, and merged rows are split back to their assigned speakers.

Voice Generation and Timing Options#

During generation, choose from four primary timing and alignment strategies:

• Concise: Fits the translation naturally without cutting off words or overstretching audio.
• Smart Fit: Dynamically retimes and resynthesizes segments. You can adjust speed limits using the Fastest speed (audio_rate_cap, 1.0 to 2.0, default 1.5) and Slowest speed (min_audio_rate, 0.5 to 1.0, default 0.85) sliders.
• Stretch Video: Stretches or slows the video frames to match the duration of the translated audio.
• Lip Sync: Coordinates facial/lip movements with the generated audio track.

Fit Prediction Badges#

Before starting generation, VoiceStudio evaluates each translated line and displays fit prediction badges:

• Tight fit: The line can be spoken in time but requires audible speed-up.
• Won't fit +Ns: The translation is too long and will be trimmed. Check Suggest shorter lines to prompt the LLM for shorter, meaning-preserving rewrites.

Exporting Dubbed Video#

Export controls are located in the sidebar panel. You can configure options individually per language.

Audio and Track Settings#

• Default Audio Track: Select which language track plays by default in standard media players. Choose Original if the source audio should play first.
• Background Mix: Enable or disable the preserved background music and ambient sounds.
• Audio Format: Export audio-only stems as MP3 (with bitrate choices) or WAV format. You can also package individual language stems as a ZIP file.

Subtitle Burn-In (Hardsub)#

To burn subtitles directly into the video frames, enable Burn subtitles into picture (hardsub). Two layout styles are supported:

• Line: Standard, static bottom-of-screen captions.
• Karaoke (word highlight): Highlight colors sweep across words in sequence. This uses word timings recorded during transcription.
  Karaoke burn-in is disabled when using the dual-layout (original + translation) subtitle display. Stretch Video mode also disables subtitle burn-in; use sidecar exports instead.

You can download the karaoke styling script separately as an .ass sidecar file, or export standard .srt and .vtt subtitle files.

Troubleshooting#

Unreadable Source File#

If the ingestion process reports that a source file is unreadable, verify that the file plays correctly in standard players. This error occurs if a download was interrupted (resulting in a file padded with zeros) or if the storage drive has run out of space. Check your directory in Settings → Storage.

Optional Translation Package Missing#

If you receive an error stating that deep_translator or openai is missing:

1. Source/Development Install: Click the highlighted Install button next to the Engine label or manually install the package using uv pip install. Restart the backend to load the package.
2. Packaged Build: Use the built-in, offline Argos or NLLB-200 engines which require no external dependencies.

Argos CTranslate2 Runtime Errors#

On Linux kernels that restrict executable stacks, the Argos CTranslate2 runtime may fail to load. The API will return argos_runtime_unavailable. Resolve this by upgrading your environment to Python 3.12+ or clearing the executable stack flag:

patchelf --clear-execstack <path_to_ctranslate2_library>

Alternatively, switch to NLLB-200 as your offline translation engine.