# Video Dubbing

Translate and dub video files with timed speech while preserving the original speaker's voice.

Video Dubbing in VoiceStudio allows you to translate and dub video or audio files while preserving the original speaker's voice. The system coordinates transcription, automatic translation, speaker voice cloning, and precise alignment of dubbed speech with the original video timeline.

## Upload and Ingestion

The dubbing pipeline starts with importing your media source. You can upload local video/audio files or paste a direct URL.

### Media Sources
* **Local Files**: VoiceStudio accepts standard audio and video container formats. The ingestion process verifies that the data drive has enough storage space to accommodate a local copy. It refuses playlists or manifest files renamed as video files.
* **URL Imports**: Paste an `http://` or `https://` link to ingest video or audio through the integrated `yt-dlp` pipeline. 
  :::warning
  Live streams and upcoming premieres cannot be imported until the recording is finished and available. By default, local network addresses or loopback redirects are refused. To allow imports from a media server on your local network, set the environment variable:
  ```bash
  OMNIVOICE_ALLOW_PRIVATE_URL_IMPORTS=1
  ```
  :::
* **Sign-In Cookies**: For authenticated websites, you can supply an optional `cookies.txt` file (limited to 1 MB) under the URL sign-in options. This cookie selection clears immediately upon ingest and is never persisted.

### Hints and Captions
* **Diarization and Hints**: You can guide transcription by supplying spoken-language hints and specifying expected speaker counts (ranging from 1 to 20).
* **Subtitle & Caption Ingest**: When importing a URL, you can automatically download available captions. VoiceStudio selects the closest source-language track, normalizes its timing, and maps the cues to the segment editor. If caption tracks are missing, the system falls back to the normal Automatic Speech Recognition (ASR) path.
* **Importing Subtitles Manually**: You can drop or choose an `.srt` or `.vtt` file to replace current segment text and timings. VoiceStudio retains speaker voice references where the timing overlaps with the new cues. Malformed, overlapping, or duration-clamped cues are flagged in the sidebar.

## Translation Engines

Translation is pluggable. Select your preferred engine in the Dub tab's **Engine** dropdown.

| Engine | Category | Required Package | Key Required |
| :--- | :--- | :--- | :--- |
| **Argos** | Offline | `argostranslate` (bundled) | No |
| **NLLB-200** | Offline | None (uses `transformers`) | No |
| **Google Translate** | Online | `deep_translator` | No |
| **DeepL** | Online | `deep_translator` | Yes (`DEEPL_API_KEY`) |
| **Microsoft Translator** | Online | `deep_translator` | Yes (`MICROSOFT_API_KEY`) |
| **MyMemory** | Online | `deep_translator` | No |
| **LLM** | LLM-based | `openai` | Yes (usually) |

> **Note:** Signed desktop installers (`.dmg`, `.msi`, AppImage, `.deb`) ship with a read-only, code-signed Python environment. Optional packages cannot be installed in-app on these builds. Choose **Argos** or **NLLB-200** for fully offline, zero-install translation. If you run VoiceStudio from source, you can install optional online packages via the UI or by running:
> ```bash
> uv pip install deep_translator openai
> ```

### Pinning the LLM Provider
To bypass the in-app selector on shared or headless machines, pin your preferred active LLM provider using the `LLM_DEFAULT_PROVIDER` environment variable:
```bash
LLM_DEFAULT_PROVIDER=groq # Supported: openai, openrouter, orcarouter, cheaperinference, cerebras, iflytek, ollama, custom, etc.
```

### Paste Translation
If you have already translated your transcript using an external service or human translator, click **Paste Translation** above the segment table. The system auto-detects three input shapes:
* **Timestamped**: Maps to existing segments based on `.srt` or `.vtt` time overlaps.
* **Numbered lines** (e.g., `1. ...` or `[2] ...`): Maps by line number.
* **Plain lines**: Maps sequentially by line position. Blank lines serve as separators.

Applying a pasted translation only writes to the currently selected language tab and marks the modified lines for selective regeneration.

## Translation Quality and LLM Settings

Translation quality is determined by three operational modes:
* **Fast**: Direct, one-shot translation from the selected engine without timing awareness.
* **Cinematic**: Uses an LLM to refine the literal translation into natural, in-context phrasing.
* **Autofit**: Cinematic translation combined with a strict fit-to-time pass. The LLM rewrites long lines so their spoken duration fits within the segment's allocated timeframe.

Cinematic and Autofit modes require a configured LLM provider under [Installation & Setup](/getting-started).

### Two-Stage LLM Quality
When using the **LLM (OpenAI-compatible)** engine, you can activate two extra quality steps:
1. **Auto Glossary**: Runs a single LLM pass over the entire transcript to extract a theme summary and terminology map. This context is injected into every segment's translation prompt to ensure character names and domain terms remain consistent. Manual glossary entries always override auto-extracted terms.
2. **Reflect Pass**: Executes a critique-and-rewrite cycle for each segment to convert stiff direct translations into natural spoken dialogue. This uses three LLM calls per segment instead of one.

### Custom Translation Style
If you use **Translate with agent**, you can fill in a **Translation style prompt** (up to 5,000 characters). This allows you to describe the desired tone, target audience, formality, or idiomatic handling. The style prompt persists in your project settings and applies to all target languages.

### Regional Dialects
Specify BCP-47 dialect codes (such as `es-MX` for Mexican Spanish, `es-AR` for Rioplatense Spanish, or `pt-BR` for Brazilian Portuguese) to inject localized vocabulary and grammatical instructions into LLM prompts.

## The Editing Workspace

The editing interface provides full control over the timing, translation, and voices of individual segments.

```
+--------------------------------------------------------------+
|                         Video Preview                        |
+--------------------------------------------------------------+
| [ Timeline Segment Block ]   [ Drag / Nudge / Resize ]       |
+--------------------------------------------------------------+
| Row 1: Source Text     | Editable Translation | [Voice Chip] |
| Row 2: Source Text     | Editable Translation | [Voice Chip] |
+--------------------------------------------------------------+
```

### Proportional Timing Lane
* Drag segment blocks horizontally to adjust their start and end points.
* Grab edge handles to resize blocks.
* Use keyboard shortcuts to nudge or delete selected blocks.
* Overlapping segments are highlighted on the timeline with visual warning badges.

### Segment Row Editing
* **Split & Merge**: Split segments at your cursor or merge a segment with its neighbor. VoiceStudio preserves speaker, voice, direction, and gain attributes during these changes.
* **Segment Controls**: Expand the speaker header to adjust individual segment volume gain (0 to 2), set custom speed rates, or add unique direction notes.
* **Bulk Operations**: Check multiple segments to perform bulk voice assignments, language resets, or deletions.
* **Undo/Redo**: The editor retains up to 50 edit states in its history. This history resets when you load a new source file, import subtitles, or switch projects.

## Advanced Speech Controls

### Mirror Source Delivery (Prosody Mirror)
Select **More → Mirror source delivery** to automatically generate voice directions matching the original actor's speech. 

VoiceStudio measures pitch, loudness, syllable rate, and voicing on the separated vocals track and compares each segment against the speaker's baseline. It populates empty direction fields with corresponding descriptive tags:

| Original Delivery (vs. Baseline) | Assigned Direction |
| :--- | :--- |
| Louder, faster, and higher | `urgent` |
| Louder, higher, or more animated | `energetic` |
| Quieter, lower, and flatter | `calm` |
| Loud but unhurried | `announcing` |
| Mostly unvoiced and quiet | `whispered` |
| Faster or slower syllables | `quick` / `slow` |

This analysis runs locally using NumPy and takes approximately 10 seconds per hour of dialogue. 

### Sound Preservation
Background-preserving previews and exports keep original stereo elements (music, audience reactions, ambience) intact outside dialogue intervals. Inside dialogue segments, the system mixes dubbed speech over the separated background audio with a 10 ms transition crossfade.

### Create Story
When your dubbed project contains multiple speakers, click **Create Story** to carry all segments and cast voices directly into the [Audiobooks & Stories](/guides/audiobooks) editor. Speakers are mapped to corresponding Story characters, and merged rows are split back to their assigned speakers.

## Voice Generation and Timing Options

During generation, choose from four primary timing and alignment strategies:
* **Concise**: Fits the translation naturally without cutting off words or overstretching audio.
* **Smart Fit**: Dynamically retimes and resynthesizes segments. You can adjust speed limits using the **Fastest speed** (`audio_rate_cap`, 1.0 to 2.0, default 1.5) and **Slowest speed** (`min_audio_rate`, 0.5 to 1.0, default 0.85) sliders.
* **Stretch Video**: Stretches or slows the video frames to match the duration of the translated audio.
* **Lip Sync**: Coordinates facial/lip movements with the generated audio track.

### Fit Prediction Badges
Before starting generation, VoiceStudio evaluates each translated line and displays fit prediction badges:
* **Tight fit**: The line can be spoken in time but requires audible speed-up.
* **Won't fit +Ns**: The translation is too long and will be trimmed. Check **Suggest shorter lines** to prompt the LLM for shorter, meaning-preserving rewrites.

## Exporting Dubbed Video

Export controls are located in the sidebar panel. You can configure options individually per language.

### Audio and Track Settings
* **Default Audio Track**: Select which language track plays by default in standard media players. Choose **Original** if the source audio should play first.
* **Background Mix**: Enable or disable the preserved background music and ambient sounds.
* **Audio Format**: Export audio-only stems as MP3 (with bitrate choices) or WAV format. You can also package individual language stems as a ZIP file.

### Subtitle Burn-In (Hardsub)
To burn subtitles directly into the video frames, enable **Burn subtitles into picture (hardsub)**. Two layout styles are supported:
* **Line**: Standard, static bottom-of-screen captions.
* **Karaoke (word highlight)**: Highlight colors sweep across words in sequence. This uses word timings recorded during transcription.
  :::note
  Karaoke burn-in is disabled when using the dual-layout (original + translation) subtitle display. Stretch Video mode also disables subtitle burn-in; use sidecar exports instead.
  :::

You can download the karaoke styling script separately as an `.ass` sidecar file, or export standard `.srt` and `.vtt` subtitle files.

## Troubleshooting

### Unreadable Source File
If the ingestion process reports that a source file is unreadable, verify that the file plays correctly in standard players. This error occurs if a download was interrupted (resulting in a file padded with zeros) or if the storage drive has run out of space. Check your directory in **Settings → Storage**.

### Optional Translation Package Missing
If you receive an error stating that `deep_translator` or `openai` is missing:
1. **Source/Development Install**: Click the highlighted **Install** button next to the Engine label or manually install the package using `uv pip install`. Restart the backend to load the package.
2. **Packaged Build**: Use the built-in, offline **Argos** or **NLLB-200** engines which require no external dependencies.

### Argos CTranslate2 Runtime Errors
On Linux kernels that restrict executable stacks, the Argos CTranslate2 runtime may fail to load. The API will return `argos_runtime_unavailable`. Resolve this by upgrading your environment to Python 3.12+ or clearing the executable stack flag:
```bash
patchelf --clear-execstack <path_to_ctranslate2_library>
```
Alternatively, switch to **NLLB-200** as your offline translation engine.