💻 GitHub:
https://github.com/kadirb4rut/video-dubbing-translator

The current pipeline combines Whisper / WhisperX, VoxCPM2, audio separation, translation, FFmpeg/MoviePy, and optional LatentSync into one workflow.

What it does

The pipeline looks like this:

Video → vocal/background separation → Whisper → WhisperX alignment → translation → VoxCPM2 voice cloning → duration fitting → video reconstruction → optional LatentSync

In practice, it:

• extracts and separates speech from the background audio

• transcribes the speech with Whisper

• aligns speech timing with WhisperX

• translates each speech segment

• uses the original speaker as a reference for VoxCPM2

• generates new speech at 48 kHz

• adjusts generated segments to approximately match the original timing

• mixes the original background audio back in

• rebuilds the video

• can optionally apply AI lip-sync with LatentSync

Why VoxCPM2?

The project originally used XTTS-v2, but I wanted a commercially friendlier and more capable multilingual voice-cloning backend.

I eventually migrated the pipeline to VoxCPM2.

VoxCPM2 supports 30 languages and produces native 48 kHz speech, while also supporting reference-based / zero-shot voice cloning.

The current version of my application exposes 11 validated dubbing target languages, while the underlying VoxCPM2 model supports a broader 30-language set.

English → Turkish real-world test

I ran a full end-to-end test using a 15-second English video.

🎬 Original English video:
https://github.com/kadirb4rut/video-dubbing-translator/releases/download/demo-videos/input-highlight-15s.mp4

🇹🇷 Turkish dub generated with VoxCPM2:
https://github.com/kadirb4rut/video-dubbing-translator/releases/download/demo-videos/target-voxcpm2-highlight-15s.mp4

The test went through the actual production pipeline rather than an isolated TTS demo:

separation → transcription → alignment → translation → four VoxCPM2 generations → duration fitting → reconstruction → background mix

The resulting video remained approximately the same duration as the original, and the Turkish speech retained noticeable characteristics of the source speaker.

The surprising part: it ran on an 8 GB M1

One of the more interesting experiments was running the entire base pipeline on an 8 GB Apple M1 Mac.

It worked.

It was definitely not fast — VoxCPM2 inference on CPU/bfloat16 took a long time — but the complete pipeline successfully finished.

For one direct Turkish VoxCPM2 generation during testing:

• model load: ~59 seconds

• synthesis: ~139 seconds

• output: 48 kHz mono audio

The complete 15-second video test required multiple speech generations and took significantly longer.

For actual use, an NVIDIA CUDA GPU is clearly the more practical option.

Local-first, not completely offline

I describe the project as local-first rather than fully offline.

Video processing, transcription, voice generation, audio separation and reconstruction run locally.

However, the current translation stage uses deep-translator with Google Translate, so transcript text is sent to an external translation service.

One item on the roadmap is adding an optional fully local translation backend.

Current stack

The main pieces are:

• Python 3.10

• Whisper

• WhisperX

• VoxCPM2

• FFmpeg

• MoviePy

• vocal/background separation

• LatentSync 1.6 (optional)

• local browser GUI

The base dubbing pipeline does not require CUDA.

LatentSync does.

Open source

The project is available here:

👉 https://github.com/kadirb4rut/video-dubbing-translator

The original project code is MIT licensed, while the project documents the licenses and terms of the third-party models and components separately.

I'm particularly interested in feedback from people working on:

• speech AI

• TTS

• voice cloning

• multilingual speech

• dubbing

• audio/video processing

• local AI

I'd also be very interested to see real-world performance numbers from people running the pipeline on NVIDIA GPUs.

If you work in this area, I'd love to hear what you would improve first: speaker similarity, multilingual pronunciation, timing, or the translation layer?