OpenVoice

Chatterbox, from Resemble AI, supplies the default voice, expressive controls and zero-shot cloning. Its package pins much of its stack exactly: torch 2.6.0, transformers 5.2.0, safetensors 0.5.3, gradio 6.8.0, and numpy below 2. VoxCPM2, from OpenBMB, designs voices from text and accepts LoRA adapters. Rather than force VoxCPM2 into Chatterbox’s pins, I gave each engine its own uv-locked environment and its own process. The two lock files resolve 130 and 161 packages; of the 104 they share, 10 land on different versions, including numpy 1.26 against 2.4.

The FastAPI backend fronts both through one abstract engine interface, and a single speech route dispatches on each voice’s engine. VoxCPM2 runs as an always-on worker that listens only on localhost, and the backend calls it over HTTP. The worker loads the base model once with LoRA layers attached, then hot-swaps a per-voice adapter on request. The current adapter is cached, so repeat calls skip the reload, and torch.compile stays off so the swaps stay clean. The browser sees one port that serves both the UI and the API.

The release belongs to the HonedEdge project, a nonprofit I’m standing up, so every component had to pass that filter, weights included.

A third engine, IndexTTS2, had been integrated for emotion and inflection, and it ran, though it saw little use. Its weights come under bilibili’s own Model Use License Agreement. That licence calls itself open-source, yet it counts model outputs within its definition of derivative works, requires a separate licence above a scale threshold, and makes whoever distributes the model answerable for downstream recipients. I judged it incompatible with an AGPL release and removed it before the public tag: its adapter, its worker and its UI.

One remnant stays on purpose: a tag stripper on the speech route, so an inline emotion tag left in older text is never read aloud.

Voice Lab runs that recipe as one job with five monitored stages. You can record a browser tab for 30 minutes and walk away: the build starts itself when the capture ends. Most stages shell out to blocking external tools (ffmpeg, an audio slicer, Whisper speech recognition, the LoRA trainer), so the job runs on a background thread. The browser polls a job endpoint every two seconds, and the state is also written to a status file on disk. The failure modes were the hard part:

• The slicer exits with an error on a Unicode print even when it succeeds, so success is judged by the clips it produced, not by its exit code.
• Speech recognition failed silently because it inherited the app’s model-cache setting. The job now strips that variable for the slicing and speech-recognition tools, and if either produces nothing, it shows that tool’s error output.
• A five-clip test crawled, because on Windows every data-loader worker re-imports PyTorch. Below 64 samples, the job now loads data in-thread.
• An idle-sleeping PC killed a long build. The server now holds a Windows keep-awake for the length of the job, and the browser holds a screen wake lock while recording.

Browser capture turns off auto-gain, echo cancellation and noise suppression, so the training audio is a faithful copy. Each build trains into a fresh checkpoint directory and checks that the step-100 adapter exists before registering anything. If the external tools aren’t configured, the build endpoint answers 503 “not configured” and everything else keeps working.

The agent built blind A/B sets: the same script and voice, one change at a time, several generations each. I judged them by ear, and every verdict was logged so a settled question stayed settled. For fine-tuned voices, that produced a recipe that now lives in code rather than in anyone’s memory:

• LoRA rank 32 and alpha 32, on both the language-model and diffusion layers
• 100 training steps
• guidance 2.0
• 28 diffusion steps instead of the worker’s default 10, the one setting that clearly improved quality

The same sessions showed that a voice’s delivery comes from sampling, not from any setting, and voice design behaves the same way. So Text to Voice renders three takes around guidance 2.0, at 1.75, 2.0 and 2.25, and you keep the best. A studio-recording steer is added to every description so the model doesn’t invent a room, and the chosen take is frozen as a reference clip, which turns a random design into a stable, repeatable voice. Library previews are a real synthesis in the voice, not its source clip.

Listening also caught a bug no type checker would. Reading the wrong sample-rate attribute wrote 48 kHz audio under a 16 kHz header, so it played back slow and low. The worker now reads the model’s true output rate.

• Voice clips, the voice registry and the adapters stay out of git by where they live: a gitignored data/voices folder (adapters included), plus a blanket ignore on adapter weights. The only built-in voice is the engine’s default, defined in code, so a fresh clone works with no data at all.
• The registry stores bare filenames, resolved per machine, so a fine-tuned voice moves between machines as three pieces: its registry entry, its reference clip and its adapter folder.
• Eight environment variables set every machine-specific location: model cache, built UI, build cache, Voice Lab scratch space and four external training tools. Model weights and the build cache stay out of the backed-up, synced tree; irreplaceable data stays in it.
• Rewritten commits stay reachable by hash on GitHub, so the public repository was recreated with one clean initial commit rather than force-pushed.

For whoever installs it, one admin run of setup.bat builds both engines from their frozen lock files with CUDA 12.4 PyTorch, builds the UI, pre-downloads both models, and checks that the CUDA runtime library actually loads, because a basic Windows display driver can provide nvidia-smi without it. openvoice.bat starts, stops and reports on the two processes.