Live · Open source · openearproject.org

OpenEar

Real-time speech captioning and translation into roughly two hundred languages, running entirely on one machine in the building. No cloud subscription. No app to install. No audio ever leaves the premises.

LivePythonFastAPI ONNX RuntimeCTranslate2CUDA WebSocketAGPL-3.0

What it does

And why it exists

Audio comes off a soundboard feed. The server transcribes it continuously, translates it, and pushes the result over WebSocket to every phone and tablet on the local network — with each listener independently choosing their own language. Someone scans a QR code in the lobby and reads along in Korean while the person next to them reads along in Spanish.

It started smaller than that. One member of my congregation is deaf and couldn't always get an interpreter. Accessibility for one person turned into two hundred languages for anyone, which is a fairly typical shape for this kind of problem — the accommodation is usually the general solution wearing a smaller hat.

~200languages available to each listener

0bytes of audio leaving the building

2production ML models on one machine

124 minof audio in the model evaluation

The decision I'm proudest of

Choosing the model that lost

The obvious speech-recognition choice scored better. I didn't pick it.

I ran a five-phase evaluation across 124 minutes of audio and three recognition models, measuring word error rate under varying conditions. The first finding was that chunk size dominates everything: transcribing in three-second pieces produced a 14.4% error rate, while ten-second pieces produced 3.0%. That alone reshaped the audio pipeline.

The trade-off

On raw accuracy, Whisper large-v3 beat NVIDIA's Parakeet by 0.3% — 2.8% against 3.1%, which is inside the noise but still a loss. I chose Parakeet anyway, because Whisper's punctuation is inconsistent at chunk boundaries, and the downstream translation model uses capitalization and terminal punctuation to detect where sentences end.

For a verb-final language like Korean, a sentence boundary in the wrong place doesn't degrade the translation — it destroys it, because the verb hasn't arrived yet. So the model that wins on the headline metric loses on the metric that actually reaches the listener. The number that matters is the one at the end of the pipeline.

There was a second reason, and it's the kind of thing you only see if you're holding the whole system: Parakeet runs acceptably on the CPU. Putting recognition on the CPU leaves the entire GPU budget free for a 3.3-billion-parameter translation model. One decision, two constraints satisfied.

Architecture

Real-time, on one box, with volunteers running it

Segmentation on meaning, not on the clock. The pipeline watches the trailing audio for a silence threshold and cuts only between five and ten seconds — so segments land at natural pauses rather than at arbitrary intervals, which is what keeps sentences intact for the translator downstream.

Translation cost is O(languages), not O(clients). Fifty people reading in Korean cost one inference, not fifty — the result is computed once per distinct language and fanned out through a cache. This is the difference between a system that works in a room of ten and a system that works in a room of four hundred.

Inference never blocks the event loop. Model calls are dispatched to an executor, so WebSocket connections stay responsive while the GPU is busy. English listeners get fragments the moment they're transcribed; translation listeners wait for a complete sentence, because a half-sentence translates into nonsense.

It has to install by right-click. The people running this are volunteers in a sound booth, not operators. Setup is a single script that installs dependencies, downloads about five and a half gigabytes of models, and configures the machine. It serves on port 80 so nobody has to type a port number, and it prints a QR code for the lobby.

An engineering constraint, used as a fundraising argument

The part I didn't expect to be the hard part

The translation model needs a GPU, and the churches that most need this software are the ones that can least afford one. So the specification names a particular card — not the fastest, but the only current one that draws all its power from the PCIe slot itself.

That single property means it drops into a church's existing office computer without a power supply upgrade, without a case swap, without a technician. It collapses the cost of a deployment from "a new machine and someone to build it" down to the price of one card, which is small enough to ask a manufacturer to donate. The hardware constraint and the funding strategy turned out to be the same problem.

Status and honest notes

Where it actually is

Version 0.5.1. It runs, it's been demonstrated publicly, and it received the Pursuing and Embracing Diversity award at district conference. Church testing is underway; it is not yet deployed at third-party sites, and I'd rather say that plainly than let "live" imply more than it should.

The evaluation harness is a word-error-rate benchmark rather than a unit test suite — there are no unit tests for the server itself, which is the gap I'd close first if this were a team project rather than a one-person one.

OpenEar listener view captioning a sermon in English

The same passage of the same sermon, read three ways. Two details worth catching. The address bar says zora.local — a machine in the building, not a service somewhere, which is the whole architectural claim sitting in plain sight. And the A− / A+ controls are there because this began as accessibility for one person; a caption nobody can read at their own text size is not accessible, it is just present.

Twenty-seven seconds, unedited, no audio. It opens on “Waiting for captions…”, then text arrives ghosted and brightens as it settles — provisional output shown as provisional rather than asserted and then silently corrected. Then it scrolls. Stills can prove three languages; only this proves the words are arriving while the sentence is still being spoken.