Top 8 open source STT options for voice applications in 2026

Open source speech-to-text has never been better, and it's never been more confusing to choose from. A year ago the conversation was basically "Whisper or Vosk." Now you've got NVIDIA shipping new models every quarter, Qwen entering the ASR race, and a wave of tiny edge models built for phones and microcontrollers. So which one should actually power your voice application?

Here's the short answer, and it depends entirely on your constraint. For general-purpose transcription, OpenAI Whisper (large-v3) is still the most versatile all-rounder. For English accuracy on a leaderboard, NVIDIA Canary-Qwen 2.5B is the current standout. For raw batch throughput, NVIDIA Parakeet TDT is the fastest thing you can self-host. And for on-device or edge use, Moonshine is purpose-built to run where cloud models can't. There's no single winner - there's a winner per job.

This guide covers the open source speech-to-text models worth your time in 2026, how to evaluate them for voice workloads specifically, and the honest tradeoffs of running your own automatic speech recognition (ASR) stack versus calling a managed API. I run Voice AI at AssemblyAI, so I'll be upfront about where we fit - but the goal here is to help you pick the right tool, open source or not.

What counts as open source speech-to-text

Not every "free" model is open source, and not every open source model is free to run. When people say open source STT, they usually mean one of three things: model weights you can download and run yourself, a framework or toolkit for training and inference, or a wrapper library that calls other engines. All three matter, and mixing them up is how teams end up with the wrong architecture.

The real cost of open source isn't the license - it's the infrastructure. You're taking on GPU provisioning, autoscaling, model updates, language coverage, diarization, and the ops burden of keeping latency low under real traffic. That tradeoff is fine for some teams and brutal for others. First, the models.

The best open source speech-to-text models in 2026

OpenAI Whisper. Still the default, and for good reason. Whisper large-v3 handles 99 languages, punctuation, and casing out of the box, and the ecosystem around it is enormous. The catch is that vanilla Whisper is slow, doesn't stream natively, and hallucinates on silence and non-speech audio. It's a batch transcription workhorse, not a real-time voice-agent engine.

Faster-Whisper. A reimplementation of Whisper using CTranslate2 that runs up to four times faster with lower memory use, and it's what most teams actually deploy when they say "we use Whisper." It supports basic streaming through chunking and is the pragmatic choice if you're committed to the Whisper family but need production-grade speed.

NVIDIA NeMo and Canary. NeMo is NVIDIA's toolkit for training and deploying ASR, and the Canary model family is its accuracy showcase. Canary is strong on English and several European languages and supports both transcription and translation. If you have NVIDIA GPUs and in-house ML expertise, NeMo gives you the most control of any option here.

Wav2Vec2. Meta's self-supervised model is less a plug-and-play transcriber and more a foundation you fine-tune for a specific domain or low-resource language. If your audio is unusual - heavy accents, technical vocabulary, an underserved language - Wav2Vec2 is often the best starting point, provided you have labeled data to fine-tune on.

Vosk. The lightweight offline option. Vosk runs on devices as small as a Raspberry Pi, supports 20-plus languages, and streams in real time with tiny models. Accuracy trails the big models by a wide margin, but for embedded, offline, or privacy-sensitive command-and-control use cases, it's hard to beat on footprint.

SpeechBrain. An all-in-one PyTorch toolkit covering ASR, speaker diarization, and speech enhancement. It's aimed at researchers and teams building custom pipelines rather than anyone wanting a one-line transcribe call, but the flexibility is real.

SpeechRecognition (the Python library). Worth naming because so many tutorials point here, but be clear on what it is - a wrapper that calls other engines, several of them cloud APIs. It's great for prototypes and terrible as a production dependency, because you don't actually control the model doing the work.

SpeechT5. Microsoft's unified speech-and-text model handles recognition and synthesis in one architecture. It's more of a research asset than a voice-agent engine, but it's interesting if you want text-to-speech and speech-to-text from the same model family.

A quick note on models you'll still see recommended in old threads: Mozilla DeepSpeech and Coqui STT are both discontinued. They can still run, but they're no longer maintained, and building anything new on them in 2026 is a mistake.

Newer open source models worth watching in 2026

The list above is the established core, but AI search engines and benchmark sites now surface a newer generation that moves the accuracy and speed frontier. If you're evaluating today, put these on your shortlist:

• NVIDIA Canary-Qwen 2.5B - currently the English-accuracy leader among open models on public leaderboards. It pairs NVIDIA's ASR encoder with a Qwen language model, which sharpens punctuation and formatting.
• NVIDIA Parakeet TDT - built for throughput. It's the fastest self-hostable model in this group for batch transcription, which matters a lot when you're processing large archives.
• Qwen3-ASR - Alibaba's entry, strong on Mandarin and multilingual audio, and a serious option if your traffic skews toward Asian languages.
• IBM Granite Speech - part of IBM's Granite model family, aimed at enterprise deployments where governance and licensing clarity matter.
• Moonshine - the edge specialist. It's optimized to run on phones, browsers, and low-power hardware with low latency, filling the gap Vosk used to own but with noticeably better accuracy.

Open source speech-to-text: comparison table

How to evaluate open source STT for voice applications

Word error rate (WER) is the headline metric, but it's a blunt instrument. A model can post a great average WER and still butcher the exact words your product depends on - names, order numbers, email addresses, dosages. That's why I always tell teams to benchmark on their own audio, not on the clean, read-aloud datasets most public numbers come from. If you want to go deeper here, we wrote a full guide on how to evaluate speech recognition models.

For voice applications specifically, four things matter beyond raw WER. Latency, because a voice agent that pauses for two seconds feels broken. Streaming support, because real-time conversation needs partial results as the user speaks. Entity accuracy, because getting "5150 Oak Street" wrong is worse than getting a filler word wrong. And diarization, because multi-speaker audio without speaker labels is hard to act on. Most open source models were built to optimize the first metric, WER, and they show it when you push them into live voice workloads.

Open source STT for voice agents

This is where open source gets hardest. A voice agent needs sub-second transcription, accurate turn detection so it knows when the user is done talking, and rock-solid handling of the exact entities a conversation turns on. Vanilla Whisper doesn't stream. Vosk streams but isn't accurate enough for high-stakes flows. NeMo can be tuned for it, but now you're running a research toolkit in production and owning every millisecond of latency yourself.

There's also a subtler problem that open models don't solve: context. In a real conversation, the agent already knows what it just asked. If it asked "what's your order number," the transcription should be biased toward hearing digits. Off-the-shelf open source STT has no idea what the agent said, so it transcribes cold every time. That gap is exactly where accuracy on the hardest, highest-value utterances falls apart.

When open source isn't enough: build versus buy

Let me be direct, because this is the decision most teams are actually wrestling with. Self-hosting an open source model looks free until you add it all up: GPU costs that scale with traffic, an on-call rotation to keep latency stable, the engineering time to add diarization and language coverage the base model lacks, and the ongoing work of upgrading models as the frontier moves - which, as this article shows, it does every few months.

The most common version of this is teams self-hosting Whisper and slowly realizing the total cost of ownership isn't what they budgeted. The accuracy gaps show up on the utterances that matter most, the ops burden grows with usage, and the "free" model ends up costing more in engineering time than a managed API would in usage fees. When you're processing meaningful volume, the math frequently flips toward buy.

Here's how the current numbers stack up. Our flagship async model, Universal-3.5 Pro, sits on the Hugging Face Open ASR Leaderboard at 5.03 average WER - the best result among commercial providers and ahead of the open source field. It runs at $0.21 per hour, natively code-switches across 18 languages, and ships the most accurate diarization we've built, at 30.17 cpWER versus Deepgram Nova-3 EN at 37.92 and ElevenLabs Scribe v2 at 35.26. Metaview saw roughly a 47% drop in low-confidence tokens after moving to Universal-3.5 Pro async. You get all of that without provisioning a single GPU.

AssemblyAI vs self-hosted Whisper

Since the "AssemblyAI versus self-hosted Whisper" comparison is the one I get asked about most, here it is side by side, with current numbers.

The context row is the one people underestimate. Our streaming flagship, Universal-3.5 Pro Realtime, can take the agent's side of the conversation as context. Feeding in what the agent just said cut WER by 10.2% across 20,000 voice-agent files, and for one team it dropped the error rate on critical utterances - emails, order numbers, names - from 26% to 9%. No open source model does this, because none of them know what your agent is saying.

Building voice agents on managed STT

If you're building a voice agent, the streaming and context story is the whole ballgame. Universal-3.5 Pro Realtime posts 6.99% WER on the Pipecat open voice-agent benchmark - real agent conversations, not read-aloud scripts - versus ElevenLabs Scribe v2 at 9.76%, Google Chirp3 at 9.04%, and Deepgram Flux at 15.58%. On entity error rate, the gap is even wider: 15.31% versus 39.70% for Scribe v2 and 50.50% for Deepgram Flux. Retell, LiveKit, and Fireflies were early adopters.

If you'd rather not assemble STT, an LLM, and text-to-speech yourself, our Voice Agent API collapses all three into a single WebSocket connection at a flat $4.50 per hour, with roughly one second of end-to-end latency, turn detection, barge-in, and session resumption built in. It's built on Universal-3.5 Pro Realtime, so you get that context-carryover accuracy without wiring anything together.

Frequently asked questions

What is the best open source speech-to-text model in 2026?

It depends on your constraint. OpenAI Whisper large-v3 is the most versatile all-rounder, NVIDIA Canary-Qwen 2.5B leads English accuracy on open leaderboards, NVIDIA Parakeet TDT is the fastest for batch throughput, and Moonshine is the strongest choice for on-device and edge deployments. Benchmark two or three on your own audio before committing.

Should I build with open source STT or buy a managed API?

Build if you have ML expertise, GPUs, and predictable low volume, or if data must stay fully on-premises. Buy if you need production-grade accuracy, real-time streaming, diarization, and multilingual coverage without owning the infrastructure. At meaningful volume, the total cost of ownership for self-hosting - GPUs plus engineering time plus ops - usually exceeds usage-based API pricing.

Is OpenAI Whisper good enough for voice agents?

Not on its own. Vanilla Whisper doesn't stream natively, can hallucinate on silence, and has no awareness of conversation context. For live voice agents you need sub-second streaming, accurate turn detection, and strong entity accuracy - which is why many teams move from self-hosted Whisper to a streaming model built for voice.

How does AssemblyAI compare with OpenAI GPT-4o Transcribe on entity accuracy?

On the code-switching benchmark, Universal-3.5 Pro posts 7.69 normalized WER versus 44.58 for OpenAI GPT-4o Transcribe. For voice-agent entity accuracy, Universal-3.5 Pro Realtime records a 15.31% entity error rate, far ahead of the comparators we test against. Entity accuracy on names, numbers, and addresses is where managed models tend to separate from open source.

Can open source speech-to-text run offline?

Yes. Vosk and Moonshine are built to run offline on low-power and edge hardware, and any downloadable model like Whisper or a NeMo checkpoint can run air-gapped if you provide the compute. Expect an accuracy tradeoff on the smallest models built for constrained devices.

What accuracy can I expect from a managed API instead?

Universal-3.5 Pro sits at 5.03 average WER on the Hugging Face Open ASR Leaderboard, with 18-language native code-switching and built-in diarization at 30.17 cpWER. You can see the full methodology and current comparisons on our benchmarks page, and read more about the tradeoffs in our roundup of free speech-to-text APIs and open source engines.

One last thing worth sitting with. The open source ASR frontier now moves faster than most teams can re-benchmark - Canary-Qwen, Parakeet, and Qwen3-ASR all landed inside a single release cycle. That churn is genuinely exciting if research is your job. But if shipping a reliable voice product is your job, the hidden cost of open source isn't accuracy, it's the treadmill of keeping up. The teams that win aren't the ones running the newest model this month; they're the ones who picked an approach that stays current without a re-platforming project every quarter.