Whisper accuracy benchmark 2026: we tested large-v3, turbo, and small across 12 languages
A reproducible Whisper benchmark: we measured the word error rate of large-v3, large-v3-turbo, and small across 12 languages on the FLEURS test set. 1,749 transcriptions, full methodology, and downloadable raw data.
We ran 1,749 transcriptions to measure how accurate open-source AI transcription actually is in 2026. Three Whisper models (large-v3, large-v3-turbo, and small) transcribed 50 utterances in each of 12 languages from the public FLEURS test set. The headline: large-v3 reaches a word error rate of roughly 3 to 6 percent in high-resource European languages (Spanish 2.9%, Italian 3.2%, English 4.1%), which is close to the commonly cited range for human transcription. In Hindi it climbs to 15.7%, in Egyptian Arabic to 14.6%. Within one model, the gap between the best and worst language is more than 5x, which dwarfs the difference between model versions.
Averages are only half the story, so this article also reports what the averages hide: in English and Italian, more than 60% of sentences came back word-perfect, while Hindi and Cantonese managed 4%. And when the smaller models fail on hard languages, they do not fail gracefully: we caught four textbook repetition-loop hallucinations in the raw output, all from turbo and small, none from large-v3.
Every number in this article comes from a test we ran ourselves in July 2026, with a fixed random seed, a published script, and downloadable raw results. You can reproduce all of it, or check our work line by line.
What we tested and how
Most transcription accuracy numbers on the web are copied from vendor pages or from other articles, without a test behind them. Here is our exact setup:
| Item | Setting |
|---|---|
| Test set | Google FLEURS test split (public academic dataset, read speech) |
| Sampling | First 50 utterances of the shuffled test split per language (shuffle seed 42, deterministic), roughly 9 minutes of audio per language |
| Models | Whisper large-v3, large-v3-turbo, and small (open-source MLX conversions) |
| Inference | temperature 0, source language pinned (no auto-detect), no conditioning on previous text |
| Normalization | Matches the Whisper paper: EnglishTextNormalizer for English, BasicTextNormalizer elsewhere (via transformers) |
| Metrics | WER (word error rate); for Chinese, Japanese, and Cantonese the primary metric is CER (character error rate) |
| Software | mlx-whisper 0.4.3, jiwer 4.0.0, transformers 5.11.0, datasets 3.6.0 |
| Hardware | Apple M3, MLX inference, fp16 |
| Test date | July 2026 |
We picked the 12 languages to span resource levels and writing systems rather than to flatter the models: six high-resource European languages, three space-free CJK languages that need character-level scoring, plus Korean (agglutinative, space-delimited), Hindi (Devanagari), and Egyptian Arabic.
One sampling detail for anyone auditing the raw file: FLEURS often includes several recordings of the same sentence by different speakers, sharing one sentence id. Our resume logic deduplicates by sentence id, so the English large-v3 cell holds 49 utterances instead of 50; every other cell holds 50, for 1,749 transcriptions in total.
Two honest caveats before the table. First, FLEURS is read speech: clean audio, steady pace, one speaker. Real meetings and interviews score significantly worse, because background noise, accents, and overlapping speakers all push error rates up (for practical mitigation steps, see how to transcribe noisy audio). Treat these numbers as each language's ceiling, not a promise. Second, 50 utterances per language is a sample, not the full test set. Individual figures carry a margin of a few percentage points, so read the magnitudes and the ordering, not the decimals.
Results: 12 languages, 3 models
Numbers are WER %, lower is better. For Japanese, Mandarin, and Cantonese (marked *) the figure is CER %, because these languages do not separate words with spaces.
| Language | large-v3 | large-v3-turbo | small |
|---|---|---|---|
| English | 4.1% | 4.7% | 6.6% |
| Spanish | 2.9% | 3.3% | 6.5% |
| French | 5.8% | 6.7% | 14.3% |
| German | 4.6% | 6.0% | 12.8% |
| Italian | 3.2% | 3.9% | 8.2% |
| Portuguese | 6.6% | 7.2% | 11.0% |
| Japanese * | 6.6% | 5.8% | 16.4% |
| Korean | 14.9% | 14.5% | 21.7% |
| Mandarin * | 6.7% | 8.0% | 19.5% |
| Cantonese * | 10.5% | 43.3% | not supported |
| Hindi | 15.7% | 22.3% | 42.4% |
| Arabic (Egypt) | 14.6% | 16.1% | 32.7% |
Three things stand out. The gap between languages is far larger than the gap between models: Spanish 2.9% versus Hindi 15.7% on the same large-v3. The turbo model costs only 0.3 to 1.4 points in the six European languages, but 6.6 points in Hindi, and in Cantonese it collapses from 10.5% to 43.3% CER. And small degrades catastrophically outside high-resource languages: 42.4% in Hindi and 32.7% in Arabic make it a preview tool, not a deliverable.
The "not supported" cell is literal: Cantonese's yue language token was introduced with the large-v3 generation (the official model card lists "a new language token for Cantonese" among the changes), so the older-generation small model rejects the language outright. In our run it raised an unsupported-language error, which is itself a useful data point: before comparing model quality, check that the model supports your language at all.
Beyond the averages: most sentences are perfect, a tail is not
A mean error rate flattens a distribution that is actually very skewed. Per language on large-v3, here is the share of utterances transcribed with zero errors, the median, and the 90th percentile:
| Language | Word-perfect utterances | Median | 90th percentile |
|---|---|---|---|
| English | 61% | 0.0% | 12.5% |
| Spanish | 54% | 0.0% | 7.7% |
| French | 38% | 4.3% | 12.5% |
| German | 56% | 0.0% | 14.3% |
| Italian | 62% | 0.0% | 10.5% |
| Portuguese | 42% | 3.7% | 18.8% |
| Japanese * | 36% | 3.2% | 15.8% |
| Korean | 34% | 13.7% | 35.3% |
| Mandarin * | 40% | 3.0% | 17.8% |
| Cantonese * | 4% | 9.7% | 17.6% |
| Hindi | 4% | 14.9% | 27.5% |
| Arabic (Egypt) | 28% | 10.3% | 37.5% |
Two readings matter. In the strong languages, the median utterance is error-free: English's 4.1% average is produced almost entirely by a minority of harder sentences, which is why a transcript there feels near-perfect with occasional flaws to fix. And the perfect-rate column separates languages that the average lumps together: Cantonese (10.5%) and Arabic (14.6% mean) look closer than they are, but only 4% of Cantonese sentences came back flawless versus 28% for Arabic. If you proofread by scanning for the bad patches, the tail percentile predicts your workload better than the mean.
Finding 1: the language gap dwarfs the model gap
Within large-v3, the spread between the best language (Spanish, 2.9%) and the worst (Hindi, 15.7%) is more than 5x. English (4.1%) versus Egyptian Arabic (14.6%) is close to 4x. Meanwhile, within any single European language, upgrading from turbo to large-v3 moves the number by around one point (English 4.7% to 4.1%, German 6.0% to 4.6%).
The asymmetry is even sharper for small. In English it trails large-v3 by only 2.5 points. In French it degrades from 5.8% to 14.3%, and in Hindi from 15.7% to 42.4%. The same checkpoint that looks like a reasonable trade-off in English is unusable in Hindi.
The practical conclusion: whether a model supports your language well matters far more than which version of the model you pick. If you work in Spanish, almost anything modern is fine. If you work in Hindi or Cantonese, the model choice decides whether the transcript is usable at all.
Finding 2: what turbo actually costs you
On our M3, typical throughput was roughly 3.8x real time for large-v3, 5.2x for turbo, and 14.3x for small, averaged across runs (individual runs varied with system load, and absolute speeds depend on hardware; the ratios are the signal). So turbo runs about 1.4x faster than large-v3.
In the six high-resource European languages, that speed costs almost nothing: turbo trails large-v3 by about 0.8 points on average, and in Japanese and Korean turbo actually edged out large-v3 slightly in our sample (5.8% vs 6.6% CER, 14.5% vs 14.9% WER). This is why commercial transcription services have broadly moved to distilled and accelerated models.
But turbo is not a free lunch. Hindi degrades by 6.6 points (22.3% vs 15.7%), Egyptian Arabic by 1.5, and Cantonese collapses from 10.5% to 43.3% CER, which is unusable. The decision to trade accuracy for speed has to be made per language, not globally.
Finding 3: languages without word boundaries need a different ruler
Chinese, Japanese, and Cantonese do not separate words with spaces, so a "word" error rate depends entirely on how you split the text, and different tokenizers produce wildly different WER for identical transcripts. Academic practice, including the Whisper paper itself, is to use character error rate (CER) for these languages, and that is what we report.
Measured that way, Japanese (6.6% CER) and Mandarin (6.7% CER) sit in the same quality band as the strong European languages on large-v3. Cantonese is the sharpest counterexample in our whole dataset: 10.5% CER on large-v3, 43.3% on turbo, no support at all on small. One language, three models, and the outcome ranges from usable to impossible.
Korean deserves a footnote: its 14.9% looks poor, but Korean is agglutinative and uses spacing conventions that inflate space-delimited WER, so its character-level accuracy is better than the number suggests. It still needs more proofreading than the European languages, just not as much as a naive WER reading implies.
The takeaway for evaluating any vendor: if a tool claims "95% accuracy in Chinese" without saying whether that is WER or CER, on what audio, with what normalization, the number carries no information. Our WER explainer covers the metric mechanics in detail.
Finding 4: when the smaller models fail, they fail all at once
Averages suggest errors are sprinkled evenly through a transcript. The raw output says otherwise. Scanning all 1,749 hypotheses for degenerate output (error rate above 100%, or output more than twice the reference length), we found exactly four cases: three from turbo (two Cantonese, one Hindi) and one from small (Hindi). Large-v3 produced zero across all 599 of its utterances.
All four are the same failure mode: a repetition loop. The worst Cantonese case scored 372% CER (insertions can push the metric past 100%): turbo transcribed the first few words plausibly, then locked onto a single character and emitted it 46 times in a row. The Hindi failures look identical in shape, one word stamped over and over. On the very same utterance, large-v3 produced a near-perfect transcript at 6.9% CER. The exact strings are in the raw data (utterance id 1980 in results.jsonl) if you want to inspect them.
That Cantonese case quietly shows a second failure: before entering the loop, turbo had already drifted out of Traditional Chinese into simplified characters and Mandarin word choices, while large-v3 stayed in the reference's script. For Cantonese speakers this matters as much as the error rate, because the "transcript" is sliding toward a different written language.
Hallucination in Whisper-family models is documented beyond our sample: Koenecke et al. (2024) audited Whisper transcriptions and found on the order of 1% of segments contained fabricated content, concentrated around pauses and disfluent speech. Our read-speech rate is lower (4 in 1,749, about 0.2%), which fits: clean, continuous audio gives the model fewer silences to fill. The practical rule we take from both: the worst transcription failures are not uniformly distributed noise but total local corruption, they cluster in smaller checkpoints on weaker languages, and a quick scan for repeated tokens catches most of them.
How our large-v3 numbers compare with the published large-v2 table
Our per-language accuracy guide compiles the Whisper paper's published large-v2 FLEURS results. Lining those up against our July 2026 large-v3 measurements for the overlapping languages (paper numbers are the full test split, ours a 50-utterance sample, so read direction rather than decimals):
| Language | Paper large-v2 | Our large-v3 | Direction |
|---|---|---|---|
| Spanish | 3.0% | 2.9% | flat, already strong |
| English | 4.2% | 4.1% | flat, already strong |
| Italian | 4.0% | 3.2% | slightly better |
| German | 4.5% | 4.6% | flat |
| French | 8.3% | 5.8% | better |
| Japanese (CER) | 5.3% | 6.6% | flat within noise |
| Korean | 14.3% | 14.9% | flat within noise |
| Mandarin (CER) | 14.7% | 6.7% | halved |
| Arabic | 16.0% | 14.6% | slightly better |
| Hindi | 21.5% | 15.7% | markedly better |
The pattern is consistent with where the extra training data went: the languages that were already near their floor stayed put, and the big gains landed on Mandarin (cut in half), Hindi, and French. Cantonese does not appear in the large-v2 table at all, which is the bluntest version of the same story: between model generations, the most important change for some languages is not a lower error rate but existence.
What these numbers mean for your work
Translating error rates into felt experience: 5% means one error every 20 words, a quick proofread. 15% means one error every 7 words, and editing starts to cost real time. Above 30%, whole passages need re-listening, and the transcript is a draft at best.
- High-resource European languages (English, Spanish, French, German, Italian, Portuguese): large-v3 lands at 3 to 7 percent on clean speech, solidly in "trust the first draft, then skim" territory. Turbo costs under 1.5 points and is fine for routine work. Small degrades to 13 to 14 percent in French and German; do not use it for anything you deliver.
- Japanese, Mandarin, Korean: Japanese and Mandarin at 6 to 7 percent CER on large-v3 are first-draft trustworthy, as long as you read the number as character-level. Korean's WER overstates its errors, but plan for more correction than the European group.
- Cantonese, Hindi, Arabic: the caution zone. Hindi and Egyptian Arabic sit at 15 to 16 percent even on large-v3, one error every 6 or 7 words, so budget substantive editing. Cantonese is only viable on large-v3; pick the wrong variant and the error rate quadruples.
Remember that all of this is read-speech ceiling performance. Real recordings score worse, and the three variables that matter most are recording quality, speaker count, and accent strength. For multi-speaker material, speaker attribution is a separate error class on top of word accuracy; see what speaker diarization is for how that works.
If you want the per-language picture across roughly 100 languages rather than our 12, our per-language accuracy guide compiles the Whisper paper's published FLEURS table, tiered from near-human to effectively unusable. That page is a compilation of published results; this one is the benchmark we ran ourselves, and the two agree where they overlap.
Reproduce it, or check our work
Everything needed to verify or extend this benchmark is public:
- results.jsonl: all 1,749 per-utterance results, with reference text, model output, and per-utterance WER/CER
- results-summary.csv: the aggregated language x model table behind this article
- bench.py: the exact script that produced the data, with deterministic sampling (seed 42)
- Data package README: setup commands, versions, and licenses
Run python bench.py --n 50 and you select the same utterances we did, thanks to the fixed shuffle seed; increase --n to grow the sample (the FLEURS test split has roughly 350 to 900 utterances per language). The script streams FLEURS from Hugging Face, transcribes locally, scores with Whisper-paper normalization, and appends one JSONL line per utterance, so interrupted runs resume cleanly.
Attribution: reference transcriptions come from Google's FLEURS dataset (Conneau et al., 2022), redistributed under its CC-BY-4.0 license. Whisper model weights are OpenAI's, released under Apache 2.0.
Frequently asked questions
Is Whisper the most accurate open-source transcription model?
It is the most widely reproduced and language-broad open family, which is why it makes a good public baseline. Other open-source families exist, including Meta's MMS and Seamless lines and many language-specific fine-tunes, but they differ in language coverage, licensing, and evaluation conventions, so a fair cross-family comparison needs its own controlled test. We chose Whisper because the weights are fixed and public, results are reproducible, and one method covers all 12 languages.
Why didn't you test commercial services like Otter, Rev, or HappyScribe?
Commercial terms of service commonly restrict publishing benchmark results, the models behind the APIs are opaque and change without notice, and a number measured today may be meaningless next quarter. Open-source weights are frozen, so anyone can re-run our exact test indefinitely. Vendor accuracy claims are on their websites; note how rarely they disclose the test set and normalization.
Are 50 utterances per language enough?
Enough for magnitudes and ordering, not for fine-grained significance testing. That is exactly how we present the results. The full FLEURS test split has roughly 350 to 900 utterances per language, and growing the sample is a one-parameter change to the published script.
Do read-speech results represent real recordings?
No. Everything we measured here is ceiling performance: clean, single-speaker, steadily paced audio. On real meetings, phone calls, and interviews, expect meaningfully higher error rates, with the damage growing as noise, accent strength, and speaker overlap increase. What transfers from this benchmark to the real world is the ordering (a language or model that wins on clean speech usually wins on messy speech too), not the absolute percentages.
How exactly do I reproduce the test?
Download bench.py, install the dependencies listed in the data package README, and run python bench.py --n 50. The seed is fixed in the script, sampling is deterministic, and results append to a JSONL file that resumes after interruption. The script uses MLX (Apple Silicon); on other hardware, swap the transcription call for openai-whisper or faster-whisper with the same settings.
Do these models hallucinate?
In our read-speech sample, rarely but catastrophically when it happens: 4 of 1,749 outputs (about 0.2%) were repetition loops, all from turbo or small on Cantonese and Hindi, none from large-v3. Published audits of conversational and disfluent speech report higher rates, on the order of 1% of segments (Koenecke et al., 2024). Expect more of this on real-world audio with long pauses, and scan outputs for repeated tokens.
Which model does Vocova use?
This benchmark measures open-source models, not Vocova's production pipeline, and the point it makes is general: whether your language is well supported matters more than the model version number. Vocova routes transcription to high-accuracy model tiers for multilingual work and adds speaker labels, translation, and export formats on top. The most direct evidence is a test run on your own audio in your own language.
Sources and further reading
- FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech (Conneau et al., 2022), the dataset all our samples come from
- google/fleurs on Hugging Face, dataset distribution and CC-BY-4.0 license
- Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2022), the Whisper paper whose normalization and CER conventions we follow
- openai/whisper-large-v3 model card, documents the Cantonese language token added in large-v3 and the Apache 2.0 weight license
- Careless Whisper: Speech-to-Text Hallucination Harms (Koenecke et al., 2024), the external audit of Whisper hallucination rates we compare our failure counts against
- mlx-whisper on PyPI, the inference runtime used for this benchmark
- jiwer on GitHub, the WER/CER scoring library
Want to see multilingual transcription on your own material? Paste an audio or video link into Vocova and check the output against this article's expectations for your language.
