# Vocova WER benchmark, July 2026

Raw data and script for the article "AI transcription accuracy in 2026: we benchmarked Whisper in 12 languages" (https://vocova.app/blog/ai-transcription-accuracy-benchmark-2026).

## Files

- `results.jsonl`: 1,749 per-utterance results. Each line: model, FLEURS config, utterance id, duration, reference text, hypothesis text, WER, and CER (CER only for ja/zh/yue).
- `results-summary.csv`: aggregated per language x model: mean, median, 90th percentile, and share of utterances with zero errors (primary metric: CER for ja_jp, cmn_hans_cn, yue_hant_hk; WER otherwise).
- `bench.py`: the exact script that produced the data. Deterministic sampling (shuffle seed 42, first 50 of the FLEURS test split per language). Resumable; appends one line per utterance.

## How to reproduce

```sh
uv venv -p 3.12 .venv
uv pip install -p .venv/bin/python mlx-whisper "datasets>=3,<4" soundfile jiwer transformers huggingface_hub
.venv/bin/python bench.py --n 50 --out results.jsonl
```

Versions used for the published run (July 2026): mlx-whisper 0.4.3, mlx 0.31.2, jiwer 4.0.0, transformers 5.11.0, datasets 3.6.0, on an Apple M3 (MLX, fp16). Requires Apple Silicon; on other hardware, port the `mlx_whisper.transcribe` call to `openai-whisper` or `faster-whisper` (same models, same normalization, comparable results).

## Licenses and attribution

- Reference transcriptions (`ref` field) come from Google's FLEURS dataset, redistributed here under its CC-BY-4.0 license. Attribution: FLEURS (Conneau et al., 2022), https://huggingface.co/datasets/google/fleurs, paper: https://arxiv.org/abs/2205.12446
- Whisper model weights are released by OpenAI under Apache 2.0; MLX conversions by the mlx-community.
- `bench.py` and the derived results (hypotheses, WER/CER numbers, summary CSV): MIT. Reuse freely with a link back to the article.