Speech-to-Text on Intel Iris Xe, Part 3: Testing Whisper Accuracy with FLEURS and Common Voice

How well does my speech-to-text service hear me, and how do I test that before I ship a change? Part 1 moved Whisper onto idle Intel Iris Xe GPUs, and Part 2 built and released its container images. Those parts were about running the service; this one is about the words it writes.

I dictate in Russian, English, Polish and German, so Part 3 of 4 tests the service in all four. It relies on public recordings from FLEURS and Common Voice rather than my own dictation, so most numbers describe other people’s speech, not mine. I cover how I score Whisper large-v3-turbo on them with word error rate (WER), and how I turn the scoring into a check I can repeat. The short answer: after one fix, WER on clean read speech was between 5.03% and 5.85% in all four languages. On spontaneous English and Russian it roughly doubled, and every change I tested on fresh data failed at least one gate I had written down in advance. I accepted all three anyway, with each risk on the record.

All numbers below come from specific earlier releases and score the final transcript of a complete recording, not the live preview shown while I speak. I have not rerun every test set after later runtime and capture changes.

Just here for the procedure? Jump to the evaluation checklist.

What my GitLab pipeline tests, and what it misses

#

My release pipeline from Part 2 already does more than check that the service starts. Each Whisper release build runs the unit tests, then puts the candidate image on an Intel GPU. There, the best of three speed runs on a frozen 25-second clip must reach 6x realtime, and seven synthetic Russian clips with 338 reference words must stay at or below 8% word error rate. A separate round trip feeds the candidate speech produced by the stable text-to-speech image, and four short Russian phrases must come back with at most 25% word error rate. A phrase that misses gets one fresh synthesis, so the round trip is a compatibility check rather than an unbiased score.

Those gates catch a broken build. They cannot tell me how Whisper handles a quiet speaker, an accent or a spontaneous Polish sentence: every recording they score for accuracy is synthetic Russian speech.

How I measure speech-to-text accuracy

#

Testing accuracy needs three things:

• Recordings with references. Every recording comes with a transcript that people have checked, so the recognizer is never graded against its own output.
• A metric. Word error rate, or WER, counts substitutions, deletions and insertions and divides them by the number of reference words. I summed edits and words across a whole corpus instead of averaging per-clip percentages, which would give a two-word clip the same weight as a long one. Both texts are normalized first, lowercased and stripped of punctuation, because raw WER, as the Whisper paper puts it, “penalizes all differences between the model’s output and the reference transcript including innocuous differences in transcript style”.
• Limits fixed in advance. Pass and fail thresholds for each measure, written down before the run.

WER alone was not enough. I also tracked language identification, the share of clips for which the service reported the language actually spoken, and empty results, recordings that came back with no text at all. And I read the worst rows, because an average can improve while individual recordings get much worse.

Public speech-to-text benchmarks: FLEURS and Common Voice

#

A test set of my own recordings would have measured one speaker. Public corpora already pair speech from many speakers with checked transcripts:

• FLEURS from Google: 2,009 sentences from English Wikipedia, translated by people and read aloud in 102 languages, under CC BY 4.0. Clean read speech makes it a good first control and a poor model of dictation.
• Common Voice from Mozilla, now distributed through the Mozilla Data Collective. In its Scripted Speech datasets, volunteers read prepared sentences and other volunteers validate the recordings. In its Spontaneous Speech datasets, people answer open questions in their own words, and the community transcribes and reviews them. The audio is released under CC0, but each download needs an account and the acceptance of terms that forbid identifying speakers or re-sharing the data. Every language is a separate dataset, so I downloaded only the four I dictate in.

Mozilla publishes new Common Voice releases about once a quarter. The September 2026 release produced Scripted Speech 27.0 and Spontaneous Speech 5.0, covering 295 and 80 languages. A new release is a steady source of recordings, but not automatically fresh ones: the splits are regenerated, so freshness has to be checked clip by clip against everything already used. Each comparison also has to freeze its release, split and text normalization, so that a newer dataset never passes for a better model. The results below use the June releases.

Automatic captions on video platforms are where many people meet speech recognition, and I use them too: live, useful and visibly imperfect. YouTube itself says its automatic captions are generated by machine learning and that their quality may vary. They are another system’s output, not checked transcripts, so they set my everyday expectation but play no part in my tests.

Published numbers are a reference point, not a ranking. OpenAI’s announcement of large-v3-turbo reports FLEURS word error rates of 5.3% for Russian, 4.4% for English, 5.4% for Polish and 5.8% for German, with beam search of width 5. On the FLEURS test splits for those languages, my service measured 5.44% for Russian, 5.03% for English after the fix described below, 5.85% for Polish and 5.17% for German. It used greedy decoding, automatic language detection, a weight-only int8 OpenVINO export, a speech detector in front of the model and a basic text normalizer. Published German FLEURS scores also vary across evaluation pipelines: the Open ASR Leaderboard listed 3.67% for turbo in October 2026 (pinned results).

First, I had to fix the reference transcripts

#

My first useful baseline combined the Russian test split of FLEURS with seven clips cut from one Piper text-to-speech (TTS) rendering of a technical script. I proofread the seven clips' references by hand against the audio; accepting the initial machine transcripts unchanged would have let the recognizer grade itself.

In two Russian TSV rows the transcript opens with a quotation mark, and my scorer’s CSV parser treated it as quoting: it joined the transcript and character-token columns and counted the appended spelling tokens as 314 words the recognizer had deleted. Taking those two references from the raw transcript column instead moved Russian-normalized word error rate from 6.39% to 4.46%, without changing the model.

Those scores use a Russian normalizer with explicit punctuation, number and ё handling. Letting Whisper detect the language instead of forcing Russian gave 4.51% WER on the same public set. Later multilingual tables use basic normalization, my scorer’s own normalizer modeled on OpenAI’s BasicTextNormalizer from release v20250625. It lowercases text, drops spans in parentheses and square brackets, turns Unicode marks, symbols and punctuation into spaces and collapses whitespace. It is not identical to the upstream class: Unicode normalization and bracket handling differ. Neither folds ё nor spells out numbers, so these percentages are not directly comparable with this table.

The synthetic sample became the release gate’s regression check for technical terms, although some of its scored errors are TTS pronunciation artifacts. Both datasets subsequently influenced development; rerunning them could catch regressions, but could no longer confirm quality on untouched data.

Several attractive tuning ideas did not survive these checks: fp16 weights showed no measurable gain over int8, a Russian fine-tune was worse on FLEURS, and global hotwords made some long recordings degenerate.

FLEURS in four languages: quiet speech never reached Whisper

#

The next control used the complete Russian, English, Polish and German FLEURS test sets: 3,042 clips and 9.48 hours. Each reference was the dataset’s normalized “transcription” field. In eight Russian, English and German rows the transcript opens with a quotation mark, so the same parser fell back to their “raw_transcription” field; after basic normalization each of those references equals its own “transcription” field. Each clip ran once through automatic language detection, with no retry, language hint, prompt or hotwords. Language ID is the share of clips at least three seconds long whose detected language was correct; a clip with no detected language counts as a miss.

All 3,042 requests succeeded, and English still failed the limits I had fixed before the run: at most 8% basic WER, at least 99% language ID and no empty results in every language. A four-language average would have hidden it. All 24 English misses returned in about 0.05-0.07 seconds, before decoding, with no detected language. Voice activity detection (VAD), the check that decides whether a recording contains speech before Whisper runs, had classified quiet speech as something it could discard. My gateway used Google’s WebRTC VAD for that check.

A separate diagnostic Pod used the same image and model with the gateway’s VAD admission threshold, STT_VAD_SPEECH_RATIO_MIN, set to 0 instead of 0.40, so the speech check could no longer reject a clip. All 24 clips produced English text, at 5.07% WER on that subset. Replacing only those rows would have brought full-set English WER to 5.03%.

Removing VAD was not a solution: the silence, room-tone and noise controls then produced invented text. I needed to admit quiet speech while preserving the reason the guard existed.

My first fix normalized the detector’s analysis copy by its whole-file average (RMS) level. An adversarial review broke it with one full-scale sample or a 30 ms click: the spike changed the global level enough to disable normalization, and all 24 recordings fell below the threshold again.

The replacement used the 90th percentile of 30 ms frame levels, a minimum useful signal level and a 4x gain cap. Only the detector’s copy changed; Whisper still received the original audio samples.

flowchart TB
A["Original recording"] --> B["Level boost, at most 4x,
on an analysis copy"]
B --> C{"WebRTC VAD
admits
speech?"}
C -->|yes| D["Whisper
decodes the
original audio"]
C -->|no| E["Empty
result"]

All 72 quiet-speech variants passed: the 24 originals, each with a spike, and each with a click. The silence, room-tone and noise controls stayed empty. A complete FLEURS rerun then delivered the predicted English result: 5.03% WER, 100% language ID and no empties. The other three languages reproduced their earlier scores.

That fixed the observed FLEURS failure. Fresh speech would decide what else I had missed.

Testing on fresh Common Voice speech

#

For the next comparison I used Common Voice releases from June 2026. I retained aggregate results and release identities; raw recordings, speaker identifiers and row-level transcripts stayed private.

A rule fixed before inference excluded references that became empty after normalization. It removed 38 English rows and none from the other sets. An empty reference only means there was nothing to score against; those rows were not checked as non-speech. That left 3,274 clips per image, or 6,548 requests in the paired comparison.

Both images received the same clips once, in a fixed interleaved order, without hints or retries. WER changes below are candidate minus baseline, in percentage points, so a negative change is an improvement. Each speaker-level 95% interval came from 2,000 paired bootstrap resamples of whole speakers with the fixed seed 20260828, two-sided, between the 2.5th and 97.5th percentiles: 370 Russian clips from 11 people are not 370 independent samples. With only 11 Russian speakers, the Russian interval was supporting evidence only. The 65 German clips from 16 speakers had neither a WER-change gate nor an interval, and their remaining checks were provisional. Polish read speech also had a sentence-level check, because several people can read the same sentence.

Three limits stayed visible. The German sample was small, Polish was read speech, and the Whisper turbo model card states only the scale of its training data, not its sources, so I could not exclude overlap with public recordings. A paired comparison can measure a change on these clips without proving performance on arbitrary unseen speech. The remaining 7,587 Polish test rows stayed sealed, split before any output into two halves for later candidates. Both halves held only clips no earlier comparison had used, but the split was random by clip; it was not built to exclude speakers or sentences already heard. The first subset had taken one clip from each of 2,500 speakers, and their other clips stayed in the remainder.

Each comparison changed one mechanism against the image then in production. Every arm used greedy decoding, automatic language detection and no prompt or hotwords:

In the first pair, both images contained the same 296 Debian packages. No baseline used Silero or the fallback, and Silero was off in both arms of the fallback comparison. Each candidate enabled one mechanism through its own setting. My retained records do not tie an image digest to those two runs, so I name each candidate by its configuration.

The first rejection exposed older empty results

#

The hardened VAD caused no new empty results. It recovered one additional Polish clip. On clips both images admitted, text matched after the scorer’s basic normalization and language labels matched.

The frozen result, the verdict of the gates I had written down before the run, was still REJECT.

My gate demanded zero empty results with nonempty references. English retained four, Russian one and Polish 53, all shared with the baseline. Their individual causes were not all established; counting them did not make them one diagnosed defect.

I accepted the narrower release claim: the change must drop no clip that the baseline transcribed. That amendment was made after seeing the output. The original rejection stayed in the record.

The absolute error rates were about twice my FLEURS results. Language ID here covers clips at least three seconds long:

• English Spontaneous: 10.43% basic WER and 92.6% language ID.
• Russian Spontaneous: 11.50% WER and 98.4% language ID.
• Polish Scripted subset: 11.06% WER, against the baseline’s 11.08%.
• German: no empty results, but only 65 clips.

Good FLEURS language identification had not predicted spontaneous English. The Polish read-speech score also missed my proposed 8% target.

The result was useful enough for my deployment, with incomplete natural-speech coverage. It was not confirmation of all the original targets.

A second detector recovered speech, including difficult speech

#

The shared empties motivated a second opinion from Silero VAD. WebRTC stayed the primary detector. A pinned Silero v4 ONNX model ran on CPU through the existing OpenVINO runtime only when WebRTC rejected a recording.

flowchart TB
A{"WebRTC VAD
admits
speech?"} -->|yes| D["Whisper
decodes the
original audio"]
A -->|no| B{"Silero VAD
admits
speech?"}
B -->|yes| D
B -->|no| C["Empty
result"]

This widened admission without touching recordings the first detector already accepted. Today the final service and the live preview both use this order: WebRTC first, Silero only after a rejection.

The first sealed Polish half contained 3,794 clips. On them the baseline returned 53 empty results, a different set from the 53 Polish clips above. The candidate recovered 41 of them, with 39 in the target language. All 3,741 jointly admitted clips retained equal normalized text and language labels. Corpus WER improved from 10.23% to 9.83%.

Two recovered clips were short, mumbled Polish with three-word references. Whisper rendered them in the wrong language, still inside my allowed set of Russian, English, Polish and German. One broke my rule that every newly admitted clip of at least three seconds must be detected in its target language. The other counted against my catastrophic-error gate, which treats a row as catastrophic at a basic WER of at least 80% with an edit count reaching the smaller of eight or the reference word count. Its empty baseline already met that threshold, and the wrong-language text made the row worse.

Raising the Silero threshold could not separate those recordings from useful recoveries: one wrong decode had a speech score of 0.946, above a correct recovery at 0.888. Both contained speech. The detector had done its job; the recognizer struggled with what it had newly been allowed to hear.

The frozen verdict remained REJECT. After seeing the output, I accepted degradation only for newly admitted recordings whose baseline was empty, capped at 5% of that newly admitted set per corpus; the measured share was two of 41. The exception never covered a clip the baseline had already transcribed.

A language fallback improved WER and still made bad choices

#

My everyday languages are Russian, English, Polish and German. A stray detection outside that set could turn a whole transcript into another language: on FLEURS, one Russian clip came back entirely in Ukrainian. So I tested a bounded language fallback on the second sealed Polish half, 3,793 clips.

The rules left in-set detections alone. Out-of-set detections received one forced-Russian decode if another pass fitted the request deadline; otherwise the original transcript was returned. In the shipped combination, a clip admitted only by Silero stays empty if Whisper detects a language outside the set; Silero was off in this comparison.

Russian was my dominant workload. It was also the wrong fallback for Polish speech.

The candidate changed 179 jointly admitted transcripts, every one associated with an out-of-set baseline detection, most often Icelandic (115 clips). No in-set detection changed, and the empty sets matched. The full-half WER improved from 9.87% to 9.40%, a change of -0.47 percentage points.

The tail, the worst individual rows, went the other way. There were 74 newly catastrophic or worsened catastrophic rows. The upper end of the speaker-level 95% interval for the WER change was +0.43 points, above the +0.3 points I had allowed. Only the 179 re-decoded rows changed error counts; the interval still resampled the whole half. My retained summary records that upper end and that the half spans more than a thousand speakers, but not the lower end or the exact speaker count. The frozen verdict was REJECT again.

Across the 179 affected recordings, aggregate WER fell from 121% to 106%. WER can exceed 100% when insertions add enough words, and neither score describes useful transcription. Not every affected baseline transcript was catastrophic, though: 56 of the 179 were not.

After seeing the output, I accepted the interval and tail violations only inside that declared re-decode set.

This Polish evaluation did not show a benefit for my dominant Russian workload. The only Russian evidence was development data: on FLEURS the fallback turned the one known Ukrainian detection back into Russian, and it also forced one Polish clip detected as Icelandic into Russian. I shipped it as a trade-off for my own workload, with the failed gates and the regressions on the record. After a fallback the service reports the forced language, so language ID on those rows describes the forced decode, not Whisper’s first detection.

The proposed successor would select the best allowed language from the model’s detection distribution instead of always forcing Russian. That design still needs fresh data; it is not the mechanism these results measured.

What actually shipped

#

The three changes side by side:

The later two mechanisms shipped together in rel-526534cf after image and isolated performance checks. Each met fresh data once, alone and as read Polish speech; the pair never did. Their promotion rested on my recorded risk acceptances, not on a retrospectively declared confirmation.

What comes next

#

The remaining gaps are practical: short mumbled speech and amplified noise that the primary WebRTC detector can admit. The negative fixtures catch known failures; they do not establish a statistical false-accept rate on natural non-speech.

The evaluation is not continuous integration yet. Its scoring code is versioned and unit-tested in the service repository, but no GitLab job runs the natural-speech suite; I ran it by hand for the three changes above. Next on my list:

• a separate, manually triggered GitLab job with one frozen corpus, explicit baseline and candidate images, locked scoring dependencies and a machine-readable verdict;
• a language fallback that picks the best allowed language from Whisper’s own detection distribution instead of always forcing Russian;
• fresh Polish data. Between June and September the Polish scripted dataset grew by only 60 clips in total, from 148,736 to 148,796 (release statistics), and there is still no Polish spontaneous set.

Claude Code built the confirmation tooling, implemented the Silero second opinion and the language fallback, investigated the results, drafted the original evaluation article and restructured this part around testing. Codex reworked the article for this four-part series, checked it against the retained records and gathered the CI and public-benchmark sources for this version.

These checks describe recordings that reached the service. The final part follows the audio before that boundary: an Open WebUI microphone error whose cause stayed unknown, the two capture defects the browser tests reproduced, and what a USB reconnection still did not explain.

My speech-to-text evaluation checklist

#

The three changes above went through this procedure. Claude Code helped write and review the plans and audited the first one before I froze it; I approved every freeze and decided every post-output amendment.

1. Check the references. Use human-checked transcripts and confirm that the scorer reads every row exactly as written. Exclude references that are empty after normalization from speech WER, under a rule fixed before inference, and test verified non-speech recordings separately for invented text.
2. Freeze before output. Record the protocol, corpus hashes, image digests, decoding settings and gates before any new inference; a later edit becomes a new version.
3. Give every corpus a role. FLEURS and synthetic clips are for development and regression. Common Voice data is sealed confirmation: each comparison opens a whole set or a seeded selection once, and whatever it opens is retired.
4. Compare pairs. Send the baseline and the candidate the same clips once, in a seeded interleaved order, with no prompt and no per-clip retries, and resample whole speakers for intervals, plus whole sentences as a second check when many people read the same text.
5. Score languages separately and read the tails. An average can improve while the worst rows get worse.
6. Keep the verdict. Never move a threshold after seeing output; an interval that reaches its limit fails that gate. If a rejected candidate ships, record the risk acceptance beside the frozen REJECT.

Hashes and frozen files detect drift and accidental reuse. They cannot stop a deliberate change by someone who controls the runner and every file, and that someone is me, which is why each amendment stays on the record.