Speech-to-Text on Intel Iris Xe, Part 2: Building Images I Could Ship

My first Intel speech-to-text image still carried a CUDA runtime. None of the inference used it. Removing that baggage was satisfying, but a smaller image did not solve the harder problem: knowing that the image I tested was the image actually serving my dictation.

At one point, all four running Pods were serving an older image than the one I had promoted, and a fix that had passed its gates never served a request. Their imagePullPolicy was Always, which pulls only when a container starts; it restarts nothing. An updated registry tag had not changed their already-running processes.

This is Part 2 of 4 in my speech-to-text series. Part 1 followed the hardware and serving path. Here I open the service itself: why it is a gateway, how a request reaches the GPU and where its time goes, what I built and what I reused, and how its image is built, tested and released. Part 3 examines recognition quality, and Part 4 follows audio from the browser microphone to the finished prompt.

Just here for the procedure? Jump to the release checklist.

Why it is a gateway

#

The speech-to-text (STT) gateway is the shared entry point for final Whisper transcriptions of complete recordings; a separate live-preview service shows draft text while I am still speaking. Open WebUI, my chat front end, Nextcloud and my agents never load Whisper themselves. They send a recording to an OpenAI-compatible transcription endpoint, and the gateway owns everything from admission to the GPU.

Keeping these rules in one service means a new consumer inherits them instead of reimplementing them. It also gives each rule one owner: when a long recording is rejected, the gateway decided it, and its metrics say why.

How a request reaches the GPU

#

A final Whisper transcription travels through these components:

flowchart TB
A["Open WebUI, Nextcloud
or an agent"] --> B["LiteLLM
stt or whisper-1 alias"]
B --> C["Envoy least-request,
configured by Cilium"]
C --> D["Gateway admission:
size, duration, queue"]
D --> E["Worker process:
ffmpeg audio decode,
voice-activity check"]
E --> F["OpenVINO Whisper
on the Iris Xe iGPU"]
F --> G["JSON transcript
back to the caller"]

LiteLLM, my model proxy, maps the stt and whisper-1 model names to the gateway. An Envoy proxy with least-request load balancing, configured through Cilium, favors replicas with fewer active requests. The gateway itself is a FastAPI application on Uvicorn. Each Pod runs one transcription at a time. When a benchmark shared a serving Pod’s card in an earlier test, the serving Pod slowed to less than half its speed and the two together processed slightly less audio than it had alone. That result is why I kept one transcription slot per Pod. Its CPU cores go to audio decoding, voice activity detection and HTTP, while the Whisper decode runs on the integrated GPU (iGPU).

Where a request spends its time

#

The gateway records its own histograms. In a seven-day window, it recorded about 100 successful client-labelled transcriptions, and its histograms gave these estimates:

The waiting row covers requests that acquired a worker, including any wait for upload admission; requests rejected while waiting are absent. The processing and recording rows cover successful responses, and backend processing includes cleanup. The bracketed ranges are the histogram buckets that hold each estimate, and the estimates assume requests spread evenly inside them. The sample is small, and the rows do not pair up request by request, so I do not divide them into a speed. For these observed requests, backend processing, not waiting, took nearly all of the measured time.

The queue limits from Part 1, four waiting requests per worker and at most 120 seconds of waiting, matter when a long recording occupies a worker; least-request routing reduces the risk of short requests queuing behind it.

Other hops need separate measurements. In my earlier routing comparison, a 25-second file ran at 11.88x realtime (seconds of audio per second of processing) through Envoy and 11.42x directly. The comparison did not show an obvious slowdown through the proxy, but it did not isolate or quantify Envoy’s overhead. In the same routing tests, a short clip took about 2.1 seconds when a worker was free, about as long as that 25-second file. Realtime multiples therefore say little about a few seconds of dictation. I have not measured LiteLLM or the upload separately, so I do not claim a number for them.

What I built and what I reused

#

Most of the stack is upstream software. I built the service API, its resource controls, the worker lifecycle, observability, artifact checks, integration patches and release gates. OpenVINO GenAI runs Whisper, Optimum Intel exports its existing weights, and upstream tools provide voice-activity detection, GPU compilation, routing, builds and scans.

What I built:

• the gateway API, admission and duration probe;
• worker-process supervision, warmup, the repetition guard and metrics;
• the Dockerfile, its test target and the image build;
• the model export job and its pinned environment;
• the hardware job and the HTTP roundtrip checks;
• CI templates for tests, scans and digest promotion;
• the Open WebUI patches and the GitOps manifests.

What I reused:

• Whisper large-v3-turbo and Qwen3-ASR-0.6B weights, OpenVINO and OpenVINO GenAI;
• Debian 13 with its Python 3.13, uv, FastAPI, Uvicorn and ffmpeg;
• Optimum Intel for the export;
• Google’s WebRTC voice-activity detector, with the Silero VAD model as its second opinion and to skip a speech-free start of 30 seconds or more;
• Intel’s compute runtime and graphics compiler packages;
• GitLab CI, BuildKit, Harbor, Trivy and crane;
• Open WebUI, LiteLLM, Envoy through Cilium and Argo CD;
• tini as the container’s init process.

Why I build my own images

#

Ready-made speech servers did not fit this service. Speaches, WhisperLiveKit and RealtimeSTT had no Intel GPU path when I evaluated them, and faster-whisper had no Intel GPU backend. I also saw no clear benefit in switching to whisper.cpp from the OpenVINO path I already had. In that evaluation, I treated WhisperLive as a streaming option with an OpenVINO backend and kept it on a watch list.

OpenVINO GenAI runs Whisper well on the Iris Xe, but it is a library, not a service: the HTTP API, admission, queue, voice-activity gate, protection against repetition loops and metrics are mine to build. The image also has to carry the Intel GPU driver’s userspace, in versions I choose and verify.

Open WebUI needs a second kind of customization: coordinated frontend and backend changes for an authoritative server-recognition route, replaceable live-preview text, final transcription after Stop, and the capture lifecycle that Part 4 examines. Together the two kinds produce the four artifacts below; describing them as one “Whisper Docker image” would hide half the integration.

The small Silero VAD model ships inside the gateway image, while the larger recognition models have their own provisioning path. The preview image is built on top of a promoted gateway image: it inherits the gateway’s Intel driver and voice-activity code, installs its own hash-locked OpenVINO packages, and moves to a new base only after a gateway release.

GitLab builds the images, Harbor keeps them

#

I build these images in GitLab CI on self-hosted runners in my Kubernetes cluster; the Kubernetes executor gives each CI job its own Pod. A build job starts a short-lived, rootless BuildKit daemon, builds with buildctl, pushes the image to Harbor, a container registry that runs in my cluster, and removes the daemon when it ends. The common build, scan and promotion steps live in a shared GitLab CI templates repository; each image project adds its own checks.

Harbor sits between the pipelines and Kubernetes. The cluster pulls released images from it, the builds pull their upstream base images through its proxy-cache projects for Docker Hub and GitHub’s registry, and it keeps the builder’s layer cache between jobs. Debian packages come through my own apt cache, apart from the CA certificates a build first fetches from Debian so that apt can trust the cache.

The gateway’s images move through four kinds of tags in Harbor:

In the release tags, <sha> is the short Git commit hash. candidate exists only because my build template pushes under a tag name. The checks address the build’s digest, so another pipeline moving that tag cannot change what a run tests.

A retention rule always keeps rel- tags and an immutability rule stops them from moving, while candidates stay eligible for cleanup. Running Pods still prove nothing about a replacement Pod, so my release checks confirm that the deployed and rollback digests can be pulled.

Removing CUDA was the easy size win

#

Replacing the CUDA Ubuntu base with an Intel-only Ubuntu image took the gateway from 2.59 GB to about 955 MB.

I then moved the gateway to Debian 13 and installed separately verified Intel driver packages. That build came out at about 970 MB, roughly the same size. These are rounded sizes from the records of those builds, not fresh measurements of today’s images.

The Debian move was therefore not another size reduction. It let the runtime use a small Debian base while I select and hash-check the Intel GPU userspace packages myself.

The image owns the userspace half of the GPU path described in Part 1: the OpenCL runtime that Intel’s OpenVINO GPU setup guide requires. I left out the Level Zero packages (Intel’s low-level oneAPI GPU interface) that the guide’s Ubuntu command also installs, because the OpenVINO GPU plugin uses OpenCL on this hardware.

My build downloads selected packages from Intel’s compute runtime (NEO) and graphics compiler (IGC) releases, currently NEO 26.35.39758.10 and IGC 2.41.5, in a separate stage and checks their SHA-256 hashes before installation. Intel builds those .deb files for Ubuntu, so on Debian 13 my hardware job, not a vendor support matrix, qualifies them.

The gateway runs as an unprivileged user. Its image creates the writable directories needed for temporary audio and OpenVINO compilation. Creating them in the image fixed an easy mistake to miss: a writable Kubernetes volume had masked a missing cache directory in the image itself. The image should not depend on an accidental mount to survive startup.

The Dockerfile, abridged

#

This is the STT gateway’s Dockerfile cut down to its stage structure. It is a schematic, not a runnable file: comments stand in for omitted setup and checks, public upstream names replace the Harbor and apt mirrors my builds pull through, and the uv image digest is left out.

# syntax=docker/dockerfile:1.19
# Public upstream names (mirror in my setup).
ARG BASE_IMAGE=debian:trixie-slim
ARG UV_IMAGE=ghcr.io/astral-sh/uv:0.12.23
ARG APT_MIRROR=http://deb.debian.org/

FROM ${UV_IMAGE} AS uv-bin

FROM ${BASE_IMAGE} AS base
# ... CA trust, apt upgrade, Python 3.13 and tini, app user and group 10001,
# app-owned /app, /models/stt, /tmp/stt-gateway and /var/cache/openvino
ENV CI=true PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 \
    PATH="/venv/bin:${PATH}" HOME=/tmp/stt-gateway \
    TMPDIR=/tmp/stt-gateway XDG_CACHE_HOME=/tmp/stt-gateway/.cache
WORKDIR /app

FROM base AS intel-driver
WORKDIR /intel
COPY intel-gpu-driver.sha256 ./
# ... install curl; turn the pin file into "sha256 url" lines in ./pins
RUN set -eux; \
    : > SHA256SUMS; \
    while read -r sha url || [ -n "${sha}" ]; do \
      file="$(basename "${url}")"; \
      curl -fsSL --retry 5 --retry-all-errors --max-time 300 \
        -o "${file}" "${url}" || exit 1; \
      printf '%s  %s\n' "${sha}" "${file}" >> SHA256SUMS || exit 1; \
    done < pins; \
    test "$(grep -c '' SHA256SUMS)" -eq "$(grep -c '' pins)"; \
    sha256sum -c --strict SHA256SUMS

FROM base AS silero-vad
WORKDIR /silero
COPY silero-vad.sha256 ./
# ... the same download loop, sha256sum -c --strict, and an exact size check

FROM base AS builder
COPY --from=uv-bin /uv /uvx /bin/
ENV UV_COMPILE_BYTECODE=1 UV_LINK_MODE=copy \
    UV_PROJECT_ENVIRONMENT=/venv UV_PYTHON_DOWNLOADS=never
# ... install python3.13-venv; cache mounts omitted here
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-dev --no-install-project --python /usr/bin/python3.13
COPY README.md LICENSE ./
COPY app ./app
RUN uv sync --frozen --no-dev --no-editable --python /usr/bin/python3.13

FROM builder AS dev
# ... ffmpeg, uv sync --frozen --extra dev, test inputs

FROM dev AS test
# ... a verified Silero copy; ruff, format check, mypy and pytest in one && chain;
# its status goes to /reports/exit instead of failing the build, and pytest
# writes /reports/report.xml when it runs

FROM scratch AS test-report
COPY --from=test /reports /

FROM base AS runtime
# ... mount /intel from intel-driver; install ffmpeg, the OpenCL loader and the
# verified .debs; purge selected Mesa and LLVM packages in the same layer
# ... as root: the OpenCL driver's libraries resolve and ffmpeg decodes a tone
COPY --from=builder --chown=app:app /venv /venv
RUN install -d -m 0755 /opt/stt-vad
COPY --from=silero-vad --chmod=0444 /silero/silero_vad.onnx \
     /opt/stt-vad/silero_vad_v4.onnx
USER 10001:10001
# ... as UID 10001: app and OpenVINO imports, a VAD read and a tone decoded
# through the app's ffmpeg launcher; an HTTP /healthz HEALTHCHECK
EXPOSE 8080
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["/venv/bin/uvicorn", "app.main:create_app", "--factory", \
     "--host", "0.0.0.0", "--port", "8080", "--proxy-headers", \
     "--ws", "websockets", "--ws-max-size", "32768", \
     "--ws-max-queue", "4", "--ws-per-message-deflate", "false"]

What this structure buys me:

• Downloads live in their own stages. Only the fetch stages install curl, and they verify every Intel package and the Silero model against a pinned SHA-256 before anything uses them. The runtime stage receives the verified files, not the tools that fetched them.
• Every download step stops the build on its own. set -e does not fire inside an && chain, so a loop of curl ... && printf ... lets a failed download that is not the last one drop out of the manifest unnoticed. Each step ends with || exit 1, and the manifest must name every pin before sha256sum checks it.
• Dependencies come before code. uv sync --frozen installs the locked dependencies before the application source is copied, so a code change does not invalidate that layer, and the lockfile cannot drift during a build. --frozen trusts uv.lock as written; --locked would also fail a lock that no longer matches pyproject.toml.
• The test branch exports its verdict. The test stage captures the checks’ exit status so failure evidence survives the build step. The checks stop at the first failure; test-report exports the status and any available JUnit report from an empty scratch image. CI reads the status and requires the report.
• The runtime starts from base. It does not inherit the builder, dev or test stages. It receives the virtual environment, the verified driver packages and the model file, then runs as UID 10001 under tini, which forwards signals to Uvicorn and reaps child processes.
• Writable paths belong to the app. Home, temporary files and the XDG cache point into /tmp/stt-gateway, and the OpenVINO compile cache uses /var/cache/openvino; the image creates both, owned by the app user.
• The runtime checks itself at build time. As root, the final stage checks that the Intel OpenCL driver’s libraries resolve and that ffmpeg decodes a generated tone. As the serving user, it then imports the application and OpenVINO, reads the Silero model and decodes a tone through the app’s own ffmpeg launcher. None of this runs Whisper or proves execution on a physical Intel GPU; the hardware job in the pipeline does that.

The runtime is not minimal in every sense. It carries Intel’s graphics compiler, because the GPU path compiles kernels while it runs, and the Dockerfile alone does not make the root filesystem read-only.

A lockfile does not freeze every input

#

The STT gateway currently uses Debian’s Python 3.13 and a frozen uv lockfile (uv.lock). The serving dependencies include OpenVINO 2026.4.1 with the matching OpenVINO GenAI and OpenVINO Tokenizers packages.

The word “frozen” applies to a particular input, not the whole build:

Floating system packages let a rebuild collect available distribution fixes. They also mean two builds from the same source commit can produce different bytes, so Git alone cannot reconstruct a release. Each release needs its own artifact identity: the image digest, a SHA-256 content hash that changes whenever the image content does.

The Harbor and apt caches described above help with download and dependency work, while scheduled builds skip the layer cache kept in Harbor, so a nominal rebuild does not quietly retain all the old package layers. Those caches improve the build path; the released digest identifies the image I deploy, while the model files and the GitOps values that run it are identified separately, by the model revision with its checksums and by a Git revision.

Model export and serving are different environments

#

The serving container does not need the whole model-export toolchain. Optimum Intel, which exports Hugging Face models to OpenVINO, belongs to my Whisper export environment, not the STT gateway’s runtime lockfile.

That exporter currently uses a separate pinned OpenVINO development image and a Python 3.12 dependency lock with OpenVINO 2026.2.1 and Optimum Intel 2.0.0, while serving runs OpenVINO 2026.4.1, and Optimum remains export-only. The preview exporter has an environment of its own. Giving the serving runtime and both exporters one version label would hide the compatibility boundary I actually need to test.

I keep the model revision, file checksums and export settings with the artifact. Provisioning verifies the expected files before the worker uses them. Each Pod then has its own OpenVINO compile cache; a successful download is not the same thing as a warmed model ready for its first transcription.

flowchart TB
A["Pinned model revision
and export environment"] --> B[Verified model files]
B --> C["Serving Pod loads them
with its Intel driver"]
C --> D["OpenVINO compiles
on the allocated GPU"]
D --> E["Warmup decodes, then
real transcription"]

This is also why an upgrade needs more than importing a Python package successfully. The files must still load, compile on the intended device and produce acceptable output through the application API.

Test the candidate, then promote its existing digest

#

The STT gateway’s release pipeline first checks the source through its test target, then builds the runtime candidate. Image scans, hardware tests and API checks use that candidate’s digest. Promotion then uses crane to point the release tags in Harbor at that same digest: crane tag adds a tag to the existing manifest in the registry without pulling or rebuilding the image, and crane digest reads each tag back; the job fails unless it names the tested digest. The permanent rel- tag goes on before stable moves.

flowchart TB
A["Source and
locked inputs"] --> B["GitLab CI: unit, lint
and type checks"]
B --> C["Build the candidate,
push it to Harbor"]
C --> D["Image scans, hardware
and API checks
on its digest"]
D --> E["crane tags that digest
in Harbor, then reads
each tag back"]
E --> F["Digest into GitOps,
rollout, Pod digests
matched, real request"]

The last step answers the opening incident. Moving a tag in the registry changes nothing for Pods that are already running. The release is complete only when the tested digest is written into my GitOps values: that changes the Deployment, so Kubernetes rolls the Pods, and Argo CD’s sync status now covers the digest itself. Synced still describes manifests, not running processes, so I match the running Pods’ image digests to the promoted one; a functional transcription through the real route then closes the release.

CI reads the verdict that the test-report stage exports and rejects failed or empty runs. That enforcement matters because BuildKit builds only the stages a target depends on, so a Docker stage named test does not by itself make the runtime build pass it.

The gateway pipeline’s hardware job runs the candidate image on one Intel GPU allocation, which can land on a card that is already serving dictation. Its current contract includes a speed fixture, nonempty Russian output, repetition and segment checks, plus seven checksum-verified accuracy clips containing 338 normalized reference words. Those clips are cut from one Piper text-to-speech recording of a scripted technical text, so they test technical terms in synthetic speech, not natural dictation. That small set is a release regression check; Part 3 covers the broader accuracy evaluation.

The gateway pipeline also runs a separate HTTP roundtrip: my text-to-speech (TTS) gateway synthesizes each phrase during the run, and the candidate transcribes it. Four ordinary phrases gate accuracy and speed. If a gating phrase misses its word-error threshold, the procedure may synthesize it once more and keep the better result. That makes it a bounded integration check with a known retry policy, not an unbiased measure of natural dictation.

Neither job opens my physical headset in the Brave browser. The browser and audio-source checks remain a separate part of acceptance.

What the CVE scan actually decides

#

I scan the exact candidate digest with Trivy. The pipeline separates two kinds of findings because I do not want a decision about package risk to accidentally permit a high-severity secret in an image.

First, the secret scan blocks on HIGH and CRITICAL findings; Trivy rates many of its built-in secret rules MEDIUM or LOW, and those pass this gate. Then the vulnerability scan covers OS and language packages, selecting HIGH and CRITICAL vulnerabilities that already have a fix. Those findings are advisory under my current policy. A scan that fails to finish counts as a failure, not a clean result.

This policy does not establish that an image is free of vulnerabilities. Scheduled rebuilds help collect fixes once upstream provides them. They do not replace reading the findings or deciding which risks are acceptable. Package findings used to block promotion; they become blocking again when a promoted image gains an external consumer, or when I change this policy explicitly.

How the images changed

#

• First Intel image: Ubuntu with a CUDA runtime that no inference used, 2.59 GB.
• Intel-only Ubuntu: CUDA removed, about 955 MB.
• Debian 13: about the same size, with hash-checked Intel driver packages.
• Current release path: one tested candidate digest, scanned and checked on the GPU, then promoted unchanged with crane, written to Git and matched against the running Pods.

The smaller Intel image was a useful first step. The lasting improvement was being able to follow an artifact from its inputs to a running transcription. Part 3 asks what the resulting words were worth, including the cases where every request succeeded and the transcript was still empty or wrong.

Codex drafted this part, checked it against my repositories and saved records, abridged the Dockerfile and traced the GitLab and Harbor sources. Claude Code wrote the sections on the gateway, its request path and timing, reuse, the case for my own images, the GitLab and Harbor build path and the Dockerfile’s structure, reviewed the rest and made some of the corrections.

My STT release checklist

#

This is the procedure I carry forward from the stale-Pod incident and the later hardware checks:

1. Identify the inputs. Record source, dependency locks, Intel driver files, model revision and export environment. Keep exporter and serving versions distinct.
2. Read the test outcome. Require a real, nonempty successful run; a test-stage name or exported report alone is insufficient.
3. Build a candidate once. Capture its digest; the scans and hardware checks use it.
4. Read both scans. Keep their output. Separate secret findings, package risk and a scanner that never completed. Preserve any explicit risk acceptance.
5. Exercise the candidate on the intended GPU. Check device execution, output, bounded accuracy and the actual API route. Record the fixture and retry policy.
6. Promote that digest. Tag it in the registry without rebuilding (I use crane tag), read each tag back (crane digest) and require the tested digest, then write the digest into the GitOps values.
7. Roll out and inspect the running process. Confirm the previous release’s digest is still available for rollback, match the running Pod’s image digest to the promoted one, check the Service endpoints, then send a functional transcription.
8. Check the user-facing boundary that changed. Browser capture changes need browser evidence.