Gemini 3.5 Transcribe needs an action-commit gate, not a WER victory lap

Google has released Gemini 3.5 Transcribe in preview for recorded files and live streams. The launch page emphasizes more than 85 languages, code-switching, speaker labels, timestamps, and improved handling of names, numbers, and other alphanumeric content. Those are useful capabilities. They also invite a dangerous product shortcut: treating a polished transcript as permission to act.

Imagine a support call where a user says, “Move the renewal to May 14, but do not charge the new card yet.” A provisional transcript drops the negation, assigns the sentence to the account owner instead of a colleague, and reaches a billing tool before the text is revised. The final transcript can be excellent while the product outcome is still wrong.

The immediate change for a small team is to separate recognition from authority. A transcript may be good enough for search or notes before it is good enough to update a CRM, create a ticket, send a message, or authorize money movement. Promotion should depend on the exact spoken job, the stability of critical fields, the speaker’s authority, the transcript’s finality, and a reversible effect path.

This field note provides a proposed 24-case voice-to-action release gate. We did not run Gemini 3.5 Transcribe, reproduce Google’s launch results, or compare vendors for this article. Every fixture, threshold, and result field below is a template for a team to run against its own product.

What launched, and what the launch does not establish

Google’s Gemini 3.5 Transcribe model documentation lists two preview surfaces: gemini-3.5-transcribe for audio files and gemini-3.5-transcribe-live for live streaming. Both support automatic detection across more than 85 languages and mid-session code-switching. The documented surfaces are not identical: file processing supports word-level timestamps, while live streaming does not; requesting timestamps can reduce transcription accuracy; live diarization is unsupported; and file mode supports up to eight speakers while attribution for three or more is experimental.

The DeepMind announcement says the system is optimized for multilingual transcription and reports improvements on Google’s chosen evaluations. Those figures are first-party release evidence. They do not predict performance on your microphones, customer accents, product names, call mix, silence patterns, network path, or action schema.

The correct bounded conclusion is that a new candidate exists with a broad and product-relevant surface. It is not that “85+ languages” means equal quality in 85 languages, that speaker labels prove identity, or that a lower average error rate makes downstream automation safe.

A transcript has at least four states

Teams often store one transcript string and overwrite it as more audio arrives. That representation erases the distinction that matters most for automation.

Use four states instead:

1. Provisional: partial text emitted while speech is still arriving. It may be revised, extended, or withdrawn.
2. Finalized: the recognizer has closed a segment. Final means stable within the service protocol, not necessarily correct.
3. Verified: a product rule, deterministic parser, or human has checked the fields that matter for the proposed action.
4. Authorized: the product has established that the speaker and current user may request this specific effect.

Treat the states as cumulative evidence, not four interchangeable labels. Only a segment that is finalized, verified, and authorized can support an irreversible or externally visible effect. Finalization answers “will this text change again?” Verification answers “did we recover the required meaning?” Authorization answers “may this person cause this action?” These are separate questions.

A speaker label such as Speaker 2 is diarization, not authentication. A timestamp is location evidence, not semantic confidence. A fluent sentence is not evidence that the audio contained the sentence. The release gate must preserve these type boundaries all the way to the tool call.

Why average word error rate is the wrong release gate

Word error rate (WER) counts substitutions, deletions, and insertions relative to a reference. It remains a useful baseline; the NIST OpenSAT evaluation plan uses it for automatic speech recognition evaluation. But WER weights every reference word equally, while products do not.

Changing “Tuesday” to “Thursday,” $15 to $50, or “do” to “do not” can dominate an otherwise accurate transcript. A long meeting can achieve a low WER while corrupting its only decision. A short speaker turn such as “yes” can disappear without moving a duration-weighted metric much.

Speaker metrics have the same aggregation problem. The original Balanced Error Rate paper argues that duration-based diarization error can underweight short segments and less-talkative speakers. That matters when the quiet participant is the approver, the objector, or the person whose account would change.

Keep WER, but add product-shaped measures:

• critical-token error rate for names, dates, amounts, units, negations, identifiers, and consent words;
• action-field exact match after normalization, with no partial credit for a wrong account or amount;
• speaker-authority match, which checks whether the utterance was attributed to an eligible actor;
• revision escape rate, the share of effects prepared or committed from text that later changed;
• unsupported-content rate, where text has no audible basis;
• abstention quality, including whether the system asks for repair when evidence is insufficient.

Do not collapse these into one weighted score. A zero-tolerance field should remain a hard gate.

Start with one consequential spoken job

Suppose a four-person SaaS team is adding voice intake to a subscription-support product. During a call, the assistant may draft a case summary, extract an account name, propose a renewal-date change, and prepare a confirmation message. The current human workflow requires the account owner to approve billing changes.

The product has four effect classes:

This table prevents a common design error: assigning the same confidence threshold to a note and a charge. The system may use a rough transcript to improve search while refusing to derive a billing effect from it.

Use the product’s real nouns. If your job is clinical dictation, emergency dispatch, legal evidence, accessibility, employment screening, or financial advice, this lightweight protocol is not sufficient. Those domains require specialist validation and applicable legal or regulatory review.

Build a 24-case fixture, not a clean-audio demo

Create eight fixture families and record three independent examples in each. Twenty-four cases will not estimate a rare-event rate. It is small enough, however, for a team to inspect every audio file, intermediate transcript, proposed action, confirmation screen, and final effect.

Use speakers who have consented to the test. Record at least one microphone and acoustic path representative of production. Include a domain name that general dictation systems are unlikely to know, a multi-digit identifier, a decimal amount, a date that can be confused with another locale, and a genuine code-switched phrase if your users speak that way.

The code-switching evaluation study found that evaluation choices such as transliteration and normalization change correlation with human judgments. Do not treat one English-centered tokenizer as a universal scoring oracle. Preserve both raw text and a versioned normalization procedure, and have a fluent reviewer adjudicate launch languages.

The action-commit receipt

Create one receipt for every fixture. The transcript should be an input to the receipt, not the receipt itself.

voice_action_receipt:
  fixture_id: "renewal-correction-en-ja-02"
  provider:
    model: "pin-exact-model-id"
    api_mode: "file|live"
    region: "record-service-region"
  audio:
    source_hash: ""
    consent_record: "consent/test-speaker-02.md"
    captured_at_utc: ""
    retention_class: "delete-after-qa"
  transcript:
    provisional_events: []
    finalized_text: ""
    finalized_at_utc: ""
    speaker_segments: []
    critical_fields:
      renewal_date: { value: null, source_span: "", verified: false }
      payment_instruction: { value: null, source_span: "", verified: false }
    revisions: []
    unsupported_content: []
  authority:
    authenticated_actor: ""
    claimed_speaker: ""
    allowed_effects: []
    confirmation_event: null
  proposed_effect:
    tool: "billing.change_renewal_date"
    arguments: {}
    idempotency_key: ""
    reversible_until: ""
  outcome:
    decision: "draft|clarify|confirm|commit|refuse"
    committed_at_utc: null
    observed_terminal_state: null
    rollback_tested: false

The source span is essential. A reviewer should be able to jump from renewal_date back to the exact audio and transcript segment that produced it. If the system cannot explain where an action argument came from, it should not prepare the action.

An empty confidence field is better than invented certainty. The Gemini documentation describes supported features, not a universal per-word confidence contract. If a provider does not expose a calibrated confidence value, do not manufacture one from token probability, sentence fluency, or an LLM’s self-rating.

Use a two-phase action path

The safest useful architecture is not “never automate voice.” It is a two-phase path that keeps low-risk assistance fast and high-consequence effects bounded.

Phase 1: prepare. Consume only finalized segments. Extract proposed fields, retain source spans, classify the effect, and check speaker authority. The system may draft a note or confirmation, but it cannot perform the effect.

Phase 2: commit. Present the critical fields in a form the user can inspect. Require explicit confirmation for the relevant effect, bind it to an idempotency key and expiry, then execute through a tool policy that checks the same authorization independently. Read back the observed terminal state instead of trusting a success-shaped model response.

For live transcription, quarantine provisional text from action queues. A corrected utterance such as “Make that May 14—not May 4” should produce a revision event, not two competing dates. If a tool has already been prepared from May 4, invalidate the proposal before accepting the replacement.

For batch audio, finality still does not prove authority. A recording may contain quoted speech, playback from another device, or multiple participants. The product should know whether it is summarizing an artifact or accepting an instruction from an authenticated person.

Test hallucination as an effect problem

Language-model-based recognizers can emit plausible text without a matching acoustic basis. The Careless Whisper study examined Whisper transcriptions and reported that roughly one percent of its sampled audio transcriptions contained a hallucinated phrase or sentence; the study also found disproportionate risk in speech with longer non-vocal intervals. That result belongs to the studied model, samples, and period. It is not a measured Gemini 3.5 failure rate.

The transferable lesson is methodological: test silence and uncertainty for unsupported content, and score the downstream effect rather than only the text. Add three negative controls:

1. room noise with no speech;
2. a long pause after an incomplete instruction;
3. low-volume background speech that is not directed at the product.

A passing system emits no authoritative instruction from those controls. If it produces text, the action layer must still refuse to promote it. This separates recognizer failure from containment failure. A transcript hallucination that stays inside a marked draft is serious; the same hallucination changing an account is a system-level incident.

Treat speaker labels as routing hints, not identity

Diarization answers “which acoustic cluster spoke when?” It does not answer “which legal or account identity is this?” The gap becomes visible in overlapping speech, short acknowledgements, transferred calls, shared devices, and replayed audio.

Test a short owner confirmation interrupted by a longer non-owner turn. Record duration-weighted diarization error, but also score every authority-bearing segment separately. The Balanced Error Rate proposal is useful precisely because a short or less-talkative speaker can disappear inside aggregate duration. Your product gate can be simpler: any misattributed authorization word fails the case.

Authentication should come from the product session, a deliberate step-up challenge, or a trusted business process—not a generated Speaker 1 label. Voice biometrics, if used at all, require a separate threat model for replay, synthetic voice, consent, fallback, and account recovery. This article does not treat diarization as a biometric control.

Make privacy part of the fixture

Audio contains more than words. It can reveal other people, background activity, location clues, health information, emotion, and identity-linked voice characteristics. A transcript-only retention diagram misses the original file, partial events, screenshots or operator tools, provider logs, application traces, and derived action arguments.

Google’s Gemini API additional terms distinguish unpaid and paid services and warn against submitting sensitive information to unpaid services. Its zero-data-retention guidance documents feature-specific exceptions and configuration: Interactions storage must be disabled explicitly, Live API session resumption can retain conversation state for up to 24 hours, and File API objects require deletion. These are current provider documents, not a substitute for checking the exact account, region, contract, and API path used by your product.

The NIST Privacy Framework offers a broader outcome-based way to inventory and manage privacy risk. For the fixture, make the data path concrete:

• record who consented and whether bystanders can be captured;
• name every stored artifact and derived field;
• set an owner, purpose, access boundary, and deletion trigger;
• test withdrawal during a live session;
• delete one fixture and verify the provider, application, logs, and derived stores;
• prohibit raw production audio from entering an opt-in improvement dataset by accident.

A privacy statement that has never survived a deletion drill is a policy claim, not operational evidence.

Promotion rules should be non-averaged

Run the 24 fixtures against the current baseline and the candidate under the same input pack. Pin the model identifier, mode, region, client version, normalization code, tool schema, and confirmation UI. Repeat any stochastic or streaming case enough to expose revision behavior, but do not call 24 examples statistically representative.

Promote only if every hard gate passes:

Report ordinary WER, critical-token error, speaker errors, clarification rate, manual review time, latency to a usable draft, and latency to an authorized commit separately. A candidate may improve note quality while increasing confirmation friction. That can support a limited launch for drafting without authorizing automatic actions.

Use four decisions: draft-only, limited-action, hold, or reject. Name the exact surface. Do not promote a model brand across every voice workflow because one meeting-note fixture passed.

Failure modes to review before launch

Provisional text enters a queue. The UI later looks correct, but a stale action is already waiting.

Final is mistaken for true. Protocol finality prevents further edits; it does not validate content.

Speaker number becomes account identity. Diarization clusters speech and silently acquires authorization it never proved.

The correction is transcribed as a second instruction. Both the original and replacement values survive into downstream state.

A good WER hides the only harmful word. The system misses a negation, amount, unit, date, or identifier.

Human confirmation repeats the same ambiguity. A screen asks “Proceed?” without showing the normalized fields and intended effect.

The tool response is treated as the outcome. A success message arrives even though business state did not change, changed twice, or changed the wrong record.

Raw audio outlives the stated purpose. The application deletes a transcript while a provider file, resumable session, debug log, or evaluation dataset remains.

Where this protocol applies—and where it does not

Use this gate when speech becomes structured input for customer support, scheduling, CRM updates, order preparation, field operations, meeting follow-ups, or other product actions. It is especially valuable when a team is moving from “transcribe and display” to “transcribe and do.”

It is intentionally conservative for effects that leave the product. A private draft can tolerate more uncertainty when the user can correct it before use. A commercial commitment, message sent as the user, permission change, or destructive action deserves explicit field-level confirmation and an independent tool policy.

This protocol is not an independent benchmark of Gemini 3.5 Transcribe, proof of language-wide fairness, a biometric evaluation, a compliance certification, or a substitute for domain experts. The launch is in preview; feature behavior, pricing, retention controls, limits, and model aliases may change. Google’s published performance remains vendor evidence until reproduced on a disclosed task and corpus.

A 48-hour Build Lab plan

Hours 0–4: choose one spoken job and classify its effects. Mark which outputs may remain drafts and which require verification, authority, and explicit confirmation.

Hours 4–10: build the eight fixture families with consenting speakers. Freeze reference transcripts, critical fields, speaker authority, expected decisions, and deletion instructions before running a candidate.

Hours 10–18: instrument provisional, revision, and finalized transcript events. Keep action preparation behind a feature flag. Add source spans, idempotency keys, and terminal-state reads.

Hours 18–28: run the 24-case pack against the baseline and candidate. Have fluent reviewers adjudicate code-switching and critical fields. Inspect every negative control and speaker-authority case.

Hours 28–36: execute only synthetic or sandbox effects. Force correction, duplicate delivery, timeout, failed confirmation, unauthorized speaker, and rollback cases. Verify that provisional or unsupported text cannot escape.

Hours 36–42: trace consent, storage, logs, retention, and deletion. Repeat one deletion until every named copy and derived record has an observed terminal state.

Hours 42–48: sign a draft-only, limited-action, hold, or reject receipt. Record the tested languages, microphones, acoustic conditions, effect classes, model ID, API mode, and retest triggers.

Gemini 3.5 Transcribe makes richer speech input easier to integrate. The release decision should therefore get stricter, not looser. A transcript can help a product long before it is allowed to speak with the user’s authority.

References

1. Google DeepMind, Intelligent transcription with Gemini 3.5 Transcribe
2. Google AI for Developers, Gemini 3.5 Transcribe model documentation
3. Google AI for Developers, Zero data retention in the Gemini Developer API
4. Google AI for Developers, Gemini API Additional Terms of Service
5. NIST, OpenSAT 2020 Evaluation Plan
6. Koenecke et al., Careless Whisper: Speech-to-Text Hallucination Harms
7. Liu and Yu, BER: Balanced Error Rate for Speaker Diarization
8. Hamed et al., Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition
9. NIST, Privacy Framework