Files
kandev/apps/web/components/settings/voice-mode-settings.tsx
T
João Salavisa 1fdeda904b feat: add voice mode for chat input (#1159)
* feat: add voice mode for chat input

Adds a voice input button to the chat composer with three transcription engines:
in-browser Web Speech API, in-browser Whisper via @huggingface/transformers, and
a server-side OpenAI Whisper fallback gated on KANDEV_VOICE_OPENAI_API_KEY.

User-configurable from a new /settings/voice-mode page:
- Master on/off toggle (defaults to on; opt-out, not opt-in)
- Engine preference (auto / Web Speech / Whisper Web / Whisper Server)
- Language hint (auto-detect or BCP-47 pinned)
- Activation mode (click-to-toggle or hold-to-talk)
- Auto-send after transcription
- Whisper Web model size (tiny / base / small)
- Editable Cmd+Shift+M keyboard shortcut

Voice settings live under user_settings.voice_mode as a nested JSON object —
no schema migration. Setting changes broadcast via the existing
user.settings.updated WS event so any open chat tab updates live.

The mic button is rendered on the right of the composer alongside the send
button (round primary fill; destructive fill with pulse ring while recording).
On browsers without a usable engine (e.g. iOS Chrome over HTTP) the button
stays visible as a muted greyed icon — tapping it surfaces a toast with the
reason (typically a missing secure context).

Whisper Web runs in a Web Worker with fp16 decoder weights (avoids the
MatMulNBits scale-missing crash that q4/q8 hit on WASM). Models are pulled
from the onnx-community/* mirror and cached by transformers.js in the Cache
Storage on first use.

* fix: address voice mode PR review feedback and E2E selector clash

Bot-review fixes from #1159:
- Map http.MaxBytesError from c.FormFile to 413 instead of bare 400.
- Strip BCP-47 region suffix before passing language hint to Whisper Web.
- Recover from worker crash: terminate + null worker + reset ready flag so
  next init/transcribe creates a fresh worker instead of hanging.
- Read live store state via storeApi.getState() in voice-mode settings
  savers to avoid stale-closure clobbering concurrent settings updates.
- Detach Web Speech callbacks before abort() so trailing onend/onerror
  events don't mutate hook state after cancel().
- Spread caller init before method/body in voice-api fetch so a caller-
  supplied init can't override the multipart POST.
- Replace t.Fatalf with t.Errorf+return inside test HTTP handler goroutine
  (FailNow from non-test goroutine is unsafe).
- Add Enabled/AutoSend zeroing assertions in partial-update test so the
  bool-overwrite-on-PATCH behavior is locked in explicitly.

E2E fix:
- toolbar-overflow spec used button.rounded-full which now matches both
  the send and voice buttons (strict-mode violation). Switched to the
  submit-message-button testid for an unambiguous locator.

* fix: scope voice settings rollback + whisper worker error to changed keys

Round 2 of PR review feedback on #1159:
- voice-mode settings rollback now restores only the keys patched by the
  failing request (not the whole snapshot), preserving concurrent edits
  to unrelated voiceMode fields.
- Shortcut rollback diffs previous vs next and reverts only the changed
  keys (restore prior value or delete if absent), preserving other
  concurrent shortcut edits.
- Whisper worker error handler captures the worker reference at attach
  time and only clears refs when this.worker still matches it — a stale
  error from a previously replaced worker no longer nukes the active one.
- Update stale whisper-web-models JSDoc that still pointed at Xenova
  mirrors.

* fix: voice mode progress scale, hold-mode race, race-aware rollback

Round 3 of PR review feedback on #1159:
- Normalise Whisper progress to 0–1 in ensureWhisperClient (transformers.js
  emits 0–100, the button's display + the `ready: 1` convention expect
  0–1 — fixes the 5000% mid-download display).
- Claim driverRef synchronously at the top of finishCapture so concurrent
  pointerup + pointerleave invocations early-return instead of racing and
  clobbering a freshly-started recording's ref.
- Race-aware rollback: voice-mode and shortcut catch branches now skip
  reverting a key when latest[key] !== next[key]. A failed earlier request
  no longer overwrites a later successful save to the same key.
- Update the Worker wire-protocol comment to show onnx-community/whisper-base
  instead of the stale Xenova/* example.

---------

Co-authored-by: Kandev Agent <agent@kandev.dev>
2026-06-01 14:32:39 +01:00

519 lines
19 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"use client";
import { useCallback, useMemo, useState } from "react";
import { IconAlertTriangle, IconMicrophone } from "@tabler/icons-react";
import { Badge } from "@kandev/ui/badge";
import { Card, CardContent, CardHeader, CardTitle } from "@kandev/ui/card";
import { Label } from "@kandev/ui/label";
import { RadioGroup, RadioGroupItem } from "@kandev/ui/radio-group";
import {
Select,
SelectContent,
SelectGroup,
SelectItem,
SelectLabel,
SelectTrigger,
SelectValue,
} from "@kandev/ui/select";
import { Switch } from "@kandev/ui/switch";
import { useAppStore, useAppStoreApi } from "@/components/state-provider";
import { useToast } from "@/components/toast-provider";
import { updateUserSettings } from "@/lib/api";
import { SettingsSection } from "@/components/settings/settings-section";
import { ShortcutRecorder } from "@/components/settings/keyboard-shortcuts-card";
import { detectVoiceCapabilities, type VoiceCapabilities } from "@/lib/voice/capabilities";
import type { VoiceModeState } from "@/lib/state/slices/settings/types";
import type { KeyboardShortcut } from "@/lib/keyboard/constants";
import {
CONFIGURABLE_SHORTCUTS,
getShortcut,
type StoredShortcutOverrides,
} from "@/lib/keyboard/shortcut-overrides";
import type {
VoiceInputActivationMode,
VoiceInputEngine,
VoiceModeSettings as VoiceModeWire,
WhisperWebModelSize,
} from "@/lib/types/http-voice";
// Single source of truth for the language options. Web Speech reads `lang`,
// Whisper engines treat it as a hint. "auto" defers to the browser locale.
const LANGUAGE_OPTIONS: Array<{ value: string; label: string }> = [
{ value: "auto", label: "Auto-detect (browser language)" },
{ value: "en-US", label: "English (United States)" },
{ value: "en-GB", label: "English (United Kingdom)" },
{ value: "es-ES", label: "Spanish (Spain)" },
{ value: "es-MX", label: "Spanish (Mexico)" },
{ value: "pt-PT", label: "Portuguese (Portugal)" },
{ value: "pt-BR", label: "Portuguese (Brazil)" },
{ value: "fr-FR", label: "French" },
{ value: "de-DE", label: "German" },
{ value: "it-IT", label: "Italian" },
{ value: "ja-JP", label: "Japanese" },
{ value: "zh-CN", label: "Chinese (Simplified)" },
];
const WHISPER_MODELS: Array<{
value: WhisperWebModelSize;
label: string;
size: string;
hint: string;
}> = [
{ value: "tiny", label: "Tiny", size: "~40 MB", hint: "Fastest, lower accuracy" },
{ value: "base", label: "Base", size: "~75 MB", hint: "Balanced default" },
{ value: "small", label: "Small", size: "~240 MB", hint: "Best accuracy, slower load" },
];
function toWire(state: VoiceModeState): VoiceModeWire {
return {
enabled: state.enabled,
engine: state.engine,
language: state.language,
mode: state.mode,
auto_send: state.autoSend,
whisper_web_model: state.whisperWebModel,
};
}
// ── Save hook ────────────────────────────────────────────────────────────
function useVoiceModeSaver() {
// Read userSettings via the store API (not as a React selector) so the
// async save handler reads the latest snapshot at invocation time instead
// of capturing a stale closure. Without this, concurrent settings updates
// racing with this save (or a rejection rolling back to a stale snapshot)
// can silently overwrite unrelated fields.
const storeApi = useAppStoreApi();
const setUserSettings = useAppStore((s) => s.setUserSettings);
const { toast } = useToast();
const [saving, setSaving] = useState(false);
const save = useCallback(
async (patch: Partial<VoiceModeState>) => {
const current = storeApi.getState().userSettings;
const previous = current.voiceMode;
const next = { ...previous, ...patch };
setUserSettings({ ...current, voiceMode: next });
setSaving(true);
try {
await updateUserSettings({ voice_mode: toWire(next) });
} catch {
// Rollback only the keys this request changed AND only when the live
// value still matches what we optimistically wrote. If a newer save
// for the same key landed first, that's now the truth — reverting
// would silently roll back the user's later edit.
const latest = storeApi.getState().userSettings;
const reverted: Partial<VoiceModeState> = {};
for (const key of Object.keys(patch) as Array<keyof VoiceModeState>) {
if (latest.voiceMode[key] !== next[key]) continue;
// Cast through unknown so the per-key assignment passes strict checks.
(reverted as Record<string, unknown>)[key] = previous[key];
}
setUserSettings({
...latest,
voiceMode: { ...latest.voiceMode, ...reverted },
});
toast({ title: "Failed to save Voice Mode setting", variant: "error" });
} finally {
setSaving(false);
}
},
[storeApi, setUserSettings, toast],
);
return { save, saving };
}
// ── Engine card ──────────────────────────────────────────────────────────
type EngineOption = {
value: VoiceInputEngine;
label: string;
description: string;
badge?: string;
disabled?: boolean;
};
function buildEngineOptions(caps: VoiceCapabilities): EngineOption[] {
return [
{
value: "auto",
label: "Automatic",
description: "Use the best engine available in this browser.",
},
{
value: "webSpeech",
label: "Web Speech (in-browser)",
description: caps.webSpeech
? "Free, instant, uses your browser's built-in speech recognition."
: "Not supported in this browser.",
disabled: !caps.webSpeech,
},
{
value: "whisperWeb",
label: "Whisper Web (private, in-browser)",
description: caps.whisperWeb
? "Runs OpenAI Whisper entirely on this device. First use downloads the model (40–240 MB)."
: "Not supported in this browser.",
badge: "Local",
disabled: !caps.whisperWeb,
},
{
value: "whisperServer",
label: "Whisper Server (OpenAI)",
description: caps.audioCapture
? "Sends audio to the backend, which forwards it to OpenAI's Whisper API. Requires a configured API key on the server."
: "Not supported in this browser.",
badge: "Server",
disabled: !caps.audioCapture,
},
];
}
function EngineCard({ caps }: { caps: VoiceCapabilities }) {
const voiceMode = useAppStore((s) => s.userSettings.voiceMode);
const { save, saving } = useVoiceModeSaver();
const options = useMemo(() => buildEngineOptions(caps), [caps]);
return (
<Card>
<CardHeader>
<CardTitle className="text-base">Transcription Engine</CardTitle>
</CardHeader>
<CardContent>
<RadioGroup
value={voiceMode.engine}
onValueChange={(v) => save({ engine: v as VoiceInputEngine })}
disabled={saving}
className="space-y-3"
>
{options.map((opt) => (
<Label
key={opt.value}
htmlFor={`voice-engine-${opt.value}`}
className={`flex items-start gap-3 rounded-md border p-3 ${
opt.disabled ? "opacity-50" : "cursor-pointer hover:bg-muted/30"
}`}
>
<RadioGroupItem
id={`voice-engine-${opt.value}`}
value={opt.value}
disabled={opt.disabled}
className="mt-0.5"
/>
<div className="space-y-1">
<div className="flex items-center gap-2 text-sm font-medium">
{opt.label}
{opt.badge && <Badge variant="secondary">{opt.badge}</Badge>}
</div>
<p className="text-xs text-muted-foreground">{opt.description}</p>
</div>
</Label>
))}
</RadioGroup>
</CardContent>
</Card>
);
}
// ── Behavior card (language + mode + auto-send) ──────────────────────────
function LanguageRow() {
const voiceMode = useAppStore((s) => s.userSettings.voiceMode);
const { save, saving } = useVoiceModeSaver();
return (
<div className="space-y-2">
<Label htmlFor="voice-language">Language</Label>
<Select
value={voiceMode.language}
onValueChange={(v) => save({ language: v })}
disabled={saving}
>
<SelectTrigger id="voice-language">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectGroup>
<SelectLabel>Languages</SelectLabel>
{LANGUAGE_OPTIONS.map((l) => (
<SelectItem key={l.value} value={l.value}>
{l.label}
</SelectItem>
))}
</SelectGroup>
</SelectContent>
</Select>
<p className="text-xs text-muted-foreground">
Recognition quality drops sharply when the language doesn&apos;t match what you&apos;re
speaking.
</p>
</div>
);
}
function ModeRow() {
const voiceMode = useAppStore((s) => s.userSettings.voiceMode);
const { save, saving } = useVoiceModeSaver();
return (
<div className="space-y-2">
<Label>Activation</Label>
<RadioGroup
value={voiceMode.mode}
onValueChange={(v) => save({ mode: v as VoiceInputActivationMode })}
disabled={saving}
className="flex gap-4"
>
<Label htmlFor="voice-mode-toggle" className="flex items-center gap-2 cursor-pointer">
<RadioGroupItem id="voice-mode-toggle" value="toggle" />
<span className="text-sm">Click to start / stop</span>
</Label>
<Label htmlFor="voice-mode-hold" className="flex items-center gap-2 cursor-pointer">
<RadioGroupItem id="voice-mode-hold" value="hold" />
<span className="text-sm">Hold to talk</span>
</Label>
</RadioGroup>
</div>
);
}
function AutoSendRow() {
const voiceMode = useAppStore((s) => s.userSettings.voiceMode);
const { save, saving } = useVoiceModeSaver();
return (
<div className="flex items-center justify-between">
<div className="space-y-1">
<Label htmlFor="voice-auto-send" className="cursor-pointer">
Auto-send after transcription
</Label>
<p className="text-xs text-muted-foreground">
Submit the message as soon as the transcript is inserted.
</p>
</div>
<Switch
id="voice-auto-send"
checked={voiceMode.autoSend}
onCheckedChange={(checked) => save({ autoSend: checked })}
disabled={saving}
/>
</div>
);
}
function BehaviorCard() {
return (
<Card>
<CardHeader>
<CardTitle className="text-base">Behavior</CardTitle>
</CardHeader>
<CardContent className="space-y-5">
<LanguageRow />
<ModeRow />
<AutoSendRow />
</CardContent>
</Card>
);
}
// ── Whisper Web model card ───────────────────────────────────────────────
function WhisperModelCard() {
const voiceMode = useAppStore((s) => s.userSettings.voiceMode);
const { save, saving } = useVoiceModeSaver();
return (
<Card>
<CardHeader>
<CardTitle className="text-base">Whisper Web Model</CardTitle>
</CardHeader>
<CardContent>
<RadioGroup
value={voiceMode.whisperWebModel}
onValueChange={(v) => save({ whisperWebModel: v as WhisperWebModelSize })}
disabled={saving}
className="space-y-2"
>
{WHISPER_MODELS.map((m) => (
<Label
key={m.value}
htmlFor={`whisper-model-${m.value}`}
className="flex items-start gap-3 rounded-md border p-3 cursor-pointer hover:bg-muted/30"
>
<RadioGroupItem id={`whisper-model-${m.value}`} value={m.value} className="mt-0.5" />
<div>
<div className="text-sm font-medium">
{m.label} <span className="text-muted-foreground font-normal">· {m.size}</span>
</div>
<p className="text-xs text-muted-foreground">{m.hint}</p>
</div>
</Label>
))}
</RadioGroup>
<p className="text-xs text-muted-foreground mt-3">
The model downloads on first use and is cached in your browser. Switching models triggers
another download next time you record.
</p>
</CardContent>
</Card>
);
}
// ── Enable card (top-level on/off) ───────────────────────────────────────
function EnableCard() {
const voiceMode = useAppStore((s) => s.userSettings.voiceMode);
const { save, saving } = useVoiceModeSaver();
return (
<Card>
<CardHeader>
<CardTitle className="text-base">Enable Voice Input</CardTitle>
</CardHeader>
<CardContent>
<div className="flex items-center justify-between">
<div className="space-y-1">
<Label htmlFor="voice-enabled" className="cursor-pointer">
Show the mic button on the chat composer
</Label>
<p className="text-xs text-muted-foreground">
When off, the voice button is hidden entirely and no voice-related code runs. Settings
below are preserved and re-applied when you turn it back on.
</p>
</div>
<Switch
id="voice-enabled"
checked={voiceMode.enabled}
onCheckedChange={(checked) => save({ enabled: checked })}
disabled={saving}
/>
</div>
</CardContent>
</Card>
);
}
// ── Availability banner ──────────────────────────────────────────────────
function AvailabilityBanner({ caps }: { caps: VoiceCapabilities }) {
if (caps.webSpeech || caps.whisperWeb || caps.audioCapture) return null;
// Secure-context requirement is the most common reason capability detection
// returns all-false on mobile (when reaching the dev server over LAN HTTP).
// Spell it out so the user doesn't have to guess.
const insecure = typeof window !== "undefined" && !window.isSecureContext;
return (
<div className="flex items-start gap-3 rounded-md border border-orange-500/40 bg-orange-500/5 p-3">
<IconAlertTriangle className="h-5 w-5 text-orange-500 shrink-0 mt-0.5" />
<div className="space-y-1 text-sm">
<p className="font-medium">Voice input is unavailable in this browser.</p>
<p className="text-xs text-muted-foreground">
{insecure
? "Microphone APIs require HTTPS or localhost. You appear to be on an insecure HTTP origin — load this page over HTTPS (or http://localhost) to enable voice input."
: "Your browser doesn't expose either the Web Speech API or MediaRecorder. Try Chrome, Edge, or Safari 14.5+."}
</p>
</div>
</div>
);
}
// ── Voice keyboard shortcut card ─────────────────────────────────────────
function useShortcutSaver() {
// Same stale-closure protection as useVoiceModeSaver — read live store
// state at call time so a concurrent keyboard-shortcut change from another
// settings card isn't clobbered by this card's optimistic update / rollback.
const storeApi = useAppStoreApi();
const setUserSettings = useAppStore((s) => s.setUserSettings);
const { toast } = useToast();
return useCallback(
(next: StoredShortcutOverrides) => {
const current = storeApi.getState().userSettings;
const previous = current.keyboardShortcuts;
setUserSettings({ ...current, keyboardShortcuts: next });
updateUserSettings({ keyboard_shortcuts: next }).catch(() => {
// Rollback only the keys this request changed AND only when the live
// value still matches what we optimistically wrote. Skip otherwise so
// a newer successful save to the same key isn't reverted.
const latest = storeApi.getState().userSettings;
const restored: StoredShortcutOverrides = { ...latest.keyboardShortcuts };
const changedKeys = new Set([...Object.keys(previous), ...Object.keys(next)]);
for (const key of changedKeys) {
if (previous[key] === next[key]) continue;
if (latest.keyboardShortcuts[key] !== next[key]) continue;
if (previous[key] === undefined) delete restored[key];
else restored[key] = previous[key];
}
setUserSettings({ ...latest, keyboardShortcuts: restored });
toast({ title: "Failed to save shortcut", variant: "error" });
});
},
[storeApi, setUserSettings, toast],
);
}
function VoiceShortcutCard() {
const overrides = useAppStore((s) => s.userSettings.keyboardShortcuts);
const persist = useShortcutSaver();
const current = getShortcut("VOICE_INPUT_TOGGLE", overrides);
const handleChange = useCallback(
(_id: string, shortcut: KeyboardShortcut) =>
persist({ ...overrides, VOICE_INPUT_TOGGLE: shortcut }),
[overrides, persist],
);
const handleReset = useCallback(() => {
const next = { ...overrides };
delete next.VOICE_INPUT_TOGGLE;
persist(next);
}, [overrides, persist]);
return (
<Card>
<CardHeader>
<CardTitle className="text-base">
{CONFIGURABLE_SHORTCUTS.VOICE_INPUT_TOGGLE.label} Shortcut
</CardTitle>
</CardHeader>
<CardContent>
<ShortcutRecorder
shortcutId="VOICE_INPUT_TOGGLE"
current={current}
onChange={handleChange}
onReset={handleReset}
/>
<p className="text-xs text-muted-foreground mt-2">
Click the shortcut to record a new key combination. All keyboard shortcuts can also be
edited in General Settings.
</p>
</CardContent>
</Card>
);
}
// ── Page ─────────────────────────────────────────────────────────────────
export function VoiceModeSettings() {
const caps = useMemo(() => detectVoiceCapabilities(), []);
const enabled = useAppStore((s) => s.userSettings.voiceMode.enabled);
return (
<SettingsSection
icon={<IconMicrophone className="h-5 w-5" />}
title="Voice Mode"
description="Configure how voice input works on the chat composer."
>
<div className="space-y-4">
<EnableCard />
{/* When voice is disabled, keep showing the secondary cards but dim
them — preserves the visible configuration without implying it has
any effect right now. */}
<div className={enabled ? undefined : "opacity-50 pointer-events-none"}>
<div className="space-y-4">
<AvailabilityBanner caps={caps} />
<EngineCard caps={caps} />
<BehaviorCard />
<WhisperModelCard />
<VoiceShortcutCard />
</div>
</div>
</div>
</SettingsSection>
);
}