Files
kandev/apps/web/components/task/chat/chat-input-toolbar.tsx
T
João Salavisa 1fdeda904b feat: add voice mode for chat input (#1159)
* feat: add voice mode for chat input

Adds a voice input button to the chat composer with three transcription engines:
in-browser Web Speech API, in-browser Whisper via @huggingface/transformers, and
a server-side OpenAI Whisper fallback gated on KANDEV_VOICE_OPENAI_API_KEY.

User-configurable from a new /settings/voice-mode page:
- Master on/off toggle (defaults to on; opt-out, not opt-in)
- Engine preference (auto / Web Speech / Whisper Web / Whisper Server)
- Language hint (auto-detect or BCP-47 pinned)
- Activation mode (click-to-toggle or hold-to-talk)
- Auto-send after transcription
- Whisper Web model size (tiny / base / small)
- Editable Cmd+Shift+M keyboard shortcut

Voice settings live under user_settings.voice_mode as a nested JSON object —
no schema migration. Setting changes broadcast via the existing
user.settings.updated WS event so any open chat tab updates live.

The mic button is rendered on the right of the composer alongside the send
button (round primary fill; destructive fill with pulse ring while recording).
On browsers without a usable engine (e.g. iOS Chrome over HTTP) the button
stays visible as a muted greyed icon — tapping it surfaces a toast with the
reason (typically a missing secure context).

Whisper Web runs in a Web Worker with fp16 decoder weights (avoids the
MatMulNBits scale-missing crash that q4/q8 hit on WASM). Models are pulled
from the onnx-community/* mirror and cached by transformers.js in the Cache
Storage on first use.

* fix: address voice mode PR review feedback and E2E selector clash

Bot-review fixes from #1159:
- Map http.MaxBytesError from c.FormFile to 413 instead of bare 400.
- Strip BCP-47 region suffix before passing language hint to Whisper Web.
- Recover from worker crash: terminate + null worker + reset ready flag so
  next init/transcribe creates a fresh worker instead of hanging.
- Read live store state via storeApi.getState() in voice-mode settings
  savers to avoid stale-closure clobbering concurrent settings updates.
- Detach Web Speech callbacks before abort() so trailing onend/onerror
  events don't mutate hook state after cancel().
- Spread caller init before method/body in voice-api fetch so a caller-
  supplied init can't override the multipart POST.
- Replace t.Fatalf with t.Errorf+return inside test HTTP handler goroutine
  (FailNow from non-test goroutine is unsafe).
- Add Enabled/AutoSend zeroing assertions in partial-update test so the
  bool-overwrite-on-PATCH behavior is locked in explicitly.

E2E fix:
- toolbar-overflow spec used button.rounded-full which now matches both
  the send and voice buttons (strict-mode violation). Switched to the
  submit-message-button testid for an unambiguous locator.

* fix: scope voice settings rollback + whisper worker error to changed keys

Round 2 of PR review feedback on #1159:
- voice-mode settings rollback now restores only the keys patched by the
  failing request (not the whole snapshot), preserving concurrent edits
  to unrelated voiceMode fields.
- Shortcut rollback diffs previous vs next and reverts only the changed
  keys (restore prior value or delete if absent), preserving other
  concurrent shortcut edits.
- Whisper worker error handler captures the worker reference at attach
  time and only clears refs when this.worker still matches it — a stale
  error from a previously replaced worker no longer nukes the active one.
- Update stale whisper-web-models JSDoc that still pointed at Xenova
  mirrors.

* fix: voice mode progress scale, hold-mode race, race-aware rollback

Round 3 of PR review feedback on #1159:
- Normalise Whisper progress to 0–1 in ensureWhisperClient (transformers.js
  emits 0–100, the button's display + the `ready: 1` convention expect
  0–1 — fixes the 5000% mid-download display).
- Claim driverRef synchronously at the top of finishCapture so concurrent
  pointerup + pointerleave invocations early-return instead of racing and
  clobbering a freshly-started recording's ref.
- Race-aware rollback: voice-mode and shortcut catch branches now skip
  reverting a key when latest[key] !== next[key]. A failed earlier request
  no longer overwrites a later successful save to the same key.
- Update the Worker wire-protocol comment to show onnx-community/whisper-base
  instead of the stale Xenova/* example.

---------

Co-authored-by: Kandev Agent <agent@kandev.dev>
2026-06-01 14:32:39 +01:00

615 lines
19 KiB
TypeScript

"use client";
import { memo, useCallback, useRef, useState, type ReactNode } from "react";
import {
IconArrowUp,
IconChevronsLeft,
IconDots,
IconFileTextSpark,
IconPlayerPauseFilled,
IconAt,
IconPlugConnected,
IconPlugConnectedX,
IconPaperclip,
} from "@tabler/icons-react";
import { EnhancePromptButton } from "@/components/enhance-prompt-button";
import { GridSpinner } from "@/components/grid-spinner";
import { Button } from "@kandev/ui/button";
import { Tooltip, TooltipContent, TooltipTrigger } from "@kandev/ui/tooltip";
import { cn } from "@/lib/utils";
import { useToolbarCollapsed } from "@/hooks/use-toolbar-collapsed";
import { SHORTCUTS } from "@/lib/keyboard/constants";
import { KeyboardShortcutTooltip } from "@/components/keyboard-shortcut-tooltip";
import { getShortcut } from "@/lib/keyboard/shortcut-overrides";
import { formatShortcut } from "@/lib/keyboard/utils";
import { useAppStore } from "@/components/state-provider";
import { TokenUsageDisplay } from "@/components/task/chat/token-usage-display";
import { SessionsDropdown } from "@/components/task/sessions-dropdown";
import { ModelSelector } from "@/components/task/model-selector";
import { ModeSelector } from "@/components/task/mode-selector";
import { ContextPopover } from "./context-popover";
import { ResetContextButton } from "./reset-context-button";
import { ImplementPlanButton } from "./implement-plan-button";
import { VoiceInputButton } from "./voice-input-button";
import type { ContextFile } from "@/lib/state/context-files-store";
export type ChatInputToolbarProps = {
planModeEnabled: boolean;
planModeAvailable?: boolean;
mcpServers?: string[];
onPlanModeChange: (enabled: boolean) => void;
sessionId: string | null;
taskId: string | null;
taskTitle?: string;
taskDescription: string;
isAgentBusy: boolean;
/** Whether the input has content to send (text, comments, or context) */
hasContent?: boolean;
isDisabled: boolean;
submitDisabledReason?: string;
isSending: boolean;
onCancel: () => void | Promise<void>;
onSubmit: () => void;
submitKey?: "enter" | "cmd_enter";
contextCount?: number;
contextPopoverOpen?: boolean;
onContextPopoverOpenChange?: (open: boolean) => void;
/** Whether plan is selected as context in the popover (independent of plan panel) */
planContextEnabled?: boolean;
contextFiles?: ContextFile[];
onToggleFile?: (file: ContextFile) => void;
onImplementPlan?: (fresh: boolean) => void;
/** Callback to enhance the current prompt with AI */
onEnhancePrompt?: () => void;
/** Whether prompt enhancement is in progress */
isEnhancingPrompt?: boolean;
/** Whether utility agent is configured for AI enhancement */
isUtilityConfigured?: boolean;
/** Callback to open file picker for attaching files */
onAttachFiles?: () => void;
/** Callback to insert a transcribed voice utterance into the editor. When
* omitted, the voice button is hidden — keeps quick-chat / read-only
* variants free of a button they can't wire. */
onVoiceTranscript?: (text: string) => void;
/** Optional auto-send hook fired after a voice transcript is inserted. */
onVoiceAutoSend?: () => void;
/** Hide the sessions dropdown (for quick chat) */
hideSessionsDropdown?: boolean;
/** When true, only render the submit/cancel button — no other controls */
minimalToolbar?: boolean;
/** Hide the plan mode toggle button (for ephemeral/quick chat sessions) */
hidePlanMode?: boolean;
};
type SubmitButtonProps = {
isAgentBusy: boolean;
hasContent: boolean;
isDisabled: boolean;
submitDisabledReason?: string;
isSending: boolean;
planModeEnabled: boolean;
onCancel: () => void | Promise<void>;
onSubmit: () => void;
submitShortcut: (typeof SHORTCUTS)[keyof typeof SHORTCUTS];
};
function submitTooltipDescription(
isAgentBusy: boolean,
planModeEnabled: boolean,
submitDisabledReason?: string,
) {
if (submitDisabledReason) return submitDisabledReason;
if (isAgentBusy) return "Queue message";
if (planModeEnabled) return "Request plan changes";
return undefined;
}
function SubmitButton({
isAgentBusy,
hasContent,
isDisabled,
submitDisabledReason,
isSending,
planModeEnabled,
onCancel,
onSubmit,
submitShortcut,
}: SubmitButtonProps) {
// When agent is busy and there's nothing to send, show only the cancel button.
// When there's content to queue, show both cancel and send buttons side-by-side.
const showSendButton = !isAgentBusy || hasContent;
// Track cancel-in-flight locally so impatient retries are blocked at the
// button itself: cancelling a long-running tool (e.g. Claude Monitor) can
// take several seconds, and without this the user clicks repeatedly and
// each click hits the backend.
const [isCancelling, setIsCancelling] = useState(false);
const tooltipDescription = submitTooltipDescription(
isAgentBusy,
planModeEnabled,
submitDisabledReason,
);
// Re-entrancy guard via ref: `disabled={isCancelling}` already blocks DOM
// clicks, but the ref makes the guard effective for any programmatic caller
// and keeps `isCancelling` out of the useCallback deps so the handler
// identity is stable across the false→true→false flips.
const cancellingRef = useRef(false);
const handleCancelClick = useCallback(async () => {
if (cancellingRef.current) return;
cancellingRef.current = true;
setIsCancelling(true);
try {
await onCancel();
} catch (error) {
// Parent handlers normally swallow errors, but a rejected onCancel
// must not propagate as an unhandled rejection — and the button must
// re-enable so the user can retry.
console.error("Failed to cancel agent turn:", error);
} finally {
cancellingRef.current = false;
setIsCancelling(false);
}
}, [onCancel]);
return (
<div className="flex items-center gap-1">
{isAgentBusy && (
<Tooltip>
<TooltipTrigger asChild>
<Button
type="button"
variant="secondary"
size="icon"
className="h-7 w-7 rounded-full cursor-pointer bg-destructive/10 text-destructive hover:bg-destructive/20 disabled:cursor-not-allowed disabled:opacity-70"
onClick={handleCancelClick}
disabled={isCancelling}
data-testid="cancel-agent-button"
>
{isCancelling ? (
<GridSpinner className="text-destructive" />
) : (
<IconPlayerPauseFilled className="h-3.5 w-3.5" />
)}
</Button>
</TooltipTrigger>
<TooltipContent>{isCancelling ? "Cancelling..." : "Cancel agent"}</TooltipContent>
</Tooltip>
)}
{showSendButton && (
<KeyboardShortcutTooltip
shortcut={submitShortcut}
description={tooltipDescription}
enabled={!isDisabled || !!tooltipDescription}
>
<span className="inline-flex">
<Button
type="button"
variant="default"
size="icon"
className={cn(
"h-7 w-7 rounded-full cursor-pointer",
planModeEnabled && "bg-violet-600 hover:bg-violet-500",
)}
disabled={isDisabled}
onClick={onSubmit}
data-testid="submit-message-button"
>
{isSending && <GridSpinner className="text-primary-foreground" />}
{!isSending && planModeEnabled && <IconFileTextSpark className="h-4 w-4" />}
{!isSending && !planModeEnabled && <IconArrowUp className="h-4 w-4" />}
</Button>
</span>
</KeyboardShortcutTooltip>
)}
</div>
);
}
function PlanToggleButton({
planModeEnabled,
planModeAvailable,
onPlanModeChange,
}: {
planModeEnabled: boolean;
planModeAvailable: boolean;
onPlanModeChange: (enabled: boolean) => void;
}) {
const keyboardShortcuts = useAppStore((s) => s.userSettings.keyboardShortcuts);
const planModeShortcutLabel = formatShortcut(getShortcut("TOGGLE_PLAN_MODE", keyboardShortcuts));
const tooltip = planModeAvailable
? `Toggle plan mode (${planModeShortcutLabel}) — Agent collaborates on the plan without implementing changes`
: `Toggle plan layout (${planModeShortcutLabel}) — View and edit the plan (agent cannot read/write it without MCP)`;
return (
<Tooltip>
<TooltipTrigger asChild>
<Button
type="button"
variant="ghost"
size="sm"
data-testid="plan-mode-toggle-button"
data-plan-available={planModeAvailable}
data-plan-enabled={planModeEnabled}
className={cn(
"h-7 gap-1.5 px-2 hover:bg-muted/40 cursor-pointer",
planModeEnabled && planModeAvailable && "bg-violet-500/15 text-violet-400",
)}
onClick={() => onPlanModeChange(!planModeEnabled)}
>
<IconFileTextSpark className="h-4 w-4" />
</Button>
</TooltipTrigger>
<TooltipContent className="max-w-xs">{tooltip}</TooltipContent>
</Tooltip>
);
}
function McpIndicator({ mcpServers }: { mcpServers: string[] }) {
const hasMcp = mcpServers.length > 0;
const tooltipText = hasMcp
? `MCP Servers: ${mcpServers.join(", ")}`
: "Agent does not support MCP";
const Icon = hasMcp ? IconPlugConnected : IconPlugConnectedX;
return (
<Tooltip>
<TooltipTrigger asChild>
<div
className={cn(
"h-7 w-7 flex items-center justify-center rounded-md",
hasMcp ? "text-foreground" : "text-muted-foreground/40",
)}
>
<Icon className="h-4 w-4" />
</div>
</TooltipTrigger>
<TooltipContent>{tooltipText}</TooltipContent>
</Tooltip>
);
}
type ToolbarItemConfig = {
id: string;
collapsible: boolean;
section: "left" | "right";
render: () => ReactNode;
visible?: boolean;
};
function ToolbarExpandToggle({
isExpanded,
onToggle,
}: {
isExpanded: boolean;
onToggle: () => void;
}) {
return (
<Tooltip>
<TooltipTrigger asChild>
<Button
type="button"
variant="ghost"
size="icon"
aria-label={isExpanded ? "Collapse toolbar" : "More toolbar actions"}
className="h-7 w-7 cursor-pointer hover:bg-muted/40"
data-testid="toolbar-overflow-menu"
onClick={onToggle}
>
{isExpanded ? <IconChevronsLeft className="h-4 w-4" /> : <IconDots className="h-4 w-4" />}
</Button>
</TooltipTrigger>
<TooltipContent>{isExpanded ? "Collapse" : "More actions"}</TooltipContent>
</Tooltip>
);
}
function ToolbarRightSection({
showCollapsed,
rightItems,
sessionId,
planModeEnabled,
isAgentBusy,
hasContent,
onImplementPlan,
isDisabled,
submitDisabledReason,
isSending,
onCancel,
onSubmit,
submitShortcut,
onVoiceTranscript,
onVoiceAutoSend,
}: {
showCollapsed: boolean;
rightItems: ToolbarItemConfig[];
sessionId: string | null;
planModeEnabled: boolean;
isAgentBusy: boolean;
hasContent: boolean;
onImplementPlan?: (fresh: boolean) => void;
isDisabled: boolean;
submitDisabledReason?: string;
isSending: boolean;
onCancel: () => void;
onSubmit: () => void;
submitShortcut: (typeof SHORTCUTS)[keyof typeof SHORTCUTS];
onVoiceTranscript?: (text: string) => void;
onVoiceAutoSend?: () => void;
}) {
return (
<div className="flex items-center gap-0.5 shrink-0">
{!showCollapsed && <CollapsibleItems items={rightItems} testIdPrefix="toolbar-item-" />}
<TokenUsageDisplay sessionId={sessionId} />
{planModeEnabled && !isAgentBusy && onImplementPlan && (
<ImplementPlanButton onClick={onImplementPlan} />
)}
<div className="ml-1 flex items-center gap-1">
{onVoiceTranscript && (
<VoiceInputButton
onTranscript={onVoiceTranscript}
onAutoSend={onVoiceAutoSend}
disabled={isDisabled}
/>
)}
<SubmitButton
isAgentBusy={isAgentBusy}
hasContent={hasContent}
isDisabled={isDisabled}
submitDisabledReason={submitDisabledReason}
isSending={isSending}
planModeEnabled={planModeEnabled}
onCancel={onCancel}
onSubmit={onSubmit}
submitShortcut={submitShortcut}
/>
</div>
</div>
);
}
function buildCollapsibleItems(props: {
mcpServers: string[];
sessionId: string | null;
taskId: string | null;
taskTitle?: string;
taskDescription: string;
hideSessionsDropdown?: boolean;
isAgentBusy: boolean;
onEnhancePrompt?: () => void;
isEnhancingPrompt: boolean;
isUtilityConfigured: boolean;
}): ToolbarItemConfig[] {
return [
{
id: "mcp",
section: "left",
collapsible: true,
render: () => <McpIndicator mcpServers={props.mcpServers} />,
},
{
id: "mode",
section: "left",
collapsible: true,
render: () => <ModeSelector sessionId={props.sessionId} />,
},
{
id: "reset-context",
section: "right",
collapsible: true,
visible: !!props.sessionId && !props.isAgentBusy,
render: () => <ResetContextButton sessionId={props.sessionId!} />,
},
{
id: "sessions",
section: "right",
collapsible: true,
visible: !props.hideSessionsDropdown,
render: () => (
<SessionsDropdown
taskId={props.taskId}
activeSessionId={props.sessionId}
taskTitle={props.taskTitle}
/>
),
},
{
id: "model",
section: "right",
collapsible: true,
render: () => <ModelSelector sessionId={props.sessionId} />,
},
{
id: "enhance",
section: "right",
collapsible: true,
visible: !props.isAgentBusy,
render: () => (
<EnhancePromptButton
onClick={props.onEnhancePrompt ?? (() => {})}
isLoading={props.isEnhancingPrompt}
isConfigured={props.isUtilityConfigured}
/>
),
},
];
}
function CollapsibleItems({
items,
testIdPrefix,
}: {
items: ToolbarItemConfig[];
testIdPrefix: string;
}) {
return items.map((i) => (
<div key={i.id} data-testid={`${testIdPrefix}${i.id}`}>
{i.render()}
</div>
));
}
function AttachFilesButton({ onClick }: { onClick: () => void }) {
return (
<Tooltip>
<TooltipTrigger asChild>
<Button
type="button"
variant="ghost"
size="sm"
className="h-7 gap-1.5 px-2 cursor-pointer hover:bg-muted/40"
onClick={onClick}
>
<IconPaperclip className="h-4 w-4" />
</Button>
</TooltipTrigger>
<TooltipContent>Attach files</TooltipContent>
</Tooltip>
);
}
function MinimalToolbar({
isAgentBusy,
hasContent,
isDisabled,
submitDisabledReason,
isSending,
onCancel,
onSubmit,
submitKey = "cmd_enter",
}: Pick<
ChatInputToolbarProps,
| "isAgentBusy"
| "hasContent"
| "isDisabled"
| "submitDisabledReason"
| "isSending"
| "onCancel"
| "onSubmit"
| "submitKey"
>) {
const submitShortcut = submitKey === "enter" ? SHORTCUTS.SUBMIT_ENTER : SHORTCUTS.SUBMIT;
return (
<div className="flex items-center justify-end gap-1 px-1 pt-0 pb-0.5 border-t border-border">
<SubmitButton
isAgentBusy={isAgentBusy}
hasContent={hasContent ?? false}
isDisabled={isDisabled}
submitDisabledReason={submitDisabledReason}
isSending={isSending}
planModeEnabled={false}
onCancel={onCancel}
onSubmit={onSubmit}
submitShortcut={submitShortcut}
/>
</div>
);
}
const toolbarDefaults = {
planModeAvailable: true,
mcpServers: [] as string[],
submitKey: "cmd_enter" as const,
contextCount: 0,
contextPopoverOpen: false,
planContextEnabled: false,
contextFiles: [] as ContextFile[],
isEnhancingPrompt: false,
isUtilityConfigured: false,
hidePlanMode: false,
};
export const ChatInputToolbar = memo(function ChatInputToolbar(rawProps: ChatInputToolbarProps) {
const props = { ...toolbarDefaults, ...rawProps };
const submitShortcut = props.submitKey === "enter" ? SHORTCUTS.SUBMIT_ENTER : SHORTCUTS.SUBMIT;
const toolbarRef = useRef<HTMLDivElement>(null);
const isCollapsed = useToolbarCollapsed(toolbarRef);
const [isExpanded, setIsExpanded] = useState(false);
const showCollapsed = isCollapsed && !isExpanded;
if (props.minimalToolbar) {
return (
<MinimalToolbar
isAgentBusy={props.isAgentBusy}
hasContent={props.hasContent}
isDisabled={props.isDisabled}
submitDisabledReason={props.submitDisabledReason}
isSending={props.isSending}
onCancel={props.onCancel}
onSubmit={props.onSubmit}
submitKey={props.submitKey}
/>
);
}
const items = buildCollapsibleItems(props);
const leftItems = items.filter((i) => i.section === "left" && i.visible !== false);
const rightItems = items.filter((i) => i.section === "right" && i.visible !== false);
return (
<div
ref={toolbarRef}
data-testid="chat-input-toolbar"
className={cn(
"flex items-center gap-1 px-1 pt-0 pb-0.5 border-t border-border",
isCollapsed ? "overflow-x-auto scrollbar-hide" : "overflow-visible",
)}
>
<div className="flex items-center gap-0.5 shrink-0">
{!props.hidePlanMode && (
<PlanToggleButton
planModeEnabled={props.planModeEnabled}
planModeAvailable={props.planModeAvailable}
onPlanModeChange={props.onPlanModeChange}
/>
)}
{!showCollapsed && <CollapsibleItems items={leftItems} testIdPrefix="toolbar-item-" />}
{props.onAttachFiles && <AttachFilesButton onClick={props.onAttachFiles} />}
<ContextPopover
open={props.contextPopoverOpen}
onOpenChange={props.onContextPopoverOpenChange ?? (() => {})}
trigger={
<Button
type="button"
variant="ghost"
size="sm"
className="h-7 gap-1.5 px-2 cursor-pointer hover:bg-muted/40 relative"
>
<IconAt className="h-4 w-4" />
{props.contextCount > 0 && !isCollapsed && (
<span className="absolute -top-1 -right-1 h-4 min-w-4 rounded-full bg-muted-foreground/80 text-[10px] text-background flex items-center justify-center px-0.5 pointer-events-none">
{props.contextCount}
</span>
)}
</Button>
}
sessionId={props.sessionId}
planContextEnabled={props.planContextEnabled}
contextFiles={props.contextFiles}
onToggleFile={props.onToggleFile ?? (() => {})}
/>
{isCollapsed && (
<ToolbarExpandToggle isExpanded={isExpanded} onToggle={() => setIsExpanded((v) => !v)} />
)}
</div>
<div className="flex-1" />
<ToolbarRightSection
showCollapsed={showCollapsed}
rightItems={rightItems}
sessionId={props.sessionId}
planModeEnabled={props.planModeEnabled}
isAgentBusy={props.isAgentBusy}
hasContent={props.hasContent ?? false}
onImplementPlan={props.onImplementPlan}
isDisabled={props.isDisabled}
submitDisabledReason={props.submitDisabledReason}
isSending={props.isSending}
onCancel={props.onCancel}
onSubmit={props.onSubmit}
submitShortcut={submitShortcut}
onVoiceTranscript={props.onVoiceTranscript}
onVoiceAutoSend={props.onVoiceAutoSend}
/>
</div>
);
});