Files
kandev/apps/backend/internal/user/service/service.go
T
João Salavisa 1fdeda904b feat: add voice mode for chat input (#1159)
* feat: add voice mode for chat input

Adds a voice input button to the chat composer with three transcription engines:
in-browser Web Speech API, in-browser Whisper via @huggingface/transformers, and
a server-side OpenAI Whisper fallback gated on KANDEV_VOICE_OPENAI_API_KEY.

User-configurable from a new /settings/voice-mode page:
- Master on/off toggle (defaults to on; opt-out, not opt-in)
- Engine preference (auto / Web Speech / Whisper Web / Whisper Server)
- Language hint (auto-detect or BCP-47 pinned)
- Activation mode (click-to-toggle or hold-to-talk)
- Auto-send after transcription
- Whisper Web model size (tiny / base / small)
- Editable Cmd+Shift+M keyboard shortcut

Voice settings live under user_settings.voice_mode as a nested JSON object —
no schema migration. Setting changes broadcast via the existing
user.settings.updated WS event so any open chat tab updates live.

The mic button is rendered on the right of the composer alongside the send
button (round primary fill; destructive fill with pulse ring while recording).
On browsers without a usable engine (e.g. iOS Chrome over HTTP) the button
stays visible as a muted greyed icon — tapping it surfaces a toast with the
reason (typically a missing secure context).

Whisper Web runs in a Web Worker with fp16 decoder weights (avoids the
MatMulNBits scale-missing crash that q4/q8 hit on WASM). Models are pulled
from the onnx-community/* mirror and cached by transformers.js in the Cache
Storage on first use.

* fix: address voice mode PR review feedback and E2E selector clash

Bot-review fixes from #1159:
- Map http.MaxBytesError from c.FormFile to 413 instead of bare 400.
- Strip BCP-47 region suffix before passing language hint to Whisper Web.
- Recover from worker crash: terminate + null worker + reset ready flag so
  next init/transcribe creates a fresh worker instead of hanging.
- Read live store state via storeApi.getState() in voice-mode settings
  savers to avoid stale-closure clobbering concurrent settings updates.
- Detach Web Speech callbacks before abort() so trailing onend/onerror
  events don't mutate hook state after cancel().
- Spread caller init before method/body in voice-api fetch so a caller-
  supplied init can't override the multipart POST.
- Replace t.Fatalf with t.Errorf+return inside test HTTP handler goroutine
  (FailNow from non-test goroutine is unsafe).
- Add Enabled/AutoSend zeroing assertions in partial-update test so the
  bool-overwrite-on-PATCH behavior is locked in explicitly.

E2E fix:
- toolbar-overflow spec used button.rounded-full which now matches both
  the send and voice buttons (strict-mode violation). Switched to the
  submit-message-button testid for an unambiguous locator.

* fix: scope voice settings rollback + whisper worker error to changed keys

Round 2 of PR review feedback on #1159:
- voice-mode settings rollback now restores only the keys patched by the
  failing request (not the whole snapshot), preserving concurrent edits
  to unrelated voiceMode fields.
- Shortcut rollback diffs previous vs next and reverts only the changed
  keys (restore prior value or delete if absent), preserving other
  concurrent shortcut edits.
- Whisper worker error handler captures the worker reference at attach
  time and only clears refs when this.worker still matches it — a stale
  error from a previously replaced worker no longer nukes the active one.
- Update stale whisper-web-models JSDoc that still pointed at Xenova
  mirrors.

* fix: voice mode progress scale, hold-mode race, race-aware rollback

Round 3 of PR review feedback on #1159:
- Normalise Whisper progress to 0–1 in ensureWhisperClient (transformers.js
  emits 0–100, the button's display + the `ready: 1` convention expect
  0–1 — fixes the 5000% mid-download display).
- Claim driverRef synchronously at the top of finishCapture so concurrent
  pointerup + pointerleave invocations early-return instead of racing and
  clobbering a freshly-started recording's ref.
- Race-aware rollback: voice-mode and shortcut catch branches now skip
  reverting a key when latest[key] !== next[key]. A failed earlier request
  no longer overwrites a later successful save to the same key.
- Update the Worker wire-protocol comment to show onnx-community/whisper-base
  instead of the stale Xenova/* example.

---------

Co-authored-by: Kandev Agent <agent@kandev.dev>
2026-06-01 14:32:39 +01:00

461 lines
15 KiB
Go

package service
import (
"context"
"errors"
"fmt"
"strings"
"time"
"github.com/kandev/kandev/internal/common/logger"
"github.com/kandev/kandev/internal/events"
"github.com/kandev/kandev/internal/events/bus"
"github.com/kandev/kandev/internal/lsp/installer"
"github.com/kandev/kandev/internal/user/models"
"github.com/kandev/kandev/internal/user/store"
"go.uber.org/zap"
)
var (
ErrUserNotFound = errors.New("user not found")
ErrValidation = errors.New("validation error")
)
const (
changesPanelLayoutFlat = "flat"
changesPanelLayoutTree = "tree"
)
type Service struct {
repo store.Repository
eventBus bus.EventBus
logger *logger.Logger
defaultUser string
}
type UpdateUserSettingsRequest struct {
WorkspaceID *string
KanbanViewMode *string
WorkflowFilterID *string
RepositoryIDs *[]string
InitialSetupComplete *bool
PreferredShell *string
DefaultEditorID *string
EnablePreviewOnClick *bool
ChatSubmitKey *string
ReviewAutoMarkOnScroll *bool
ShowReleaseNotification *bool
ReleaseNotesLastSeenVersion *string
LspAutoStartLanguages *[]string
LspAutoInstallLanguages *[]string
LspServerConfigs *map[string]map[string]interface{}
SavedLayouts *[]models.SavedLayout
SidebarViews *[]models.SidebarView
DefaultUtilityAgentID *string
DefaultUtilityModel *string
KeyboardShortcuts *map[string]interface{}
TerminalLinkBehavior *string
TerminalFontFamily *string
TerminalFontSize *int
ChangesPanelLayout *string
VoiceMode *models.VoiceModeSettings
}
func NewService(repo store.Repository, eventBus bus.EventBus, log *logger.Logger) *Service {
return &Service{
repo: repo,
eventBus: eventBus,
logger: log.WithFields(zap.String("component", "user-service")),
defaultUser: store.DefaultUserID,
}
}
func (s *Service) GetCurrentUser(ctx context.Context) (*models.User, error) {
user, err := s.repo.GetUser(ctx, s.defaultUser)
if err != nil {
return nil, ErrUserNotFound
}
return user, nil
}
func (s *Service) GetUserSettings(ctx context.Context) (*models.UserSettings, error) {
settings, err := s.repo.GetUserSettings(ctx, s.defaultUser)
if err != nil {
return nil, err
}
return settings, nil
}
func (s *Service) PreferredShell(ctx context.Context) (string, error) {
settings, err := s.repo.GetUserSettings(ctx, s.defaultUser)
if err != nil {
return "", err
}
return settings.PreferredShell, nil
}
// GetDefaultUtilitySettings returns the user's default utility agent/model settings.
func (s *Service) GetDefaultUtilitySettings(ctx context.Context) (agentID, model string, err error) {
settings, err := s.repo.GetUserSettings(ctx, s.defaultUser)
if err != nil {
return "", "", err
}
return settings.DefaultUtilityAgentID, settings.DefaultUtilityModel, nil
}
func (s *Service) UpdateUserSettings(ctx context.Context, req *UpdateUserSettingsRequest) (*models.UserSettings, error) {
settings, err := s.repo.GetUserSettings(ctx, s.defaultUser)
if err != nil {
return nil, err
}
if err := applyBasicSettings(settings, req); err != nil {
return nil, fmt.Errorf("%w: %s", ErrValidation, err.Error())
}
if err := s.applyChatSubmitKey(settings, req); err != nil {
return nil, fmt.Errorf("%w: %s", ErrValidation, err.Error())
}
if err := applyLSPSettings(settings, req); err != nil {
return nil, fmt.Errorf("%w: %s", ErrValidation, err.Error())
}
if err := applySavedLayouts(settings, req); err != nil {
return nil, fmt.Errorf("%w: %s", ErrValidation, err.Error())
}
if err := applySidebarViews(settings, req); err != nil {
return nil, fmt.Errorf("%w: %s", ErrValidation, err.Error())
}
if err := applyVoiceMode(settings, req.VoiceMode); err != nil {
return nil, fmt.Errorf("%w: %s", ErrValidation, err.Error())
}
settings.UpdatedAt = time.Now().UTC()
if err := s.repo.UpsertUserSettings(ctx, settings); err != nil {
return nil, err
}
s.publishUserSettingsEvent(ctx, settings)
return settings, nil
}
// applyBasicSettings copies simple (non-validated) fields from req to settings.
func applyBasicSettings(settings *models.UserSettings, req *UpdateUserSettingsRequest) error {
if req.WorkspaceID != nil {
settings.WorkspaceID = *req.WorkspaceID
}
if req.KanbanViewMode != nil {
settings.KanbanViewMode = *req.KanbanViewMode
}
if req.WorkflowFilterID != nil {
settings.WorkflowFilterID = *req.WorkflowFilterID
}
if req.RepositoryIDs != nil {
settings.RepositoryIDs = *req.RepositoryIDs
}
if req.InitialSetupComplete != nil {
settings.InitialSetupComplete = *req.InitialSetupComplete
}
if req.PreferredShell != nil {
settings.PreferredShell = strings.TrimSpace(*req.PreferredShell)
}
if req.DefaultEditorID != nil {
settings.DefaultEditorID = strings.TrimSpace(*req.DefaultEditorID)
}
if req.EnablePreviewOnClick != nil {
settings.EnablePreviewOnClick = *req.EnablePreviewOnClick
}
if req.ReviewAutoMarkOnScroll != nil {
settings.ReviewAutoMarkOnScroll = *req.ReviewAutoMarkOnScroll
}
if req.ShowReleaseNotification != nil {
settings.ShowReleaseNotification = *req.ShowReleaseNotification
}
if req.ReleaseNotesLastSeenVersion != nil {
settings.ReleaseNotesLastSeenVersion = *req.ReleaseNotesLastSeenVersion
}
if req.DefaultUtilityAgentID != nil {
settings.DefaultUtilityAgentID = strings.TrimSpace(*req.DefaultUtilityAgentID)
}
if req.DefaultUtilityModel != nil {
settings.DefaultUtilityModel = strings.TrimSpace(*req.DefaultUtilityModel)
}
if req.KeyboardShortcuts != nil {
if err := validateKeyboardShortcuts(*req.KeyboardShortcuts); err != nil {
return err
}
settings.KeyboardShortcuts = *req.KeyboardShortcuts
}
if err := applyTerminalLinkBehavior(settings, req.TerminalLinkBehavior); err != nil {
return err
}
if err := applyChangesPanelLayout(settings, req.ChangesPanelLayout); err != nil {
return err
}
if req.TerminalFontFamily != nil {
settings.TerminalFontFamily = strings.TrimSpace(*req.TerminalFontFamily)
}
if req.TerminalFontSize != nil {
v := *req.TerminalFontSize
if v != 0 && (v < 8 || v > 24) {
return errors.New("terminal_font_size must be 0 (default) or between 8 and 24")
}
settings.TerminalFontSize = v
}
return nil
}
func applyTerminalLinkBehavior(settings *models.UserSettings, value *string) error {
if value == nil {
return nil
}
v := strings.TrimSpace(*value)
if v != "new_tab" && v != "browser_panel" {
return errors.New("terminal_link_behavior must be 'new_tab' or 'browser_panel'")
}
settings.TerminalLinkBehavior = v
return nil
}
func applyChangesPanelLayout(settings *models.UserSettings, value *string) error {
if value == nil {
return nil
}
v := strings.TrimSpace(*value)
if v != changesPanelLayoutFlat && v != changesPanelLayoutTree {
return errors.New("changes_panel_layout must be 'flat' or 'tree'")
}
settings.ChangesPanelLayout = v
return nil
}
var (
validVoiceEngines = map[string]struct{}{
"auto": {},
"webSpeech": {},
"whisperWeb": {},
"whisperServer": {},
}
validVoiceModes = map[string]struct{}{
"toggle": {},
"hold": {},
}
validWhisperWebModels = map[string]struct{}{
"tiny": {},
"base": {},
"small": {},
}
)
// applyVoiceMode validates the inbound voice-mode settings and merges them
// onto the user record. Each sub-field is validated independently so a
// partial update (e.g. just `engine`) still works.
//
// `enabled` and `auto_send` are plain bools — every PATCH carries them. The
// settings UI always sends the full VoiceMode object so partial updates that
// would otherwise zero these are not a real concern.
func applyVoiceMode(settings *models.UserSettings, value *models.VoiceModeSettings) error {
if value == nil {
return nil
}
current := settings.VoiceMode
if current.Engine == "" {
current.Engine = "auto"
}
if value.Engine != "" {
if _, ok := validVoiceEngines[value.Engine]; !ok {
return errors.New("voice_mode.engine must be 'auto', 'webSpeech', 'whisperWeb', or 'whisperServer'")
}
current.Engine = value.Engine
}
if value.Language != "" {
current.Language = strings.TrimSpace(value.Language)
}
if value.Mode != "" {
if _, ok := validVoiceModes[value.Mode]; !ok {
return errors.New("voice_mode.mode must be 'toggle' or 'hold'")
}
current.Mode = value.Mode
}
if value.WhisperWebModel != "" {
if _, ok := validWhisperWebModels[value.WhisperWebModel]; !ok {
return errors.New("voice_mode.whisper_web_model must be 'tiny', 'base', or 'small'")
}
current.WhisperWebModel = value.WhisperWebModel
}
current.AutoSend = value.AutoSend
current.Enabled = value.Enabled
settings.VoiceMode = current
return nil
}
// applyChatSubmitKey validates and applies the chat_submit_key setting.
func (s *Service) applyChatSubmitKey(settings *models.UserSettings, req *UpdateUserSettingsRequest) error {
if req.ChatSubmitKey == nil {
return nil
}
key := strings.TrimSpace(*req.ChatSubmitKey)
if key != "enter" && key != "cmd_enter" {
return errors.New("chat_submit_key must be 'enter' or 'cmd_enter'")
}
s.logger.Info("[Settings] Setting ChatSubmitKey", zap.String("value", key))
settings.ChatSubmitKey = key
return nil
}
// applyLSPSettings validates and applies LSP-related settings.
func applyLSPSettings(settings *models.UserSettings, req *UpdateUserSettingsRequest) error {
if req.LspAutoStartLanguages != nil {
if err := validateLSPLanguages(*req.LspAutoStartLanguages); err != nil {
return fmt.Errorf("lsp_auto_start_languages: %w", err)
}
settings.LspAutoStartLanguages = *req.LspAutoStartLanguages
}
if req.LspAutoInstallLanguages != nil {
if err := validateLSPLanguages(*req.LspAutoInstallLanguages); err != nil {
return fmt.Errorf("lsp_auto_install_languages: %w", err)
}
settings.LspAutoInstallLanguages = *req.LspAutoInstallLanguages
}
if req.LspServerConfigs != nil {
settings.LspServerConfigs = *req.LspServerConfigs
}
return nil
}
const maxSavedLayouts = 20
// applySavedLayouts validates and applies the saved_layouts setting.
func applySavedLayouts(settings *models.UserSettings, req *UpdateUserSettingsRequest) error {
if req.SavedLayouts == nil {
return nil
}
layouts := *req.SavedLayouts
if len(layouts) > maxSavedLayouts {
return fmt.Errorf("saved_layouts: max %d layouts allowed", maxSavedLayouts)
}
for i := range layouts {
if strings.TrimSpace(layouts[i].Name) == "" {
return errors.New("saved_layouts: layout name must not be empty")
}
}
settings.SavedLayouts = layouts
return nil
}
const maxSidebarViews = 50
// applySidebarViews validates and applies the sidebar_views setting.
func applySidebarViews(settings *models.UserSettings, req *UpdateUserSettingsRequest) error {
if req.SidebarViews == nil {
return nil
}
views := *req.SidebarViews
if len(views) > maxSidebarViews {
return fmt.Errorf("sidebar_views: max %d views allowed", maxSidebarViews)
}
seen := make(map[string]struct{}, len(views))
for i := range views {
if strings.TrimSpace(views[i].ID) == "" {
return errors.New("sidebar_views: view id must not be empty")
}
if strings.TrimSpace(views[i].Name) == "" {
return errors.New("sidebar_views: view name must not be empty")
}
if _, dup := seen[views[i].ID]; dup {
return fmt.Errorf("sidebar_views: duplicate view id %q", views[i].ID)
}
seen[views[i].ID] = struct{}{}
}
settings.SidebarViews = views
return nil
}
func (s *Service) publishUserSettingsEvent(ctx context.Context, settings *models.UserSettings) {
if s.eventBus == nil || settings == nil {
return
}
data := map[string]interface{}{
"user_id": settings.UserID,
"workspace_id": settings.WorkspaceID,
"kanban_view_mode": settings.KanbanViewMode,
"workflow_filter_id": settings.WorkflowFilterID,
"repository_ids": settings.RepositoryIDs,
"initial_setup_complete": settings.InitialSetupComplete,
"preferred_shell": settings.PreferredShell,
"default_editor_id": settings.DefaultEditorID,
"enable_preview_on_click": settings.EnablePreviewOnClick,
"chat_submit_key": settings.ChatSubmitKey,
"review_auto_mark_on_scroll": settings.ReviewAutoMarkOnScroll,
"show_release_notification": settings.ShowReleaseNotification,
"release_notes_last_seen_version": settings.ReleaseNotesLastSeenVersion,
"lsp_auto_start_languages": settings.LspAutoStartLanguages,
"lsp_auto_install_languages": settings.LspAutoInstallLanguages,
"lsp_server_configs": settings.LspServerConfigs,
"saved_layouts": settings.SavedLayouts,
"sidebar_views": settings.SidebarViews,
"default_utility_agent_id": settings.DefaultUtilityAgentID,
"default_utility_model": settings.DefaultUtilityModel,
"keyboard_shortcuts": settings.KeyboardShortcuts,
"terminal_link_behavior": settings.TerminalLinkBehavior,
"terminal_font_family": settings.TerminalFontFamily,
"terminal_font_size": settings.TerminalFontSize,
"changes_panel_layout": settings.ChangesPanelLayout,
"voice_mode": settings.VoiceMode,
"updated_at": settings.UpdatedAt.Format(time.RFC3339),
}
if err := s.eventBus.Publish(ctx, events.UserSettingsUpdated, bus.NewEvent(events.UserSettingsUpdated, "user-service", data)); err != nil {
s.logger.Error("failed to publish user settings event", zap.Error(err))
}
}
func validateKeyboardShortcuts(shortcuts map[string]interface{}) error {
for name, raw := range shortcuts {
shortcut, ok := raw.(map[string]interface{})
if !ok {
return fmt.Errorf("keyboard_shortcuts.%s must be an object", name)
}
key, ok := shortcut["key"].(string)
if !ok || strings.TrimSpace(key) == "" {
return fmt.Errorf("keyboard_shortcuts.%s.key must be a non-empty string", name)
}
if modsRaw, exists := shortcut["modifiers"]; exists {
mods, ok := modsRaw.(map[string]interface{})
if !ok {
return fmt.Errorf("keyboard_shortcuts.%s.modifiers must be an object", name)
}
for mod, v := range mods {
if _, ok := v.(bool); !ok {
return fmt.Errorf("keyboard_shortcuts.%s.modifiers.%s must be a boolean", name, mod)
}
}
}
}
return nil
}
func validateLSPLanguages(langs []string) error {
supported := installer.SupportedLanguages()
for _, lang := range langs {
if _, ok := supported[lang]; !ok {
return fmt.Errorf("unsupported language: %s", lang)
}
}
return nil
}
func (s *Service) ClearDefaultEditorID(ctx context.Context, editorID string) error {
if editorID == "" {
return nil
}
settings, err := s.repo.GetUserSettings(ctx, s.defaultUser)
if err != nil {
return err
}
if settings.DefaultEditorID != editorID {
return nil
}
settings.DefaultEditorID = ""
settings.UpdatedAt = time.Now().UTC()
if err := s.repo.UpsertUserSettings(ctx, settings); err != nil {
return err
}
s.publishUserSettingsEvent(ctx, settings)
return nil
}