CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

CamTalk is a multimodal real-time AI visual dialogue assistant. Users interact via camera and microphone — the app captures visual scenes and voice input, sends them to AI services, and responds with both text and speech. The project is currently in the design-document phase; source code is being built incrementally.

文档优先原则： 执行任何开发任务前，先读取 docs/ 下的相关设计文档（架构、接口、技术选型等），以文档为最高依据。代码实现应与文档一致；若有偏差，优先更新文档（尤其是接口文档）。

Design docs (Chinese): docs/ contains the full architecture, API contracts, user stories, cost control strategies, and technology selection rationale.

Architecture

Three-layer system:

Browser Client (React 18 + TypeScript, Vite) — media capture, edge preprocessing (VAD via @ricky0123/vad-web, keyframe detection via ONNX Runtime Web), UI rendering. Core hook: useVisionSession().
Go Gateway (gorilla/websocket, Redis, Viper, Zap) — WebSocket server, session management, model routing, AI orchestration, rate limiting. One goroutine per WebSocket connection.
Cloud AI Services — GPT-4o (LLM), Deepgram (STT), OpenAI TTS. Accessed only through the Go gateway, never directly from the browser.

Key pattern: LLM text chunks and TTS audio are streamed in parallel to the client to minimize perceived latency.

Storage: Cold/hot separation — Redis for real-time session state, PostgreSQL for conversation history and usage stats (deferred past MVP). Repository interface pattern (HistoryRepository, UsageRepository) with in-memory MVP implementations.

Tech Stack

Layer	Tech
Frontend	React 18, TypeScript, Vite, ONNX Runtime Web, @ricky0123/vad-web
Backend	Go, gorilla/websocket, Redis, Viper, Zap
LLM	GPT-4o (primary), Claude Sonnet (backup)
STT	Deepgram (primary), FunASR (self-hosted backup)
TTS	OpenAI TTS (primary), Edge TTS (free alternative)
Model routing	GPT-4o-mini for lightweight classification

Build & Run Commands

# Frontend
cd frontend && npm install
npm run dev          # Vite dev server
npm run build        # Production build
npm run lint         # ESLint
npm run test         # Vitest

# Backend
cd backend && go mod download
go run ./cmd/server          # Start gateway on :8080
go build -o bin/camtalk ./cmd/server
go test ./...                # Run all tests
go test -run TestName ./path # Run single test
go vet ./...                 # Static analysis

Infrastructure: Redis required for session state. PostgreSQL optional for MVP (in-memory fallback).

WebSocket Protocol

Endpoint: ws://localhost:8080/ws

All messages are JSON text frames with {type, request_id?, timestamp?} envelope. See docs/AI 视觉对话助手/项目实现/接口文档.md for the full contract.

Client → Server: query (image Base64 + audio Base64), config, interrupt, ping Server → Client: connected, stt_result, llm_chunk, llm_done, tts_audio, error, pong

Heartbeat: Client pings every 30s. Server disconnects after 60s of silence. Reconnection: Exponential backoff with jitter — 1s, 2s, 4s, 8s… max 30s.

REST API (Auxiliary)

GET /api/health — health check (version, uptime, active sessions)
POST /api/sessions — create session (optional, MVP auto-creates on WS connect)
DELETE /api/sessions/{id} — destroy session

Error Codes

INVALID_MESSAGE, SESSION_NOT_FOUND, RATE_LIMITED, IMAGE_TOO_LARGE, AUDIO_TOO_SHORT, LLM_TIMEOUT, LLM_ERROR, STT_ERROR, TTS_ERROR, INTERNAL_ERROR

Frontend Component Structure

Component	Responsibility
`CameraManager`	Camera stream capture
`MicManager`	Microphone audio capture
`EdgeProcessor`	VAD + keyframe detection (ONNX Runtime)
`WebSocketManager`	WS connection lifecycle
`ChatPanel`	Message display
`VideoPreview`	Camera feed display

Backend Module Structure

Module	Responsibility
WebSocket Hub	Connection management, broadcast/direct push
Session Manager	Session state, conversation history (Redis + TTL)
Model Router	Select AI model per request (rule engine + cost threshold)
AI Orchestrator	Parallel/sequential AI calls with context timeout
Rate Limiter	Per-user token bucket rate limiting

Coding Conventions

Go: Follow standard Go conventions. Use context.Context for cancellation/timeout in all AI calls. Use sync.RWMutex for concurrent map access. Struct tags use json:"snake_case".
TypeScript: Strict mode. Interfaces for all data models. WebSocket message types as discriminated unions (type field).
Commit messages: Conventional commits 格式，描述用中文。示例：feat: 添加 WebSocket 连接管理, fix: 修复心跳超时判断, docs: 更新接口文档
No auto-push: 禁止自动 push，除非用户明确要求。
Docs-first: 实现功能前先读取 docs/ 下的相关设计文档，以文档为依据进行开发。实现与文档不一致时，优先更新 docs/ 下的接口文档。

5.3 KiB Raw Blame History Unescape Escape