CamTalk/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

CamTalk is a multimodal real-time AI visual dialogue assistant. Users interact via camera and microphone — the app captures visual scenes and voice input, sends them to AI services, and responds with both text and speech. The project is currently in the design-document phase; source code is being built incrementally.

**Design docs (Chinese):** `docs/` contains the full architecture, API contracts, user stories, cost control strategies, and technology selection rationale. Read these before implementing any feature.

## Architecture

Three-layer system:

1. **Browser Client** (React 18 + TypeScript, Vite) — media capture, edge preprocessing (VAD via `@ricky0123/vad-web`, keyframe detection via ONNX Runtime Web), UI rendering. Core hook: `useVisionSession()`.
2. **Go Gateway** (gorilla/websocket, Redis, Viper, Zap) — WebSocket server, session management, model routing, AI orchestration, rate limiting. One goroutine per WebSocket connection.
3. **Cloud AI Services** — GPT-4o (LLM), Deepgram (STT), OpenAI TTS. Accessed only through the Go gateway, never directly from the browser.

**Key pattern:** LLM text chunks and TTS audio are streamed in parallel to the client to minimize perceived latency.

**Storage:** Cold/hot separation — Redis for real-time session state, PostgreSQL for conversation history and usage stats (deferred past MVP). Repository interface pattern (`HistoryRepository`, `UsageRepository`) with in-memory MVP implementations.

## Tech Stack

| Layer | Tech |
|-------|------|
| Frontend | React 18, TypeScript, Vite, ONNX Runtime Web, @ricky0123/vad-web |
| Backend | Go, gorilla/websocket, Redis, Viper, Zap |
| LLM | GPT-4o (primary), Claude Sonnet (backup) |
| STT | Deepgram (primary), FunASR (self-hosted backup) |
| TTS | OpenAI TTS (primary), Edge TTS (free alternative) |
| Model routing | GPT-4o-mini for lightweight classification |

## Build & Run Commands

```bash
# Frontend
cd frontend && npm install
npm run dev          # Vite dev server
npm run build        # Production build
npm run lint         # ESLint
npm run test         # Vitest

# Backend
cd backend && go mod download
go run ./cmd/server          # Start gateway on :8080
go build -o bin/camtalk ./cmd/server
go test ./...                # Run all tests
go test -run TestName ./path # Run single test
go vet ./...                 # Static analysis
```

Infrastructure: Redis required for session state. PostgreSQL optional for MVP (in-memory fallback).

## WebSocket Protocol

Endpoint: `ws://localhost:8080/ws`

All messages are JSON text frames with `{type, request_id?, timestamp?}` envelope. See `docs/AI 视觉对话助手/项目实现/接口文档.md` for the full contract.

**Client → Server:** `query` (image Base64 + audio Base64), `config`, `interrupt`, `ping`
**Server → Client:** `connected`, `stt_result`, `llm_chunk`, `llm_done`, `tts_audio`, `error`, `pong`

**Heartbeat:** Client pings every 30s. Server disconnects after 60s of silence.
**Reconnection:** Exponential backoff with jitter — 1s, 2s, 4s, 8s… max 30s.

## REST API (Auxiliary)

- `GET /api/health` — health check (version, uptime, active sessions)
- `POST /api/sessions` — create session (optional, MVP auto-creates on WS connect)
- `DELETE /api/sessions/{id}` — destroy session

## Error Codes

`INVALID_MESSAGE`, `SESSION_NOT_FOUND`, `RATE_LIMITED`, `IMAGE_TOO_LARGE`, `AUDIO_TOO_SHORT`, `LLM_TIMEOUT`, `LLM_ERROR`, `STT_ERROR`, `TTS_ERROR`, `INTERNAL_ERROR`

## Frontend Component Structure

| Component | Responsibility |
|-----------|---------------|
| `CameraManager` | Camera stream capture |
| `MicManager` | Microphone audio capture |
| `EdgeProcessor` | VAD + keyframe detection (ONNX Runtime) |
| `WebSocketManager` | WS connection lifecycle |
| `ChatPanel` | Message display |
| `VideoPreview` | Camera feed display |

## Backend Module Structure

| Module | Responsibility |
|--------|---------------|
| WebSocket Hub | Connection management, broadcast/direct push |
| Session Manager | Session state, conversation history (Redis + TTL) |
| Model Router | Select AI model per request (rule engine + cost threshold) |
| AI Orchestrator | Parallel/sequential AI calls with context timeout |
| Rate Limiter | Per-user token bucket rate limiting |

## Coding Conventions

- **Go:** Follow standard Go conventions. Use `context.Context` for cancellation/timeout in all AI calls. Use `sync.RWMutex` for concurrent map access. Struct tags use `json:"snake_case"`.
- **TypeScript:** Strict mode. Interfaces for all data models. WebSocket message types as discriminated unions (`type` field).
- **Commit messages:** Use conventional commits format: `feat:`, `fix:`, `docs:`, `refactor:`, `test:`, `chore:`
- **No auto-push:** Do not push to remote unless explicitly asked.
- **Docs-first:** When implementing a feature, update the relevant interface doc in `docs/` if the implementation diverges from the spec.