107 lines
5.3 KiB
Markdown
107 lines
5.3 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Project Overview
|
||
|
||
CamTalk is a multimodal real-time AI visual dialogue assistant. Users interact via camera and microphone — the app captures visual scenes and voice input, sends them to AI services, and responds with both text and speech. The project is currently in the design-document phase; source code is being built incrementally.
|
||
|
||
> **文档优先原则：** 执行任何开发任务前，先读取 `docs/` 下的相关设计文档（架构、接口、技术选型等），以文档为最高依据。代码实现应与文档一致；若有偏差，优先更新文档（尤其是接口文档）。
|
||
|
||
**Design docs (Chinese):** `docs/` contains the full architecture, API contracts, user stories, cost control strategies, and technology selection rationale.
|
||
|
||
## Architecture
|
||
|
||
Three-layer system:
|
||
|
||
1. **Browser Client** (React 18 + TypeScript, Vite) — media capture, edge preprocessing (VAD via `@ricky0123/vad-web`, keyframe detection via ONNX Runtime Web), UI rendering. Core hook: `useVisionSession()`.
|
||
2. **Go Gateway** (gorilla/websocket, Redis, Viper, Zap) — WebSocket server, session management, model routing, AI orchestration, rate limiting. One goroutine per WebSocket connection.
|
||
3. **Cloud AI Services** — GPT-4o (LLM), Deepgram (STT), OpenAI TTS. Accessed only through the Go gateway, never directly from the browser.
|
||
|
||
**Key pattern:** LLM text chunks and TTS audio are streamed in parallel to the client to minimize perceived latency.
|
||
|
||
**Storage:** Cold/hot separation — Redis for real-time session state, PostgreSQL for conversation history and usage stats (deferred past MVP). Repository interface pattern (`HistoryRepository`, `UsageRepository`) with in-memory MVP implementations.
|
||
|
||
## Tech Stack
|
||
|
||
| Layer | Tech |
|
||
|-------|------|
|
||
| Frontend | React 18, TypeScript, Vite, ONNX Runtime Web, @ricky0123/vad-web |
|
||
| Backend | Go, gorilla/websocket, Redis, Viper, Zap |
|
||
| LLM | GPT-4o (primary), Claude Sonnet (backup) |
|
||
| STT | Deepgram (primary), FunASR (self-hosted backup) |
|
||
| TTS | OpenAI TTS (primary), Edge TTS (free alternative) |
|
||
| Model routing | GPT-4o-mini for lightweight classification |
|
||
|
||
## Build & Run Commands
|
||
|
||
```bash
|
||
# Frontend
|
||
cd frontend && npm install
|
||
npm run dev # Vite dev server
|
||
npm run build # Production build
|
||
npm run lint # ESLint
|
||
npm run test # Vitest
|
||
|
||
# Backend
|
||
cd backend && go mod download
|
||
go run ./cmd/server # Start gateway on :8080
|
||
go build -o bin/camtalk ./cmd/server
|
||
go test ./... # Run all tests
|
||
go test -run TestName ./path # Run single test
|
||
go vet ./... # Static analysis
|
||
```
|
||
|
||
Infrastructure: Redis required for session state. PostgreSQL optional for MVP (in-memory fallback).
|
||
|
||
## WebSocket Protocol
|
||
|
||
Endpoint: `ws://localhost:8080/ws`
|
||
|
||
All messages are JSON text frames with `{type, request_id?, timestamp?}` envelope. See `docs/AI 视觉对话助手/项目实现/接口文档.md` for the full contract.
|
||
|
||
**Client → Server:** `query` (image Base64 + audio Base64), `config`, `interrupt`, `ping`
|
||
**Server → Client:** `connected`, `stt_result`, `llm_chunk`, `llm_done`, `tts_audio`, `error`, `pong`
|
||
|
||
**Heartbeat:** Client pings every 30s. Server disconnects after 60s of silence.
|
||
**Reconnection:** Exponential backoff with jitter — 1s, 2s, 4s, 8s… max 30s.
|
||
|
||
## REST API (Auxiliary)
|
||
|
||
- `GET /api/health` — health check (version, uptime, active sessions)
|
||
- `POST /api/sessions` — create session (optional, MVP auto-creates on WS connect)
|
||
- `DELETE /api/sessions/{id}` — destroy session
|
||
|
||
## Error Codes
|
||
|
||
`INVALID_MESSAGE`, `SESSION_NOT_FOUND`, `RATE_LIMITED`, `IMAGE_TOO_LARGE`, `AUDIO_TOO_SHORT`, `LLM_TIMEOUT`, `LLM_ERROR`, `STT_ERROR`, `TTS_ERROR`, `INTERNAL_ERROR`
|
||
|
||
## Frontend Component Structure
|
||
|
||
| Component | Responsibility |
|
||
|-----------|---------------|
|
||
| `CameraManager` | Camera stream capture |
|
||
| `MicManager` | Microphone audio capture |
|
||
| `EdgeProcessor` | VAD + keyframe detection (ONNX Runtime) |
|
||
| `WebSocketManager` | WS connection lifecycle |
|
||
| `ChatPanel` | Message display |
|
||
| `VideoPreview` | Camera feed display |
|
||
|
||
## Backend Module Structure
|
||
|
||
| Module | Responsibility |
|
||
|--------|---------------|
|
||
| WebSocket Hub | Connection management, broadcast/direct push |
|
||
| Session Manager | Session state, conversation history (Redis + TTL) |
|
||
| Model Router | Select AI model per request (rule engine + cost threshold) |
|
||
| AI Orchestrator | Parallel/sequential AI calls with context timeout |
|
||
| Rate Limiter | Per-user token bucket rate limiting |
|
||
|
||
## Coding Conventions
|
||
|
||
- **Go:** Follow standard Go conventions. Use `context.Context` for cancellation/timeout in all AI calls. Use `sync.RWMutex` for concurrent map access. Struct tags use `json:"snake_case"`.
|
||
- **TypeScript:** Strict mode. Interfaces for all data models. WebSocket message types as discriminated unions (`type` field).
|
||
- **Commit messages:** Conventional commits 格式，描述用中文。示例：`feat: 添加 WebSocket 连接管理`, `fix: 修复心跳超时判断`, `docs: 更新接口文档`
|
||
- **No auto-push:** 禁止自动 push，除非用户明确要求。
|
||
- **Docs-first:** 实现功能前先读取 `docs/` 下的相关设计文档，以文档为依据进行开发。实现与文档不一致时，优先更新 `docs/` 下的接口文档。
|