docs: 扁平化目录结构,优化文档为 Coding Agent 可读格式
- 移除 Obsidian 特有语法(callout、wikilinks、YAML frontmatter) - 目录结构从 3 层嵌套扁平化为编号文件(01~09) - 新增 docs/README.md 文档索引与推荐阅读顺序 - 精简冗余解释,保留所有代码块和实现参考 - CLAUDE.md 全面中文化
This commit is contained in:
136
CLAUDE.md
136
CLAUDE.md
@@ -1,106 +1,104 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
本文件为 Claude Code (claude.ai/code) 在本仓库中工作时提供指引。
|
||||
|
||||
## Project Overview
|
||||
## 项目概述
|
||||
|
||||
CamTalk is a multimodal real-time AI visual dialogue assistant. Users interact via camera and microphone — the app captures visual scenes and voice input, sends them to AI services, and responds with both text and speech. The project is currently in the design-document phase; source code is being built incrementally.
|
||||
CamTalk 是一款多模态实时 AI 视觉对话助手。用户通过摄像头和麦克风与 AI 交互,AI 理解视觉场景和语音输入后,以文字和语音形式给出自然回应。项目目前处于设计文档阶段,源代码正在逐步构建。
|
||||
|
||||
> **文档优先原则:** 执行任何开发任务前,先读取 `docs/` 下的相关设计文档(架构、接口、技术选型等),以文档为最高依据。代码实现应与文档一致;若有偏差,优先更新文档(尤其是接口文档)。
|
||||
|
||||
**Design docs (Chinese):** `docs/` contains the full architecture, API contracts, user stories, cost control strategies, and technology selection rationale.
|
||||
## 架构
|
||||
|
||||
## Architecture
|
||||
三层系统:
|
||||
|
||||
Three-layer system:
|
||||
1. **浏览器客户端**(React 18 + TypeScript, Vite)—— 媒体采集、边缘预处理(VAD 通过 `@ricky0123/vad-web`、关键帧检测通过 ONNX Runtime Web)、UI 渲染。核心 Hook:`useVisionSession()`
|
||||
2. **Go 网关**(gorilla/websocket, Redis, Viper, Zap)—— WebSocket 服务器、会话管理、模型路由、AI 编排、速率限制。每个 WebSocket 连接一个 goroutine。
|
||||
3. **云端 AI 服务** —— GPT-4o(LLM)、Deepgram(STT)、OpenAI TTS。仅通过 Go 网关访问,浏览器不直连。
|
||||
|
||||
1. **Browser Client** (React 18 + TypeScript, Vite) — media capture, edge preprocessing (VAD via `@ricky0123/vad-web`, keyframe detection via ONNX Runtime Web), UI rendering. Core hook: `useVisionSession()`.
|
||||
2. **Go Gateway** (gorilla/websocket, Redis, Viper, Zap) — WebSocket server, session management, model routing, AI orchestration, rate limiting. One goroutine per WebSocket connection.
|
||||
3. **Cloud AI Services** — GPT-4o (LLM), Deepgram (STT), OpenAI TTS. Accessed only through the Go gateway, never directly from the browser.
|
||||
**关键模式**:LLM 文本流和 TTS 音频流并行推送给客户端,以最小化感知延迟。
|
||||
|
||||
**Key pattern:** LLM text chunks and TTS audio are streamed in parallel to the client to minimize perceived latency.
|
||||
**存储**:冷热分离 —— Redis 存实时会话状态,PostgreSQL 存对话历史和用量统计(MVP 后引入)。Repository 接口模式(`HistoryRepository`、`UsageRepository`),MVP 用内存实现。
|
||||
|
||||
**Storage:** Cold/hot separation — Redis for real-time session state, PostgreSQL for conversation history and usage stats (deferred past MVP). Repository interface pattern (`HistoryRepository`, `UsageRepository`) with in-memory MVP implementations.
|
||||
## 技术栈
|
||||
|
||||
## Tech Stack
|
||||
| 层级 | 技术 |
|
||||
|------|------|
|
||||
| 前端 | React 18, TypeScript, Vite, ONNX Runtime Web, @ricky0123/vad-web |
|
||||
| 后端 | Go, gorilla/websocket, Redis, Viper, Zap |
|
||||
| LLM | GPT-4o(主), Claude Sonnet(备) |
|
||||
| STT | Deepgram(主), FunASR(自部署备选) |
|
||||
| TTS | OpenAI TTS(主), Edge TTS(免费替代) |
|
||||
| 模型路由 | GPT-4o-mini 用于轻量分类 |
|
||||
|
||||
| Layer | Tech |
|
||||
|-------|------|
|
||||
| Frontend | React 18, TypeScript, Vite, ONNX Runtime Web, @ricky0123/vad-web |
|
||||
| Backend | Go, gorilla/websocket, Redis, Viper, Zap |
|
||||
| LLM | GPT-4o (primary), Claude Sonnet (backup) |
|
||||
| STT | Deepgram (primary), FunASR (self-hosted backup) |
|
||||
| TTS | OpenAI TTS (primary), Edge TTS (free alternative) |
|
||||
| Model routing | GPT-4o-mini for lightweight classification |
|
||||
|
||||
## Build & Run Commands
|
||||
## 构建与运行命令
|
||||
|
||||
```bash
|
||||
# Frontend
|
||||
# 前端
|
||||
cd frontend && npm install
|
||||
npm run dev # Vite dev server
|
||||
npm run build # Production build
|
||||
npm run lint # ESLint
|
||||
npm run test # Vitest
|
||||
npm run dev # Vite 开发服务器
|
||||
npm run build # 生产构建
|
||||
npm run lint # ESLint 检查
|
||||
npm run test # Vitest 测试
|
||||
|
||||
# Backend
|
||||
# 后端
|
||||
cd backend && go mod download
|
||||
go run ./cmd/server # Start gateway on :8080
|
||||
go run ./cmd/server # 启动网关,监听 :8080
|
||||
go build -o bin/camtalk ./cmd/server
|
||||
go test ./... # Run all tests
|
||||
go test -run TestName ./path # Run single test
|
||||
go vet ./... # Static analysis
|
||||
go test ./... # 运行所有测试
|
||||
go test -run TestName ./path # 运行单个测试
|
||||
go vet ./... # 静态分析
|
||||
```
|
||||
|
||||
Infrastructure: Redis required for session state. PostgreSQL optional for MVP (in-memory fallback).
|
||||
基础设施:Redis 为会话状态必需。PostgreSQL 为 MVP 可选(内存回退)。
|
||||
|
||||
## WebSocket Protocol
|
||||
## WebSocket 协议
|
||||
|
||||
Endpoint: `ws://localhost:8080/ws`
|
||||
端点:`ws://localhost:8080/ws`
|
||||
|
||||
All messages are JSON text frames with `{type, request_id?, timestamp?}` envelope. See `docs/AI 视觉对话助手/项目实现/接口文档.md` for the full contract.
|
||||
所有消息为 JSON 文本帧,统一信封格式 `{type, request_id?, timestamp?}`。完整契约见 `docs/03-接口文档.md`。
|
||||
|
||||
**Client → Server:** `query` (image Base64 + audio Base64), `config`, `interrupt`, `ping`
|
||||
**Server → Client:** `connected`, `stt_result`, `llm_chunk`, `llm_done`, `tts_audio`, `error`, `pong`
|
||||
**客户端 → 服务端**:`query`(图像 Base64 + 音频 Base64)、`config`、`interrupt`、`ping`
|
||||
**服务端 → 客户端**:`connected`、`stt_result`、`llm_chunk`、`llm_done`、`tts_audio`、`error`、`pong`
|
||||
|
||||
**Heartbeat:** Client pings every 30s. Server disconnects after 60s of silence.
|
||||
**Reconnection:** Exponential backoff with jitter — 1s, 2s, 4s, 8s… max 30s.
|
||||
**心跳**:客户端每 30 秒 ping,服务端 60 秒无 ping 断开连接。
|
||||
**重连**:指数退避 + 抖动 —— 1s, 2s, 4s, 8s… 最大 30s。
|
||||
|
||||
## REST API (Auxiliary)
|
||||
## REST API(辅助)
|
||||
|
||||
- `GET /api/health` — health check (version, uptime, active sessions)
|
||||
- `POST /api/sessions` — create session (optional, MVP auto-creates on WS connect)
|
||||
- `DELETE /api/sessions/{id}` — destroy session
|
||||
- `GET /api/health` — 健康检查(版本、运行时间、活跃会话数)
|
||||
- `POST /api/sessions` — 创建会话(可选,MVP 在 WS 连接时自动创建)
|
||||
- `DELETE /api/sessions/{id}` — 销毁会话
|
||||
|
||||
## Error Codes
|
||||
## 错误码
|
||||
|
||||
`INVALID_MESSAGE`, `SESSION_NOT_FOUND`, `RATE_LIMITED`, `IMAGE_TOO_LARGE`, `AUDIO_TOO_SHORT`, `LLM_TIMEOUT`, `LLM_ERROR`, `STT_ERROR`, `TTS_ERROR`, `INTERNAL_ERROR`
|
||||
`INVALID_MESSAGE`、`SESSION_NOT_FOUND`、`RATE_LIMITED`、`IMAGE_TOO_LARGE`、`AUDIO_TOO_SHORT`、`LLM_TIMEOUT`、`LLM_ERROR`、`STT_ERROR`、`TTS_ERROR`、`INTERNAL_ERROR`
|
||||
|
||||
## Frontend Component Structure
|
||||
## 前端组件结构
|
||||
|
||||
| Component | Responsibility |
|
||||
|-----------|---------------|
|
||||
| `CameraManager` | Camera stream capture |
|
||||
| `MicManager` | Microphone audio capture |
|
||||
| `EdgeProcessor` | VAD + keyframe detection (ONNX Runtime) |
|
||||
| `WebSocketManager` | WS connection lifecycle |
|
||||
| `ChatPanel` | Message display |
|
||||
| `VideoPreview` | Camera feed display |
|
||||
| 组件 | 职责 |
|
||||
|------|------|
|
||||
| `CameraManager` | 摄像头流采集 |
|
||||
| `MicManager` | 麦克风音频采集 |
|
||||
| `EdgeProcessor` | VAD + 关键帧检测(ONNX Runtime) |
|
||||
| `WebSocketManager` | WebSocket 连接生命周期管理 |
|
||||
| `ChatPanel` | 消息展示 |
|
||||
| `VideoPreview` | 摄像头画面预览 |
|
||||
|
||||
## Backend Module Structure
|
||||
## 后端模块结构
|
||||
|
||||
| Module | Responsibility |
|
||||
|--------|---------------|
|
||||
| WebSocket Hub | Connection management, broadcast/direct push |
|
||||
| Session Manager | Session state, conversation history (Redis + TTL) |
|
||||
| Model Router | Select AI model per request (rule engine + cost threshold) |
|
||||
| AI Orchestrator | Parallel/sequential AI calls with context timeout |
|
||||
| Rate Limiter | Per-user token bucket rate limiting |
|
||||
| 模块 | 职责 |
|
||||
|------|------|
|
||||
| WebSocket Hub | 连接管理、广播/定向推送 |
|
||||
| Session Manager | 会话状态、对话历史(Redis + TTL) |
|
||||
| Model Router | 按请求选择 AI 模型(规则引擎 + 成本阈值) |
|
||||
| AI Orchestrator | 并行/串行 AI 调用编排,context 超时控制 |
|
||||
| Rate Limiter | 按用户的令牌桶速率限制 |
|
||||
|
||||
## Coding Conventions
|
||||
## 编码规范
|
||||
|
||||
- **Go:** Follow standard Go conventions. Use `context.Context` for cancellation/timeout in all AI calls. Use `sync.RWMutex` for concurrent map access. Struct tags use `json:"snake_case"`.
|
||||
- **TypeScript:** Strict mode. Interfaces for all data models. WebSocket message types as discriminated unions (`type` field).
|
||||
- **Commit messages:** Conventional commits 格式,描述用中文。示例:`feat: 添加 WebSocket 连接管理`, `fix: 修复心跳超时判断`, `docs: 更新接口文档`
|
||||
- **No auto-push:** 禁止自动 push,除非用户明确要求。
|
||||
- **Docs-first:** 实现功能前先读取 `docs/` 下的相关设计文档,以文档为依据进行开发。实现与文档不一致时,优先更新 `docs/` 下的接口文档。
|
||||
- **Go**:遵循标准 Go 规范。所有 AI 调用使用 `context.Context` 做取消/超时。并发 map 访问使用 `sync.RWMutex`。结构体标签用 `json:"snake_case"`。
|
||||
- **TypeScript**:严格模式。所有数据模型用接口定义。WebSocket 消息类型用可辨识联合类型(`type` 字段)。
|
||||
- **提交信息**:Conventional Commits 格式,描述用中文。示例:`feat: 添加 WebSocket 连接管理`、`fix: 修复心跳超时判断`、`docs: 更新接口文档`
|
||||
- **禁止自动 push**:除非用户明确要求。
|
||||
- **文档优先**:实现功能前先读取 `docs/` 下的相关设计文档。实现与文档不一致时,优先更新 `docs/` 下的接口文档。
|
||||
|
||||
Reference in New Issue
Block a user