shinalok
68f741af72
Phase 17: Multimodal image understanding via analyze_image tool
...
Dual-model approach (C): Qwen3-8B handles conversation, Qwen2.5-VL-7B
analyzes images on demand via analyze_image LangChain tool.
- services/model/mlx_vision_model.py: MlxVisionModel (mlx-vlm wrapper, lazy load)
- services/agent/tools.py: make_vision_tool(vision_model, image_path)
- agent_service.py: stream_response(image_path=None), dynamic tool binding
via config["image_path"] — thread-safe per-request rebinding
- container.py: vision_model Singleton provider
- config.py: vision_enabled, vision_model_id, vision_max_tokens
- api.py: image_base64 in ChatRequest, decode to temp file, cleanup after stream
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-06-02 13:52:10 +09:00
shinalok
06bcdb03ac
Implement Phase 4~14: LangGraph Agent, RAG pipeline, Gradio Web UI, voice interface
...
- Upgrade LLM to Qwen3-14B-4bit with Thinking mode (MlxChatModel as LangChain BaseChatModel)
- Add LangGraph ReAct agent with tool calling loop (search_documents, web_search, get_current_date, remember/recall_user_info)
- Add RAG pipeline: BAAI/bge-m3 embeddings + Qdrant vector store + semantic chunking (SemanticSplitter via cosine similarity)
- Replace fixed-size RecursiveCharacterTextSplitter with meaning-based SemanticSplitter (numpy only, no extra deps)
- Add Gradio Web UI (app.py): chat, document ingestion, document management tabs
- Add multi-user support (user_id isolation in DB + per-user agent cache + dropdown selector)
- Add conversation history restore from MySQL on agent init (Phase 11)
- Add UserProfileRepository for persistent user profile (remember/recall tools)
- Add thread-local DB connections to fix pymysql thread-safety with LangGraph ToolNode
- Add Phase 14 voice interface: Whisper STT (microphone → text) + macOS TTS (say -v Yuna)
- Enforce search_documents-first policy in system prompt and tool descriptions
- Update ROADMAP2.md: Phase 14 완료, Phase 13 청킹 부분 완료
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-05-27 14:06:22 +09:00
shinalok