shinalok/youlbot
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
24 Commits 1 Branch 0 Tags
main
Commit Graph

15 Commits

Author SHA1 Message Date
shinalok a05d2f474e IDEA-8: GraphRAG — NetworkX 기반 지식 그래프 
- td_knowledge_graph 테이블 (user_id, subject, relation, object 트리플)
- GraphService: MultiDiGraph 인메모리 캐시 + MySQL 영속화
- add_relation / query_entity LangChain 도구
- call_model에 그래프 요약 자동 주입 (시스템 프롬프트)
- GRAPH_ENABLED=true 환경변수로 활성화
- requirements.txt에 networkx>=3.0 추가

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 10:08:39 +09:00
shinalok 0b50444e43 IDEA-2/1/5/7: 스마트 알림, 대화 기반 RAG, CRAG, 파라미터 자동 튜닝 
- IDEA-2 스마트 알림: td_reminders 테이블, set_reminder/list_reminders 도구,
  SchedulerService(asyncio 60초 루프, D-7/D-1/D-0 Telegram push),
  FastAPI lifespan 연동, GET /reminders/{user_id} 엔드포인트

- IDEA-1 대화 기반 RAG: IngestionService.store_text() 추가,
  AgentService._maybe_index_conversation() — 응답 후 LLM 판단 → Qdrant 저장
  (CONV_RAG_ENABLED=true 활성화, background task로 응답 속도 무관)

- IDEA-5 CRAG: AgentState에 crag_fallback_used 플래그 추가,
  crag_check LangGraph 노드 — search_documents 결과 없으면 web_search 자동 주입,
  route_after_crag으로 fallback 1회 루프 제어 (CRAG_ENABLED=true 활성화)

- IDEA-7 RAG Auto-Eval: eval/auto_tune.py — API 서버 없이 파라미터 조합별
  context_precision/recall 비교, 최적 설정 추천

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-04 10:04:05 +09:00
shinalok 68f741af72 Phase 17: Multimodal image understanding via analyze_image tool 
Dual-model approach (C): Qwen3-8B handles conversation, Qwen2.5-VL-7B
analyzes images on demand via analyze_image LangChain tool.

- services/model/mlx_vision_model.py: MlxVisionModel (mlx-vlm wrapper, lazy load)
- services/agent/tools.py: make_vision_tool(vision_model, image_path)
- agent_service.py: stream_response(image_path=None), dynamic tool binding
  via config["image_path"] — thread-safe per-request rebinding
- container.py: vision_model Singleton provider
- config.py: vision_enabled, vision_model_id, vision_max_tokens
- api.py: image_base64 in ChatRequest, decode to temp file, cleanup after stream

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-02 13:52:10 +09:00
shinalok 3faf8b09ce Phase 20: RAGAS evaluation suite 
- eval/run_ragas.py: collect contexts (RetrieverService) + answers (/chat API),
  evaluate with faithfulness / answer_relevancy / context_recall / context_precision
- eval/dataset.jsonl: 5 Korean Q&A pairs for initial evaluation
- eval/requirements.txt: ragas==0.2.9, datasets, langchain-google-vertexai
- Evaluator LLM priority: OpenAI > Anthropic > local Qwen3
- Runtime shim for ragas 0.2 / langchain-community 0.4+ vertexai incompatibility

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-01 17:11:00 +09:00
shinalok 589946ab36 Phase 25: RAG sources in collapsible box + Korean thinking enforcement 
- agent_service: yield {"__sources": [...]} token instead of __meta for sources
- agent_service: inject Korean-only rule at top of system message before date
- config.py: strengthen Korean thinking instruction in system prompt
- ROADMAP: add Phase 25 entry

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-01 16:14:59 +09:00
shinalok efc5fe6961 Update ROADMAP: Phase 24 thinking UI + bug 6 TTS fix 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-01 13:55:15 +09:00
shinalok 67821250fd Tag metadata tokens as {\"__meta\"} to separate TTS from progress messages 
stream_response() now yields plain str for actual answer tokens and
{\"__meta\": str} dicts for progress/thinking/source metadata.
Consumers (WebUI, Telegram) can filter __meta tokens for TTS/display.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-31 23:08:14 +09:00
shinalok e9a6d00059 Phase 23: WebUI separation — api.py + youlbot-webui project 
- api.py: /chat SSE done event now includes run_id for feedback linking
  (format: data: {"__done": true, "run_id": "uuid"})
- api.py: Add POST /feedback endpoint with LangSmith integration
- ROADMAP.md: Add Phase 23 documentation

Note: youlbot-webui/ created at /Users/sal/workspace/youlbot-webui/
(separate project, tracked independently)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-30 22:00:22 +09:00
shinalok 2e9e8a33fe Implement Phase 22: REST API (FastAPI + SSE streaming) 
- api.py: FastAPI 앱 신규 생성
  - GET /health, POST /chat (SSE), POST /reset, POST /ingest, GET/DELETE /documents
  - SSE 포맷: data: <JSON 토큰>\n\n / data: [DONE]\n\n
  - Bearer Token 인증 (API_TOKEN 미설정 시 개발 모드)
  - user_id 파라미터로 멀티유저 지원 (기존 AgentService·DB 구조 재사용)
- config.py: api_token 필드 추가
- app.py: _get_agent에 query_rewrite_enabled 누락 수정
- requirements.txt: fastapi, uvicorn[standard], python-multipart 추가
- ROADMAP: Phase 22 ✅, Telegram Bot 클라이언트 예시 추가

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 20:11:49 +09:00
shinalok 432cc9565c Add Phase 21 (Telegram Bot) and Phase 22 (REST API) to ROADMAP 
- Phase 21: python-telegram-bot 직접 AgentService 연결 (동일 머신)
  - /start, /reset 커맨드, 스트리밍 edit_message_text, Telegram user_id → user_id 매핑
- Phase 22: FastAPI + SSE 스트리밍 REST API (원격 Python 클라이언트)
  - POST /chat, POST /ingest, GET/DELETE /documents, Bearer Token 인증
- 우선순위 재조정: Telegram(1순위) → REST API(2순위) → RAGAS(3순위) → 모델선택(4순위)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 20:01:21 +09:00
shinalok e4c56a9b6c Implement Phase 19: Query Rewriting via LangGraph node 
- query_rewrite 노드 추가 (agent → query_rewrite → tools 순서)
- route_after_agent: search_documents 호출 시에만 query_rewrite 라우팅, 그 외 직접 tools
  - tools_condition(prebuilt) 제거 → 커스텀 라우팅 함수로 대체
- query_rewrite_node: 구어체 쿼리를 키워드 중심 문장으로 변환
  - 이전 대화 2턴 컨텍스트로 대명사·지시어 해소
  - enable_thinking=False 바인딩으로 불필요한 사고 과정 제거
  - __query_rewrite 커스텀 이벤트 emit → RAG_VERBOSE 시 변환 결과 출력
- QUERY_REWRITE_ENABLED=true 로 활성화 (기본값 false)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 17:55:13 +09:00
shinalok 86370f6c1e Implement Phase 18: Hybrid Search (BM25 + Vector) 
- FastEmbedSparse(Qdrant/bm25) 기반 sparse 임베딩 추가 (fastembed 패키지)
- IngestionService: HYBRID_SEARCH_ENABLED 시 dense + sparse 동시 저장 (RetrievalMode.HYBRID)
  - _ensure_collection_schema(): sparse vector 미설정 컬렉션 자동 삭제·재생성
- RetrieverService: hybrid 스토어 + dense 폴백 구조, Qdrant 내장 RRF로 결과 통합
- container.py: sparse_embeddings Singleton 프로바이더, ingestion/retriever 양쪽 주입
- .env.example: HYBRID_SEARCH_ENABLED, SPARSE_MODEL_ID 항목 추가

활성화: .env에 HYBRID_SEARCH_ENABLED=true 설정 후 기존 문서 재수집 필요

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 17:47:17 +09:00
shinalok 145b0cc96f Implement Phase 12 feedback, Phase 13 Semantic Chunker, Phase 13-B Reranker, Bug 5 thinking fix 
- Phase 12: FeedbackRepository + td_feedback 테이블, Gradio 👍/👎 이벤트, run_id 추적, LangSmith create_feedback() 연동
- Phase 13: 커스텀 _SemanticSplitter 제거 → langchain_experimental.SemanticChunker 교체, buffer_size/threshold_type 환경변수 적용
- Phase 13-B: RerankService (Cross-Encoder), RetrieverService.search()에 reranker 통합, tools.py as_retriever() → search() 전환
- Bug 5: mlx_chat_model enable_thinking 런타임 오버라이드, agent_service stream_mode=["messages","custom"] 이중 스트림, thinking 토큰 custom 이벤트로 emit
- ROADMAP: LLM 모델명 8B 반영, RAG에 Reranker 추가, 추천 진행 순서 갱신

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-29 17:41:36 +09:00
shinalok e1d7e9cc21 Merge ROADMAP.md and ROADMAP2.md into single roadmap 
- Combine Phase 4~7 history (ROADMAP.md) with Phase 9~14 and bug fixes (ROADMAP2.md)
- Add bug 4 (age calculation) and Phase 13 Semantic Chunker to completed items
- Remove ROADMAP2.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-27 16:11:27 +09:00
shinalok 06bcdb03ac Implement Phase 4~14: LangGraph Agent, RAG pipeline, Gradio Web UI, voice interface 
- Upgrade LLM to Qwen3-14B-4bit with Thinking mode (MlxChatModel as LangChain BaseChatModel)
- Add LangGraph ReAct agent with tool calling loop (search_documents, web_search, get_current_date, remember/recall_user_info)
- Add RAG pipeline: BAAI/bge-m3 embeddings + Qdrant vector store + semantic chunking (SemanticSplitter via cosine similarity)
- Replace fixed-size RecursiveCharacterTextSplitter with meaning-based SemanticSplitter (numpy only, no extra deps)
- Add Gradio Web UI (app.py): chat, document ingestion, document management tabs
- Add multi-user support (user_id isolation in DB + per-user agent cache + dropdown selector)
- Add conversation history restore from MySQL on agent init (Phase 11)
- Add UserProfileRepository for persistent user profile (remember/recall tools)
- Add thread-local DB connections to fix pymysql thread-safety with LangGraph ToolNode
- Add Phase 14 voice interface: Whisper STT (microphone → text) + macOS TTS (say -v Yuna)
- Enforce search_documents-first policy in system prompt and tool descriptions
- Update ROADMAP2.md: Phase 14 완료, Phase 13 청킹 부분 완료

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-27 14:06:22 +09:00