Implement Phase 18: Hybrid Search (BM25 + Vector)

- FastEmbedSparse(Qdrant/bm25) 기반 sparse 임베딩 추가 (fastembed 패키지)
- IngestionService: HYBRID_SEARCH_ENABLED 시 dense + sparse 동시 저장 (RetrievalMode.HYBRID)
  - _ensure_collection_schema(): sparse vector 미설정 컬렉션 자동 삭제·재생성
- RetrieverService: hybrid 스토어 + dense 폴백 구조, Qdrant 내장 RRF로 결과 통합
- container.py: sparse_embeddings Singleton 프로바이더, ingestion/retriever 양쪽 주입
- .env.example: HYBRID_SEARCH_ENABLED, SPARSE_MODEL_ID 항목 추가

활성화: .env에 HYBRID_SEARCH_ENABLED=true 설정 후 기존 문서 재수집 필요

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
sal
2026-05-29 17:47:17 +09:00
parent 145b0cc96f
commit 86370f6c1e
7 changed files with 88 additions and 22 deletions
+13 -5
View File
@@ -14,6 +14,7 @@ from services.ui.cli_service import CliUiService
from services.events.event_bus import EventBus
from services.events.handlers import StreamTokenHandler, StreamEndHandler
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_qdrant import FastEmbedSparse
from services.rag.ingestion_service import IngestionService
from services.rag.rerank_service import RerankService
from services.rag.retriever_service import RetrieverService
@@ -96,6 +97,16 @@ class Container(containers.DeclarativeContainer):
model_kwargs=providers.Callable(lambda c: {"device": c.embedding_device}, config),
)
reranker = providers.Callable(
lambda c: RerankService(c.reranker_model_id) if c.reranker_enabled else None,
config,
)
sparse_embeddings = providers.Singleton(
lambda c: FastEmbedSparse(model_name=c.sparse_model_id) if c.hybrid_search_enabled else None,
config,
)
ingestion_service = providers.Singleton(
IngestionService,
embeddings=embeddings,
@@ -105,11 +116,7 @@ class Container(containers.DeclarativeContainer):
lambda c: c.semantic_breakpoint_threshold_type, config
),
buffer_size=providers.Callable(lambda c: c.semantic_buffer_size, config),
)
reranker = providers.Callable(
lambda c: RerankService(c.reranker_model_id) if c.reranker_enabled else None,
config,
sparse_embeddings=sparse_embeddings,
)
retriever_service = providers.Singleton(
@@ -120,6 +127,7 @@ class Container(containers.DeclarativeContainer):
top_k=providers.Callable(lambda c: c.rag_top_k, config),
reranker=reranker,
rerank_fetch_k=providers.Callable(lambda c: c.reranker_fetch_k, config),
sparse_embeddings=sparse_embeddings,
)
# Phase 3 — LangGraph Agent