Hybrid Search Engine

AgentWiki implements a hybrid search algorithm combining D1 Full-Text Search (FTS5 / BM25) with Vectorize Semantic Embeddings, fused using Reciprocal Rank Fusion (RRF).

The Search Pipeline

flowchart TD
    Q[User Search Query] --> KW[FTS5 Keyword Search (D1)]
    Q --> VEC[Vectorize Semantic Search (Workers AI)]

    KW -->|BM25 Rank List| RRF[Reciprocal Rank Fusion Fusion Engine]
    VEC -->|Cosine Similarity List| RRF

    RRF --> BOOST[Knowledge Graph Connectivity Boost]
    BOOST --> OUT[Ranked Result List]

Reciprocal Rank Fusion Formula

The composite score $RRF(d)$ for a document $d$ is calculated as:

$$RRF(d) = w_{kw} \cdot \frac{1}{k + rank_{kw}(d)} + w_{sem} \cdot \frac{1}{k + rank_{sem}(d)} + w_{graph} \cdot score_{graph}(d)$$

Where:

$k = 60$ (standard smoothing constant)
$w_{kw} = 0.35$ (Keyword weight)
$w_{sem} = 0.30$ (Semantic weight)
$w_{graph} = 0.25$ (Graph connectivity weight)