Skip to content
AgentWiki Docs

Text Extraction Pipeline

Text Extraction Pipeline

When files are uploaded to AgentWiki storage, they pass through an asynchronous extraction pipeline:

flowchart LR
    UP[File Upload to R2] --> Q[Cloudflare Queue Event]
    Q --> EXT[FastAPI Extraction Service]
    EXT --> CHK[Smart Markdown Chunker]
    CHK --> VEC[Cloudflare Vectorize Embeddings]
    CHK --> FTS[D1 FTS5 Keyword Index]
    VEC --> AI[Workers AI Auto-Summarizer]

Smart Chunking Strategy

  • Preserves code blocks and markdown tables intact.
  • Enforces heading hierarchy context on every chunk.
  • Generates 768-dimensional vector embeddings for retrieval.