Building Performance-Optimized Knowledge Bases for Real-Time AI Agents

When your AI agent is on a live voice call, every millisecond counts. A user asks a question, the agent needs to search a knowledge base, retrieve relevant context, and respond naturally, all within the conversational latency budget of a phone call. This isn't a chatbot where users tolerate a loading spinner. This is real-time.

At Sunday, we built a knowledge base system that handles document ingestion, web scraping, chunking, embedding, vector storage, and sub-second retrieval during live conversations. Here's how we designed it for performance at every layer.

Architecture Overview

The system has two distinct paths with fundamentally different performance requirements:

  1. Ingestion path (async, throughput-oriented): scrape websites, parse PDFs, chunk content, generate embeddings, upsert to vector storage
  2. Query path (sync, latency-critical): embed a question, search vectors, return context to the LLM mid-conversation

Treating these as separate concerns is the single most important architectural decision. Every optimization flows from this split.

flowchart TD
  subgraph ING["Ingestion - Background"]
    direction TB
    i1["Scrape & Parse"] --> i2["AI Sanitize"]
    i2 --> i3["Two-Phase Chunk"]
    i3 --> i4["Batch Embed"]
    i4 --> i5["Upsert to Store"]
  end
%%ROW_SPLIT%%
flowchart TD
  subgraph QRY["Query - Real-time"]
    direction TB
    q1["Call in Progress"] --> q2["LLM Queries KB"]
    q2 --> q3["Embed Query"]
    q3 --> q4["ANN Search"]
    q4 --> q5["Top-k to LLM"]
  end

1. Run Embeddings Locally

Most RAG tutorials default to a hosted embedding API. That's a network round-trip on every query, somewhere between 100-300ms that you can't afford on a voice call.

We run a local embedding model on the server instead. The key patterns:

  • Local inference eliminates network latency. Embedding a query takes single-digit milliseconds instead of 100ms+ for an API call. For real-time use cases, this is the single biggest performance win.
  • Lazy-load a singleton model. The model loads once into memory and is shared across all requests. Use thread-safe initialization (double-checked locking) to prevent duplicate loads during startup under concurrent requests.
  • Cache model weights on disk. Avoid re-downloading the model on every deployment. A configurable cache directory keeps cold starts fast.
  • Choose dimensions wisely. Larger dimension models give better retrieval quality but increase storage costs and search latency. We found 768 dimensions to be a good balance. Large enough for quality, small enough for speed.

2. Thread Pool Isolation: Live vs. Batch

This is the optimization that prevents production outages. Embedding operations are CPU-bound and block the async event loop, so they must run in thread pools. But not all embedding work is equal.

We maintain two separate, bounded thread pools. A live pool with more workers for real-time queries, and a smaller batch pool for background indexing:

_live_executor = concurrent.futures.ThreadPoolExecutor(
    max_workers=4, thread_name_prefix="embed-live"
)
_batch_executor = concurrent.futures.ThreadPoolExecutor(
    max_workers=2, thread_name_prefix="embed-batch"
)

async def async_embed_query(text: str) -> List[float]:
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(_live_executor, _do_embed_query, text)

async def async_embed_batch(texts: List[str]) -> List[List[float]]:
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(_batch_executor, _do_embed_batch, texts)

Why does this matter?

  • Live queries always get priority. A user's question is never blocked because a batch indexing job is chewing through 500 document chunks.
  • Batch indexing is naturally throttled. Fewer threads means indexing is slower but can never starve the live path.
  • Bounded pools prevent thread exhaustion. The default behavior in most async frameworks (like asyncio.to_thread) is to use an unbounded executor. Under concurrent load, this spawns unlimited OS threads and crashes the process. Bounded pools queue excess work instead.

Name your thread pools with descriptive prefixes so they show up clearly in thread dumps and monitoring dashboards when diagnosing issues.

3. Two-Phase Chunking Strategy

Naive chunking (split every N characters) destroys context. We use a two-phase approach that preserves document structure:

Phase 1: Split by document headers. Use markdown headers (H1, H2) as natural semantic boundaries. This keeps related content together. A section about "Pricing" stays intact rather than being split mid-paragraph.

Phase 2: Size-constrain each section. Sections that are too long get further split with a recursive character splitter that respects natural text boundaries (paragraph breaks first, then line breaks, then words).

The parameters we landed on:

  • ~1500 character chunks. Large enough to contain complete ideas, small enough for precise retrieval. Too small (256-512) and you lose context. Too large (4000+) and you dilute relevance.
  • ~200 character overlap. Prevents information loss at chunk boundaries. If a key sentence spans two chunks, both chunks will contain it.
  • Hierarchical separators. Split at paragraph breaks first, then line breaks, then spaces. This preserves natural text boundaries rather than cutting mid-sentence.

Preserve header metadata through the pipeline. Each chunk carries its section hierarchy (e.g., "Pricing > Enterprise Plan"), which gets stored alongside the vector and returned during retrieval. When building the embedding text, prepend the header breadcrumb to the chunk content. This way the vector representation captures both the content and its structural position in the document.

4. AI-Powered Content Sanitization

Raw scraped content is noisy. Cookie banners, navigation menus, reCAPTCHA notices, base64 image references. This noise degrades embedding quality and wastes vector storage.

We use Firecrawl for web scraping, which gives us clean markdown. But even clean markdown from a scraper contains boilerplate that doesn't belong in a knowledge base. Before chunking, every document passes through an LLM-powered sanitization step that:

  1. Strips markdown formatting (bold, italic, images) but preserves headers for structure
  2. Removes boilerplate (cookie consent, nav menus, social links, error messages)
  3. Trims excess whitespace
  4. Restructures the remaining text into clean, well-organized content

We use structured output to ensure the result is consistent, clean markdown every time.

The cost is worth it: cleaner input means better embeddings, fewer irrelevant chunks, and more accurate retrieval. We've seen cases where raw scraped content produces 3x more chunks than sanitized content, with lower average retrieval quality.

5. Vector Store Design with TurboPuffer

We use TurboPuffer as our vector database. A few design decisions made a significant difference:

Namespace-based multi-tenancy

Each team's knowledge base gets its own namespace. This provides:

  • Data isolation so queries never leak across tenants
  • Independent scaling so one team's large KB doesn't slow another's queries
  • Simple deletion where removing a KB is just deleting a namespace, no filter-based bulk deletes needed

Use ANN search, not exact search

Approximate Nearest Neighbor search trades a tiny amount of recall for dramatically faster search. For a knowledge base, this is the right tradeoff. You don't need the mathematically perfect top-5 results, you need 5 highly relevant results fast.

Separate sync and async clients

This one bit us in production. We maintain two TurboPuffer clients. A synchronous one for batch ingestion in background jobs, and an async singleton for the live query path:

class AsyncTurboPufferClient:
    def __init__(self):
        self._client = AsyncTurbopuffer(
            api_key=settings.turbopuffer_api_key,
            region=settings.turbopuffer_region,
        )

    async def query(self, namespace, query_vector, top_k=3, filter=None):
        ns = self._client.namespace(namespace)
        results = await ns.query(
            rank_by=("vector", "ANN", query_vector),
            top_k=top_k,
            include_attributes=["text", "url", "title", ...],
        )
        # parse and return results

Using the sync client on the hot path blocks the async event loop. Every concurrent request stalls while one vector query completes. The async client lets the event loop continue serving other requests while waiting for TurboPuffer to respond.

Batch upserts for ingestion

Instead of upserting vectors one at a time, batch them in groups of ~1000. This reduces the number of API calls and lets TurboPuffer optimize its internal writes.

Cache warming at call start

TurboPuffer stores vectors on disk and loads them into cache on demand. The first query against a cold namespace pays a cache-miss penalty. That's fine for a chatbot, but unacceptable on a voice call where the user is waiting for a response.

We solve this by warming the cache before the user starts talking. When a call connects, we receive a webhook event. Before the conversation begins, we identify which knowledge bases the agent has access to and fire off hint_cache_warm() for each namespace in parallel:

_warm_executor = concurrent.futures.ThreadPoolExecutor(
    max_workers=3, thread_name_prefix="kb-warm"
)

async def _handle_call_started(ctx, call_id, call_data):
    # ... resolve the agent and its KB tools ...

    kb_tools = [
        t for t in agent.tools
        if t.enabled and kb_name_from_tool_name(t.name)
    ]
    if kb_tools:
        tp = Turbopuffer(api_key=..., region=...)

        async def warm_one(tool):
            kb_name = kb_name_from_tool_name(tool.name)
            ns_name = kb_namespace(agent.team_id, kb_name)
            loop = asyncio.get_event_loop()
            await loop.run_in_executor(
                _warm_executor,
                lambda: tp.namespace(ns_name).hint_cache_warm(),
            )

        await asyncio.gather(*(warm_one(t) for t in kb_tools))

A few things to note:

  • Warming happens during call setup, not during conversation. By the time the user asks their first question, the namespaces are already hot. The latency is completely hidden.
  • Another bounded thread pool. The _warm_executor is capped at 3 workers. Previously we used asyncio.to_thread (unbounded), and with N knowledge base tools per agent and M concurrent calls, this spawned N*M threads just for warming. Bounded pools keep it predictable.
  • Parallel warming with asyncio.gather. An agent might have 3-4 knowledge bases attached. Warming them sequentially would add up. Warming in parallel means we only pay the cost of the slowest one.
  • Fire-and-forget tolerance. If warming fails for one namespace, we log a warning and move on. The query will still work, it'll just be a bit slower on the first hit. We'd rather start the conversation than block on a warming failure.

6. Tool-Based RAG: Let the LLM Decide When to Search

Rather than searching the knowledge base on every message, we expose each KB as a callable tool that the LLM can invoke. The LLM sees tools like query_kb_FAQ or query_kb_ProductDocs and decides autonomously when to call them.

This is more efficient than always-on RAG because:

  • No wasted searches. If the user says "hello", no KB query is needed.
  • Targeted retrieval. With multiple KBs, the LLM picks the right one based on the question.
  • Parallel tool execution. When the LLM calls multiple tools at once, all queries run concurrently:
tool_results = await asyncio.gather(
    *(_run_tool(tc) for tc in message.tool_calls)
)

When a KB tool is called, the full pipeline runs: embed the query using the live thread pool, search TurboPuffer using the async client, format results with section headers and relevance scores, and return structured context to the LLM. The section breadcrumb (e.g., "Pricing > Enterprise > Features") helps the LLM understand where the information comes from and cite it appropriately.

7. Durable Background Processing

Document ingestion is handled by a step-function framework that provides durability and observability. The pipeline is broken into discrete steps:

  1. Fetch the document (from object storage or scrape the URL with Firecrawl)
  2. Parse the content (PDF parsing, markdown extraction)
  3. Save the parsed content to the database
  4. Generate embeddings and upsert to TurboPuffer

Each step is independently retryable. If embedding fails, the system retries from that step without re-parsing the entire PDF. This matters when parsing is expensive and you're embedding hundreds of chunks.

For websites, we also run a scheduled job that checks daily for sites due for re-scraping. Each website has configurable scrape intervals (days, weeks, months), so knowledge stays fresh without manual intervention.

8. Observability: You Can't Optimize What You Can't Measure

Every performance-critical operation is instrumented with timing metrics:

  • Embedding duration: how long the local model takes (should be <10ms)
  • Vector search duration: TurboPuffer query time (network-bound, typically 20-50ms)
  • Total tool execution duration: end-to-end KB query time (embedding + search + formatting)
  • LLM inference duration: how long the model takes to respond after receiving context

Tag your metrics with dimensions like the embedding model name, vector dimensions, and KB name so you can slice and dice. This lets you catch regressions immediately. If embedding latency spikes, you know to check CPU contention or thread pool saturation.

The Numbers That Matter

For a real-time voice AI agent, the knowledge base query budget is roughly:

Operation Target Approach
Embed query <10ms Local embedding model
Vector search 20-50ms TurboPuffer ANN with async client
Total KB retrieval <60ms Parallel embedding + search
Ingestion per doc Seconds Background step functions

Compare this to a naive implementation using hosted embeddings (100-300ms) and a synchronous vector query (50-100ms). You'd burn 150-400ms before the LLM even starts generating a response. On a voice call, that's the difference between a natural conversation and an awkward pause.

Key Takeaways

  1. Separate ingestion from query paths. They have fundamentally different requirements. Optimize each independently.
  2. Run embeddings locally for the hot path. Network calls for embeddings are an unnecessary latency tax in production.
  3. Isolate thread pools by priority. Batch work must never starve real-time queries.
  4. Chunk intelligently. Two-phase splitting (semantic headers, then size constraints) preserves context that flat character splitting destroys.
  5. Clean your data before embedding. AI sanitization removes noise that degrades retrieval quality.
  6. Let the LLM decide when to search. Tool-based RAG is more efficient than always-on retrieval.
  7. Use async clients on the hot path. Sync I/O in an async server blocks the event loop and kills throughput.
  8. Warm your caches before they're needed. Hide cold-start latency by pre-warming during call setup.
  9. Instrument everything. You can't optimize what you can't measure.

The fastest knowledge base query is the one you don't make. The second fastest is the one that runs entirely on local hardware with pre-optimized data. Design for both.