Production RAG: what the demo leaves out

Turning 41 research papers into vectors and answering questions from them took a weekend. Hybrid search, a cross-encoder reranker, a graph that retries its own retrieval, 51 hand-written eval questions, and a Python bundle that would not fit under a 500MB limit took everything after that.

The Research Index answers questions about 41 AI research papers, from Attention Is All You Need through DeepSeek-R1, and cites the paper and page behind every claim. The part that sounds like the hard part, turning PDFs into vectors and handing search results to a model, took a weekend. Everything after that took considerably longer, and that is the part worth writing down. All of it is on GitHub, so every snippet below is a real file you can open.

A RAG demo fails in a way you cannot see. It retrieves the wrong three chunks, writes a fluent paragraph from them, and nothing in the output marks it as wrong. Most of the work of making one of these systems production-worthy is making that specific failure visible, then making it rarer. This assumes retrieval is the right call in the first place, which is a separate question I worked through in RAG or long context.

Two pipelines that share nothing but a schema

It helps to stop treating RAG as one system. Indexing runs offline, once per corpus change, with no latency budget, so it can afford to be slow and careful. Query runs online, per question, with a user waiting. The only thing the two share is the shape of what indexing wrote to the store.

That asymmetry is why indexing mistakes are expensive. They are invisible at query time. If chunking split a claim in half, retrieval will look bad, the model will look bad, and nothing in the query path points at the real cause. You fix indexing by reasoning about it up front, because you will not be debugging it later.

Chunking decides what an answer is allowed to be

PyPDFLoader reads each PDF one page at a time and hands back text with source and page attached. That metadata is not a nicety. Citation granularity is decided at load time: if a chunk does not know which page it came from, no amount of prompting will make the model cite a page later. The system's headline feature exists because of a decision made in the first ten lines of the pipeline.

From there, RecursiveCharacterTextSplitter cuts the page text into pieces of about 1000 characters with 200 characters of overlap, which produced roughly 6900 chunks across the corpus. The recursive part is the useful part. It tries to split on paragraph breaks first, then on line breaks, then on sentences, and only falls back to raw character positions when nothing better is available. A naive fixed-size split cuts through the middle of a sentence and leaves you with two chunks that are each individually meaningless.

The overlap covers the boundary case. A claim sitting across a split still appears whole in one of the two neighbouring chunks. You pay for that in duplicated text and a larger index, which is cheap next to silently losing facts at chunk edges.

Retrieval can only ever return a chunk. If the answer does not fit inside one, better ranking will not save you.

The embedding model is the choice you cannot revise cheaply

Every chunk goes through OpenAI's text-embedding-3-small and comes out as a vector. Those vectors are only meaningful relative to each other and to a query embedded by the same model. Swap the embedding model and every vector already in the store becomes noise against the new query vectors. This is not a config change, it is a full reindex of the corpus.

Because of that, embeddings stayed on OpenAI even in production, where the chat model did not. Embedding one question costs a fraction of a cent, so there was no reason to move it somewhere cheaper and accept a different vector space in return.

Dense vectors miss the words that matter most

A dense embedding places semantically similar text close together, which is what you want for a question like how does this approach differ from the previous one, where the question shares almost no wording with the paper.

It is the wrong tool for a rare literal token. An embedding compresses a whole chunk into one point, so a model name, an acronym or a specific number gets averaged away by the thousand characters around it. The chunk containing the exact string you asked about can end up further from your query than a chunk that merely discusses the same topic.

BM25 has the opposite failure mode. It scores on term frequency against literal tokens and understands nothing about meaning, which makes it excellent at the exact match a dense search dilutes and useless at paraphrase. So the index stores both: a dense vector and a BM25 sparse vector per chunk, in the same Qdrant collection.

At query time both searches run, and reciprocal rank fusion merges the two result lists. Each chunk scores 1 divided by a constant plus its rank in each list, and the scores add. Fusing on rank rather than on raw score is the whole trick, because cosine similarity and BM25 scores live on scales that cannot be compared to each other. A chunk that either method ranks highly makes it through, and a chunk both methods rank highly wins.

Qdrant does the fusion itself, so both arms are one round trip. Each prefetch arm asks for four times as many candidates as the final k, which gives the lexical side room to surface an exact-term match the dense arm ranked below the cut.

# src/rag/vectorstore.py
def hybrid_search(query: str, k: int = 5) -> list[Document]:
    prefetch_limit = max(PREFETCH_MULTIPLIER * k, 20)
    response = _client().query_points(
        collection_name=COLLECTION_NAME,
        prefetch=[
            models.Prefetch(
                query=embeddings_model.embed_query(query),
                using=DENSE_VECTOR_NAME,
                limit=prefetch_limit,
            ),
            models.Prefetch(
                query=models.Document(text=query, model=SPARSE_MODEL),
                using=SPARSE_VECTOR_NAME,
                limit=prefetch_limit,
            ),
        ],
        query=models.FusionQuery(fusion=models.Fusion.RRF),
        limit=k,
        with_payload=True,
    )
    return [_document_from_point(point) for point in response.points]

The deployment that would not fit

The first version ran as a single Vercel project, with the Next.js frontend and the FastAPI backend as two Vercel Services, and a local Chroma index hydrated from Vercel Blob on cold start. The Python function bundle came out at 724MB against a 500MB limit. Trimming unused dependencies got it to 699MB, which was progress in the way that walking one mile of a hundred is progress.

The remaining weight was not mine. chromadb hard-depends on onnxruntime, grpcio and kubernetes, all of which exist to support its distributed server mode, none of which run in embedded mode. I tried stubbing them out with empty packages carrying matching version numbers. The resolver did something worse than fail: it quietly downgraded langchain-chroma to a years-old release to satisfy my fake versions. A silent multi-year downgrade of the library that talks to my index is not a risk worth taking on a live app, so I reverted it.

The real fix was architectural. Moving to Qdrant Cloud takes the vector engine out of my process entirely and leaves a thin API client behind, and the bundle dropped to about 114MB locally. Then the deploy failed again. Vercel's effective ceiling for that project turned out to be 225MB post-optimization, and qdrant-client carries its own hard grpcio dependency baked into the main import path, so there was nothing safe left to strip.

The answer was to stop fighting the packaging model. The FastAPI backend moved to Render, which has no equivalent size ceiling, and the frontend stayed on Vercel. That split is the current architecture, and it cost one CORS configuration and nothing else.

The dependency you did not choose is the one that decides whether you can deploy.

Retrieve wide, then rerank narrow

Hybrid search returns about 20 candidates, not 5. That is deliberate, and the reason is that recall and precision are different jobs that one search cannot do well at the same time.

A vector search is a bi-encoder. Chunk vectors were computed months before your question existed, so relevance is approximated as distance in a space that never saw the query. It is fast enough to scan an entire index and coarse enough that its top result is often not the best one available, only a close one.

A reranker is a cross-encoder. It reads the query and one chunk together and scores that specific pair, which lets it notice what distance cannot, such as a chunk that discusses your topic without answering your question. It costs a full model pass per pair, so it can never run over a whole index. It needs a short list, which is exactly what the first stage produced. Cohere Rerank cuts 20 candidates to the 5 that reach the model, and with no Cohere key configured the node passes its input straight through, so a clone of the repo still works without an account.

# src/rag/reranker.py
def rerank(query: str, docs: list[Document], top_n: int = 5) -> list[Document]:
    if not docs:
        return []

    if not os.environ.get("COHERE_API_KEY"):
        return docs[:top_n]

    try:
        response = _client().rerank(
            model=os.environ.get("COHERE_RERANK_MODEL", DEFAULT_RERANK_MODEL),
            query=query,
            documents=[doc.page_content for doc in docs],
            top_n=min(top_n, len(docs)),
        )
    except Exception:
        logger.warning("Cohere rerank failed; falling back to fusion order", exc_info=True)
        return docs[:top_n]

    return [docs[result.index] for result in response.results]

Both early returns hand back the fusion order untouched. A missing key and a failed API call degrade ranking quality, which is a worse answer, rather than degrading the request, which is no answer at all.

A graph, because a chain can only fail forward

Retrieve then generate is a chain, and a chain has one direction. It has no way to express the idea that this retrieval was bad and should be attempted differently, because there is no edge pointing backwards. The best it can do is generate an answer from material it already knows is weak.

So the query path is a LangGraph state machine instead. Retrieve, rerank, grade, and then a conditional edge. The grade node is an LLM asked one narrow question about the retrieved context: does this actually contain the answer. If it does, the graph moves to generation. If it does not, the query gets rewritten and control goes back to retrieval, capped at two retries.

The cap matters as much as the loop. Self-correction without a bound is a system that can spend an unbounded amount of your money on a question that has no answer in the corpus. After two failed attempts the graph gives up and generates anyway, and the generation prompt is written to refuse rather than guess, so the honest output is I don't know.

The whole control flow is one routing function and one conditional edge. Both exits from the grade node are written down in the same place, which is the part a chain has nowhere to put.

# src/rag/pipeline.py
MAX_RETRIES = 2


def route_after_grade(state: RAGState) -> str:
    if state["is_relevant"] or state["retry_count"] >= MAX_RETRIES:
        return "generate_node"
    return "retrieve_node"


...

graph.add_edge(START, 'retrieve_node')
graph.add_edge('retrieve_node', 'rerank_node')
graph.add_edge('rerank_node', 'grade_node')
graph.add_conditional_edges(
    'grade_node',
    route_after_grade,
    {"generate_node": "generate_node", "retrieve_node": "retrieve_node"},
)
graph.add_edge('generate_node', END)

rag_graph = graph.compile()

Generation returns structured output: an answer plus the chunk IDs it drew from. Asking the model to name its sources as data rather than as prose is what makes the citation in the UI a lookup instead of a hope.

The test that proved the wrong thing

I tested the self-correction loop the obvious way, by asking something the corpus could not possibly answer: what is the capital of France. The system replied that it did not know. The loop worked.

It had not worked. The trace showed retries at 1 and the query unchanged, which means the grader had looked at a set of chunks about transformer architectures, been asked whether they answered a question about France, and said yes. The refusal I was celebrating came from the generation prompt's own guardrail, a completely different safeguard that happened to fire at the same point. Two mechanisms, one visible outcome, and the one under test was the one that failed.

The fix was in the grading prompt, which had been too permissive about what counts as relevant. Making it explicit that topical relation is not enough and that the context must literally contain the answer changed the behaviour immediately. Retries started reaching the cap and the query actually got rewritten between attempts.

# src/rag/pipeline.py, grade_node
prompt = (
    "Question: " + state["query"] + "\n\n"
    "Retrieved context:\n" + context + "\n\n"
    "Be strict. Set is_relevant to true ONLY if the context contains specific "
    "information that directly and substantively answers the question. "
    "Being on a loosely related topic is NOT enough - the context must "
    "actually contain the answer. If the context is about something else "
    "entirely, or only tangentially related, set is_relevant to false.\n"
    "If false, rewrite the query (rewritten_query) to be more specific or "
    "use different terminology than the original, so retrieval is more "
    "likely to find relevant chunks."
)
result, _ = invoke_structured(grading_llm, GradeResult, prompt)

The grader returns a rewritten query alongside its verdict, so the retry is not the same search run twice. If the parse fails, the node defaults to relevant and falls through to generation, which has its own refusal guard. That default is the reason the two mechanisms overlapped in the first place, and it is still the right default, because an unparseable grade should not turn into a 500.

A passing test tells you that something passed. It does not tell you what.

Making a public demo safe to leave running

A deployed RAG app with a working API key is a bill waiting for someone to find it. Locally the chat model is gpt-4o-mini through OpenAI. In production the app routes generation through a free OpenRouter model instead, and if the OpenRouter key is missing on a deployed host, it refuses to start.

Refusing to start is the important half. The failure being prevented is not an outage, it is silence: a config change that drops the OpenRouter key, a fallback path that quietly resolves to my own OpenAI credentials, and a bill that arrives at the end of the month reflecting whatever traffic a public URL attracted. Failing closed turns an invoice into an error message.

# src/rag/llm.py
def get_chat_llm(**kwargs) -> ChatOpenAI:
    openrouter_key = os.environ.get("OPENROUTER_API_KEY")
    if openrouter_key:
        return ChatOpenAI(
            model=OPENROUTER_FREE_MODELS[0],
            base_url=OPENROUTER_BASE_URL,
            api_key=openrouter_key,
            max_retries=3,
            timeout=45,
            extra_body={
                "models": OPENROUTER_FREE_MODELS[1:],
                "provider": {"allow_fallbacks": True, "sort": "throughput"},
            },
            **kwargs,
        )
    if os.environ.get("VERCEL") or os.environ.get("RENDER"):
        raise RuntimeError(
            "OPENROUTER_API_KEY is not set. Refusing to fall back to direct "
            "OpenAI billing on the deployed instance - set OPENROUTER_API_KEY "
            "in the host's environment variables."
        )
    return ChatOpenAI(model="gpt-4o-mini", **kwargs)

Checking both host variables was a second pass. Render sets RENDER and Vercel sets VERCEL, and naming only one of them meant the guard silently did nothing on the host it did not name. A cost guard that is quietly inactive on half your deployments is not a guard.

Free is not the same as reliable, and the way it fails is not the way I expected. The deployed app started returning 503 on every request while the same model answered a three-token test prompt from my laptop instantly. The difference was prompt size. A realistic six thousand character grading prompt came back 429 with limit_source set to upstream_provider_shared_pool, and a small one did not.

My first fix had been provider fallback, which lets one request try a different provider serving the same model. That cannot help here. If a model has exactly one provider on the free tier and that provider is saturated, there is nowhere for the request to go. The fallback has to span models, not providers.

So I measured. Four free models, the real grading prompt, one script. minimax-m3 answered in 1.5 seconds with clean JSON, nemotron answered but wrapped its reply in reasoning, and both the Google and z-ai endpoints returned 429. That table took ten minutes and replaced an afternoon of guessing which model to blame.

minimax-m3 is now primary, and the rest go into OpenRouter's models array, which moves to the next id inside a single request rather than spending a retry round trip on a model that is already known to be down.

A strict parser turned a good answer into a 500

Underneath the 503 there was a second bug with the same root cause and a much worse shape. Some queries returned a 500 instead. The trace showed the model had produced a genuinely good answer, in prose, and that with_structured_output had raised a pydantic validation error on it and thrown the whole thing away.

Strict structured output is a reasonable default when you are paying for a frontier model that reliably emits JSON. It is the wrong default for an open-weight model on a free tier, where answering in prose is common enough to be a normal case rather than an error. The parser was correct and the outcome was still a failed request.

# src/rag/llm.py
def invoke_structured(llm, schema: type[T], prompt: str) -> tuple[T | None, str]:
    instruction = (
        f"{prompt}\n\nRespond with ONLY a JSON object matching this schema, "
        f"no prose, no code fences:\n{json.dumps(schema.model_json_schema())}"
    )
    raw = str(llm.invoke(instruction).content).strip()

    candidates = [raw]
    if fenced := _JSON_BLOCK.search(raw):
        candidates.append(fenced.group(1).strip())
    start, end = raw.find("{"), raw.rfind("}")
    if start != -1 and end > start:
        candidates.append(raw[start : end + 1])

    for candidate in candidates:
        try:
            return schema.model_validate_json(candidate), raw
        except (ValidationError, ValueError):
            continue
    return None, raw

Returning the raw text alongside the parse result is the part that matters. It moves the decision about how to degrade out of the parser and into the caller, which is the only place that knows what a failed parse costs. Grading treats an unreadable verdict as relevant and falls through to generation, which already refuses when the context does not support an answer. Generation keeps the prose and pulls out whichever chunk ids appear in it, so the citations survive even when the envelope does not.

The bug hiding behind a CORS error

The deployed app broke in the browser with a message about a missing Access-Control-Allow-Origin header. Nothing about CORS had changed, which was the first clue that CORS was not the problem.

What actually happened was a 429 from the congested free model. That raised an exception the FastAPI handler did not catch, and an unhandled exception does not travel back out through the middleware stack. CORS middleware works by attaching headers to a response on its way out. No response passes through it, no headers get attached, and the browser sees a reply with no CORS headers and reports the only thing it can see. The real error, a rate limit at an upstream provider, never reached the client at all.

The fix was to catch known failure modes in the handler and return an actual JSON response, a 503 with a plain message about the model being busy. Once the failure is a response instead of a crash, it goes through the middleware, the headers attach, and the user reads what went wrong.

The error you see in the browser is usually the last thing that went wrong, not the first.

Numbers, so a change can be argued about

The evaluation set is 51 questions written by hand, in three categories. Single-document factual questions check the basic path. Cross-document comparisons force retrieval to pull good chunks from two different papers in one query, which is where a naive top-k quietly returns five chunks from the same paper. Adversarial and out-of-scope questions have no answer in the corpus and must produce a refusal, because a system that never refuses is a system that will confidently invent something eventually.

One row per question, one file. The out-of-scope rows carry an empty expected_sources, which is the whole point of them:

{"question": "What does FlashAttention avoid storing to save memory in the backward pass?", "ground_truth": "It avoids storing the large intermediate attention matrix computed during the forward pass, recomputing it on-chip during the backward pass instead of reading it back from GPU high bandwidth memory.", "expected_sources": ["flash-attention.pdf"]}
{"question": "Summarize the plot of the movie Inception.", "ground_truth": "Not answerable from the corpus. None of the indexed papers discuss films, so the system should refuse rather than hallucinate a summary.", "expected_sources": []}

Every question runs through the real pipeline, not a mock, and RAGAS scores the actual retrieved context and the actual generated answer on four metrics. Faithfulness asks whether the answer's claims are supported by the retrieved context. Answer relevancy asks whether it addressed the question. Context precision and context recall score retrieval on its own.

# eval/run_eval.py
for i, entry in enumerate(entries, 1):
    state = rag_graph.invoke({"query": entry["question"]})
    retrieved_contexts = [doc.page_content for doc in state["chunks"]]

    samples.append(
        SingleTurnSample(
            user_input=entry["question"],
            response=state["answer"],
            retrieved_contexts=retrieved_contexts,
            reference=entry["ground_truth"],
        )
    )

results = evaluate(
    EvaluationDataset(samples=samples),
    metrics=[Faithfulness(), AnswerRelevancy(), ContextPrecision(), ContextRecall()],
    llm=_judge_llm,
    embeddings=_judge_embeddings,
)

The line that makes this an evaluation rather than a demo is rag_graph.invoke. It is the same graph the API serves, so a retry or a refusal counts against the score like any other answer, and the chunks RAGAS grades are the chunks that actually reached the model on that run.

Splitting retrieval and generation into separate scores is what makes the numbers actionable. A system can be perfectly faithful to bad context, and that reads as a generation problem while being entirely a retrieval problem. The baseline before hybrid search and reranking landed was 0.75 faithfulness, 0.75 answer relevancy, 0.80 context precision, 0.77 context recall. Every run writes its per-question results to disk, so the next change to chunking or ranking gets compared against a number instead of against a few example answers that happened to look fine.

Traces answer the question metrics cannot

Because the whole pipeline is built on LangChain and LangGraph primitives, every run traces to LangSmith with no manual instrumentation. Retrieve, grade and generate each show up as a span with latency, token counts and cost attached.

Evaluation and tracing answer different questions and you need both. An eval score tells you the system got worse. A trace tells you which node did it, what the retrieval returned, how the grader ruled, and what the rewritten query looked like on the second attempt. The retry-loop bug in the previous section was only findable because the retry count and the unchanged query were sitting in a trace next to an answer that looked correct.

What I would reuse

Most of what made this work was ordering, not cleverness. A few decisions I would make again without thinking about them:

• Write the evaluation set before tuning anything, including the adversarial questions. Without a refusal category you will never notice that your system cannot say no.
• Treat retrieval as two stages by default. A wide cheap search followed by an expensive precise rerank beats one search trying to be both.
• Store dense and sparse vectors together from the start. Adding a second retrieval mode later means reindexing the corpus.
• Fail closed on anything that spends money. An app that will not boot is a much better outcome than an app that boots on the wrong credentials.
• Return a real response for known failures. An exception that escapes the handler takes your error message with it.

The instinct worth fighting hardest is the one that skips evaluation. It is the least enjoyable part to build and the only part that can tell you whether the rest of it works. A RAG pipeline without a scored eval set is a pipeline you are hoping works, and hope does not diff against next week's change.

The system is live at ai-research-paper-rag.vercel.app, running on a free model, so it is rate-limited and occasionally quiet. The code, the eval set and the architecture diagram are in the repo.