RAG Design Questions: The New FizzBuzz

Previously, we drilled the prompt and context engineering questions — now we reach the centerpiece of the AI Engineer loop. One question dominates this round, the way FizzBuzz once dominated screening — except this one actually measures the job. Give it the time it deserves: if you can walk this design confidently, the rest of the loop gets easier.

If RAG is new to you, read What is RAG? first — this post assumes the basics and focuses on answering like an engineer.

"Design a chatbot that answers questions over your company's documents."

Why they ask it: Anyone can call a chat API. This question tests whether you can architect — decompose a fuzzy request into a pipeline, justify each decision, and know where it breaks. It's also the most common real project AI Engineers get handed in their first month, so your answer is a preview of your first month.

A strong answer: is a staged conversation, not a memorized diagram. Walk it as a pipeline — ingestion, retrieval, generation, evaluation — pausing at each stage to name the decision and the trade-off. Here's the walkthrough.

Stage 1: Ingestion — chunking and embeddings

Documents (wiki pages, PDFs, tickets) get split into chunks, because you retrieve and feed the model pieces, not whole documents. Chunk size is a genuine trade-off, and saying so earns points: too small and a chunk lacks the context to be understood ("it increased 40%" — what did?); too large and each retrieval drags in noise, wasting context budget and diluting relevance. A sane default is a few hundred tokens with some overlap, but the stronger move is chunking on document structure — headings, sections, paragraphs — so each chunk is a self-contained thought. Attach metadata to every chunk (source document, section, updated date, access group); you'll need it at every later stage.

Each chunk then gets an embedding — a vector capturing its meaning — stored in a vector database alongside the text and metadata. Similar meanings land near each other in vector space, which is what makes semantic search work: a query about "time off" can find a chunk about "annual leave policy" with zero shared keywords.

Stage 2: Retrieval — top-k, hybrid, rerank

At question time, embed the user's question and fetch the top-k most similar chunks — k is a tuning knob, often somewhere around five to ten, balancing recall against noise and cost. Pure vector search stumbles on exact identifiers (error codes, product SKUs, people's names), so production systems use hybrid search: vector similarity plus classic keyword search, results merged. If quality needs another notch, add a reranker — in one line: retrieve generously (say, top 50 cheap and fast), then let a more accurate model reorder those candidates and keep the best few.

Stage 3: Generation — grounding and citations

Now assemble the prompt: the retrieved chunks as clearly delimited sources, plus grounding rules that keep the model honest.

Answer the question using ONLY the sources below.
If the sources do not contain the answer, say
"I could not find this in our documents" — do not guess.
Cite the source id after each claim, like [doc-12].

Sources:
[doc-12] Annual leave policy: employees accrue...
[doc-31] Contractor handbook, section 4: ...

Question: How much annual leave do contractors get?

Citations aren't decoration — they let users verify answers, make failures debuggable, and measurably discourage invention. The explicit "say you don't know" escape hatch matters just as much: without it, the model treats every question as answerable.

Stage 4: Evals — the stage that gets you hired

Most candidates stop at generation. Don't. Say: "and I'd measure this from day one, at two levels." Retrieval hit rate — for a test set of questions with known source documents, how often do the right chunks appear in the top-k? If retrieval misses, nothing downstream can save you, so debug this layer first. Answer faithfulness — is every claim in the answer actually supported by the retrieved sources? (Scored with LLM-as-judge against a rubric, with the caveats from Part 3.) Two metrics, two layers, and you can tell a retrieval bug from a generation bug — that sentence alone puts you ahead of most of the loop.

Then come the follow-ups. Every interviewer has three favorites, and they're delightfully predictable — here they are, each with its own mini-anatomy.

"How do you handle documents that update?"

Why they ask it: Freshness is where demo RAG dies. They want to know you see the index as a living system, not a one-time script.

A strong answer: The vector index is a derived view of the source documents, and like any derived data it needs an update strategy. On document change, re-chunk and re-embed that document and replace its chunks — which is why every chunk carries a source-document id. Trigger via webhooks or change feeds where the source system offers them, falling back to scheduled scans with content hashes to detect changes cheaply. Deletions matter as much as edits: a deleted policy that still answers questions is a lawsuit generator. State the freshness requirement as a product decision — "is an hour of staleness acceptable here?" — and design the pipeline to match.

"How do you stop it answering from documents a user shouldn't see?"

Why they ask it: This is the security question, and it's a trap for candidates who only think in prompts. Real companies have HR files and board decks in the corpus; leaking them via chatbot is a headline, not a bug ticket.

A strong answer: Enforce permissions at retrieval time, in code — never in the prompt. Store access-control metadata (groups, roles, tenant ids) on every chunk at ingestion, and filter every vector query by the requesting user's permissions, so restricted chunks never enter the candidate set. Anything that reaches the prompt is already something this user is cleared to read; the model can't leak what it never saw. Add the operational wrinkle for bonus points: permissions change, so sync ACL updates into the chunk metadata, and mirror the source systems' permission model rather than inventing your own. Asking the model nicely to "not reveal restricted documents" is not access control — a model can be talked out of an instruction, but not out of data it doesn't have.

"The bot still hallucinates — what do you do?"

Why they ask it: Because it will. This tests whether your design has defense in depth or one hopeful grounding prompt.

A strong answer: First diagnose which layer is failing, using the two eval metrics: if retrieval hit rate is low, the model is improvising because the right chunks never arrived — fix chunking, hybrid search, or k. If retrieval is fine but faithfulness is low, tighten the generation side: stricter grounding rules, lower temperature, required citations. Then add guardrails for what remains: a grounding check — an automated verifier comparing each claim in the draft answer against the retrieved sources before the user sees it; a refusal threshold — when top retrieval scores are weak, don't answer at all, return "I couldn't find this" with a pointer to a human; and always show sources, so users can verify and failures surface fast. Close with honesty: the goal is to drive the rate down and make remaining failures visible and cheap — anyone promising zero hallucination is selling something.

Your turn: build the tiny version

Follow-ups to expect (beyond the three above):

• "When would you not use RAG at all?" — when the knowledge is small and stable enough to live in the prompt, or the task needs reasoning, not lookup.
• "How does this scale to millions of documents?" — the same pipeline with a production vector database doing the indexing work; the architecture holds.
• "Where does conversation history fit?" — rewrite follow-up questions into standalone queries before embedding, or retrieval quietly degrades mid-conversation.

This design conversation is the heart of the loop, and you can now hold it end to end. Next we add the layer interviewers are increasingly obsessed with: models that don't just answer, but act. Continue to Part 5: Agent and Tool-Calling Questions.