LLM Fundamentals Questions (and Answers That Impress)

Previously, we mapped the AI Engineer role and the interview loop — now we tackle the round that opens almost every loop: LLM fundamentals. These questions look simple. They're not. They're designed to separate people who use LLMs from people who understand them, and the good news is that the understanding fits in one focused read.

If you want a gentler, longer build-up first, our explainer What is a Large Language Model? is the deep-dive companion to this post.

"Explain how an LLM actually generates text."

Why they ask it: This is the foundation everything else rests on. If your mental model is "it looks up answers in its training data," every later answer you give — about hallucination, about RAG, about temperature — will be subtly wrong, and the interviewer knows it.

A strong answer: An LLM doesn't see words — it sees tokens, small chunks of text (roughly word fragments; "understanding" might be two or three tokens). The model does exactly one thing: given all the tokens so far, it produces a probability distribution over what the next token could be. One token is picked from that distribution, appended, and the process repeats — one token at a time — until the response is complete. That's it. Everything an LLM does, from writing code to passing exams, emerges from extremely good next-token prediction learned from enormous amounts of text.

The picking step is where sampling and temperature come in. At temperature 0, the model (almost) always picks the most likely token — output becomes near-deterministic, good for extraction and structured tasks. Higher temperatures flatten the distribution so less-likely tokens get picked more often — more varied and creative, but less predictable. Mention that this is why the same prompt can give different answers on different runs: you're sampling from a distribution, not querying a database.

Follow-ups to expect:

• "Why do LLMs struggle to count letters in a word?" — because they see tokens, not characters.
• "What temperature would you use for an invoice-extraction feature?" — low or zero; you want repeatability, not creativity.
• "Is temperature 0 fully deterministic?" — almost, but not guaranteed across hardware and batching; don't promise determinism.

"What is a context window and why does it matter?"

Why they ask it: Context management is a daily, practical concern in production LLM work. This question tests whether you've actually shipped something — anyone who has, has fought the context window.

A strong answer: The context window is the maximum number of tokens the model can consider at once — system prompt, conversation history, retrieved documents, and the response it's generating all share this budget. Modern models have large windows (hundreds of thousands of tokens), which sounds like the problem is solved. It isn't, for three reasons. First, cost and latency: you typically pay per token, so stuffing the window on every request is expensive and slow. Second, attention quality: models recall information at the start and end of a long context better than material buried in the middle — the well-known "lost in the middle" effect — so more context can mean worse answers. Third, once a conversation or document set exceeds the window, something must give.

That's why real systems do context management: summarizing older conversation turns, retrieving only the most relevant documents (this is half the point of RAG), trimming boilerplate, and putting critical instructions where the model attends best. Saying "the window is big now, so I'd just put everything in" is the answer of someone who hasn't seen the invoice.

Follow-ups to expect:

• "How would you handle a chat that outgrows the window?" — summarize or drop old turns; keep the system prompt and recent turns intact.
• "Where in the prompt would you put must-follow rules?" — near the start (and repeat critical ones near the end for long contexts).
• "Does a bigger window remove the need for RAG?" — no: cost, latency, freshness, and permissions all still argue for retrieval.

"Why do models hallucinate, and how do you reduce it in production?"

Why they ask it: Hallucination is the number-one reason AI features fail in production and the number-one fear stakeholders have. They're testing whether you understand the cause well enough to engineer around it — not just name-drop the term.

A strong answer: Hallucination isn't a bug that will be patched — it's a direct consequence of how generation works. The model predicts plausible next tokens; it has no built-in mechanism to check claims against a source of truth. When the training data is thin on a topic, or the question presumes a fact that doesn't exist, the most plausible-sounding continuation may simply be false — delivered in the same confident tone as everything else.

Reducing it in production is an engineering problem with layered defenses: grounding — give the model the relevant facts at request time via RAG and instruct it to answer only from them; citations — require the model to point at the source for each claim, which both enables verification and measurably discourages invention; an explicit out — tell it that "I don't know" is an acceptable answer, so it isn't cornered into guessing; low temperature for factual tasks; and crucially, evals — a test set of questions with known answers so you can measure your hallucination rate instead of estimating it from vibes.

Follow-ups to expect:

• "Does RAG eliminate hallucination?" — no; it reduces it, and models can still misread or ignore retrieved text — hence evals.
• "How would you measure hallucination rate?" — an eval set with golden answers, plus checking generated claims against retrieved sources.
• "Why does the model sound so confident when wrong?" — fluency and correctness are independent; it's trained to produce fluent text, not calibrated certainty.

"When would you fine-tune vs use RAG vs just prompt better?"

Why they ask it: This is a judgment question. Fine-tuning sounds impressive, and inexperienced candidates reach for it first. Interviewers use this to check whether you pick tools by problem fit — and by cost.

A strong answer: Frame it as an escalation ladder, cheapest first:

The key insight to say out loud: fine-tuning teaches behavior, not facts. It's poor at adding knowledge (and hopeless at keeping it fresh) — that's RAG's job. Most production systems in 2026 are prompt + RAG, with fine-tuning reserved for the few cases with clear ROI. Starting your answer with "first I'd exhaust prompting, because it's the cheapest experiment" is exactly the engineering instinct they're listening for.

Follow-ups to expect:

• "Can you combine them?" — yes, and production systems often do: fine-tuned tone + RAG facts + careful prompting.
• "Why not fine-tune the company wiki into the model?" — stale on day one, expensive to refresh, no per-user permissions, hard to cite sources.
• "When is fine-tuning clearly worth it?" — high-volume narrow tasks where a small fine-tuned model beats a large prompted one on cost.

You've got the foundation

These four answers — generation, context, hallucination, and the prompt/RAG/fine-tune ladder — are the vocabulary every later round builds on. Next, we get practical: making models behave reliably, and proving it. Continue to Part 3: Prompt and Context Engineering Questions.