Prompt and Context Engineering Questions

Previously, we covered the LLM fundamentals questions — now we move to the round where interviewers check whether you can make a model behave reliably. Prompting sounds soft. In an interview it isn't: these questions have precise, engineering-shaped answers, and the last one in this post is the one that quietly decides whether you get labeled "tinkerer" or "engineer."

For a from-scratch tour of these ideas, keep Prompt Engineering Basics for Developers open in another tab.

"How do you get reliable structured output (JSON) from a model?"

Why they ask it: Almost every production LLM feature ends with "…and then parse the response." If your code expects JSON and the model adds a friendly preamble, your feature is down. This question tests whether you've wired a model into real code.

A strong answer: Use three layers, and name them in order. First, ask precisely: put the exact shape in the prompt and say "respond with only valid JSON matching this shape — no markdown, no explanation":

{
  "sentiment": "positive | neutral | negative",
  "confidence": 0.87,
  "reasons": ["max three short strings"]
}

Second, use the platform: modern APIs have structured-output or JSON modes where you supply a schema and the model is constrained to produce output matching it — far stronger than politely asking. Tool/function definitions achieve the same effect: the "arguments" the model produces are schema-shaped JSON. Third — and this is the part that marks you as production-minded — never trust, always validate: parse the response, validate it against the schema in your code, and on failure retry with the error message included ("your last response failed validation: missing field 'confidence' — return corrected JSON only"). One validation-and-retry loop turns a 95%-reliable prompt into a system your caller can depend on, with a logged fallback for the rare double failure.

Follow-ups to expect:

• "What if the model wraps the JSON in a markdown code fence?" — instruct against it, but strip fences defensively before parsing anyway.
• "Schema-constrained mode exists — why still validate?" — constraints guarantee shape, not sense; a syntactically valid answer can still be semantically wrong.
• "How many retries before giving up?" — one or two with feedback, then a typed failure your application handles; infinite retries hide real bugs and burn money.

"What goes in a system prompt vs the user message?"

Why they ask it: It's a quick probe of whether you architect prompts or just concatenate strings until something works. Sloppy separation here predicts sloppy, injection-prone prompt design in their codebase.

A strong answer: The system prompt is configuration — it defines the application, not the request: the model's role ("you are a support assistant for Acme's billing product"), hard constraints ("never quote prices; refer pricing questions to sales"), tone, output format rules, and how to handle edge cases. It's written by you, versioned in source control like code, and stable across requests. The user message carries the per-request task and data: the customer's question, the document to summarize, the record to classify.

Two reasons this split matters in production. Models are trained to weight system-prompt instructions more heavily, so rules placed there hold up better when user input gets weird — which is also your first, imperfect line of defense when untrusted input tries to override your rules ("ignore previous instructions…"). And operationally, a stable system prompt is testable and cacheable, while user content varies per call. A tidy mental model to say out loud: system prompt is code, user message is input — and you never let input redefine the code.

Follow-ups to expect:

• "Where would you put a retrieved document?" — in the user message (or a dedicated content block), clearly delimited as data, never appended to the system prompt.
• "Is a system prompt a security boundary?" — no; it raises the cost of prompt injection but real enforcement belongs in code, outside the model.
• "How do you manage system prompts across environments?" — version control, code review, and evals on every change, like any other code.

"How do few-shot examples change behavior, and when do they backfire?"

Why they ask it: Few-shot prompting is the most powerful cheap technique in the toolbox, and also the easiest to quietly get wrong. The "when do they backfire" half is the real question — it checks for scar tissue.

A strong answer: Few-shot examples work by pattern anchoring: instead of describing what you want, you show two to five input–output pairs, and the model continues the pattern. For anything hard to specify in words — a house style, a subtle labeling rule, an exact output format — showing beats telling, dramatically.

But the model imitates everything about your examples, including the parts you didn't intend as signal. Classic backfires: all your sentiment examples are negative, and the model over-predicts negative; every example answer is two sentences, so the model truncates answers that needed five; an example subtly contradicts a written instruction, and the model follows the example — examples usually beat instructions in a conflict. Examples also cost tokens on every single request, so five marginal examples are a permanent tax. The senior move: treat examples as part of your tested surface — curate them to be diverse and edge-case-covering, keep them consistent with instructions, and re-run evals when you change them.

Follow-ups to expect:

• "How many examples is right?" — usually two to five; measure, because returns diminish while cost doesn't.
• "Examples or fine-tuning for a consistent style?" — examples first; fine-tuning only when examples can't reach it or token cost at volume justifies it.
• "What wins when an example contradicts an instruction?" — typically the example, which is exactly why you audit them together.

"How would you evaluate whether a prompt change made things better?"

Why they ask it: This is the separator question. Tinkerers tweak a prompt, eyeball three outputs, and ship. Engineers measure. Interviewers push this question because eval discipline is the single strongest predictor that your AI feature won't silently degrade in production.

A strong answer: Start by naming the failure mode: outputs are probabilistic, so "it looks better on the two prompts I tried" is noise, and a change that fixes one case can quietly break five others. The fix is an eval set: a collection of representative inputs — real ones where possible, including the ugly edge cases — run against every prompt version, scored the same way each time.

Scoring depends on the task. For extraction and classification you have golden answers, so scoring is exact-match code, cheap to run in CI. For open-ended output — summaries, support replies — use LLM-as-judge: another model grades each output against a rubric ("is every claim supported? is the tone right?"). Immediately add the caveats, because they're where the credibility is: judges drift, favor certain styles, and can share blind spots with the generator — so you spot-check the judge against human ratings before trusting it, and keep the rubric specific. Then compare versions on the same set, watch for regressions, and for changes where business impact matters, confirm with an A/B test on live traffic. If you can also say "prompt changes go through review and the eval suite runs in CI," you've just described a mature LLM team's workflow.

Follow-ups to expect:

• "How big does an eval set need to be?" — dozens of well-chosen cases beat none; grow it from every production failure you triage.
• "What are the weaknesses of LLM-as-judge?" — style bias, drift, shared blind spots; calibrate against human judgment and re-check periodically.
• "When is A/B testing worth it over offline evals?" — when the metric that matters is a user behavior (deflection, satisfaction) that offline scoring can't see.

From behavior to architecture

You can now make a model reliable and prove it. The next round zooms out: designing a whole system around retrieval — the round that has become the heart of the AI Engineer loop. Continue to Part 4: RAG Design Questions — The New FizzBuzz.