Previously, in Part 5, you learned the agent loop and — crucially — who actually runs the tools.
Let's be honest about this part. You're interviewing for an AI engineer role, not a research position, and nobody expects you to derive backpropagation on a whiteboard. But production AI work touches evaluation, metrics, and data hygiene every single day, so interviews still probe a classic ML floor. What they're checking for is conversational fluency — can you use these words correctly in a sentence about a real system? You don't need a PhD. You need the four questions below, plus a handful of one-line definitions.
"What is overfitting and how do you detect it?"
Why they ask it: Overfitting is the single most common failure in applied ML, and it has a modern twin in LLM work. If you can't explain it, the interviewer worries you'll ship a model — or an eval — that looks great and fails in production.
A strong answer: Overfitting is memorizing instead of learning. Think of a student who memorizes last year's exam papers: perfect on those questions, lost on anything new. A model overfits when it captures the noise and quirks of its training data rather than the underlying pattern, so it performs brilliantly on data it has seen and poorly on data it hasn't. Detection is simple in principle: compare performance on training data against held-out validation data. A large and growing gap — training scores improving while validation scores stall or get worse — is the signature. Fixes, in one breath: more data, a simpler model, regularization, early stopping. Then earn a bonus point by connecting it to 2026 reality: teams overfit prompts too, by tuning against the same small eval set until the prompt aces those twenty examples and nothing else.
Try it: Overfit on purpose. Train an unrestricted decision tree (scikit-learn, ten minutes) on a small dataset and print training accuracy next to test accuracy. Watching 100% train accuracy sit beside 70% test accuracy teaches you more than any definition.
Follow-ups to expect:
- "What's underfitting, then?" Hint: a model too simple to capture the pattern — bad on both sets.
- "How does regularization help, in one sentence?" Hint: it penalizes complexity, nudging the model toward simpler patterns that generalize.
- "Can an LLM evaluation overfit?" Hint: yes — tuning prompts against a fixed eval set is exactly the same disease.
Red flag: "Overfitting is when accuracy gets too high." High accuracy is not the problem — the gap between training and unseen data is. That one word confusion tells the interviewer you've memorized the term without the mechanism, which is itself a little ironic.
"Why do we split data into train, validation, and test sets?"
Why they ask it: This is a question about honesty. Anyone can quote "80/10/10"; the interviewer wants to know whether you understand what each split is for — and whether you know how measurement quietly goes wrong.
A strong answer: Each split has one job. Training data is what the model learns from. Validation data is where you make decisions — picking hyperparameters, comparing models, tuning prompts. Test data is the final honest exam, touched as rarely as possible. The moment you start making decisions based on test results, you've silently turned it into a second validation set, and your headline number becomes a lie you're telling yourself. Then name the killer mistake: leakage — information from validation or test bleeding into training. Duplicate rows landing on both sides of the split, a scaler fitted on the full dataset before splitting, or future data leaking into the past in a time series. Leakage is lethal precisely because everything looks wonderful right up until production.
Try it: For an imaginary customer-churn model, list three concrete ways leakage could sneak in. (Start with: the "days since last login" feature computed after some customers already churned.)
Follow-ups to expect:
- "How would you split time-series data?" Hint: never shuffle — train on the past, validate on the future.
- "Why not just train and test — why the third set?" Hint: you need somewhere to make choices without contaminating the final measurement.
- "What's cross-validation, in one line?" Hint: rotating the validation slice so every data point gets a turn, for a more stable estimate.
Red flag: Reciting split ratios with no reason behind them. The percentages were never the point — honest measurement is. An interviewer would rather hear "70/15/15, because…" than "80/10/10" with a confident nod.
"What are embeddings, really?"
Why they ask it: Embeddings power semantic search and the retrieval half of every RAG system — they are probably the piece of classic ML you'll touch most as an AI engineer. The interviewer wants mechanics, not mystique.
A strong answer: An embedding is meaning turned into coordinates. A model maps a piece of text to a list of numbers — a point in a space with hundreds or thousands of dimensions — arranged so that things with similar meaning land near each other. That's the entire trick: once meaning is geometry, "find related documents" becomes "find nearby points," which is just math (cosine similarity, usually). This is what makes semantic search match "how do I get my money back" to a document titled "refund policy" even though they share no keywords, and it's the engine behind the retrieval step in RAG — the system you designed in Part 4. One practical detail that signals real experience: you must embed queries and documents with the same model, because vectors from different models live in different, incompatible spaces.
Try it: Embed ten sentences with any embedding API — a few about cooking, a few about programming. Compute pairwise cosine similarities and check that the clusters match your intuition. Half an hour, and you'll never hand-wave about embeddings again.
Follow-ups to expect:
- "How is this different from keyword search?" Hint: it matches meaning, not tokens — synonyms and paraphrases just work.
- "Do more dimensions mean better embeddings?" Hint: not automatically — bigger costs more to store and compare; quality depends on the model.
- "Can you compare vectors from two different embedding models?" Hint: no — different spaces, meaningless distances.
Red flag: "The model just understands the text." If your answer contains no coordinates, no notion of distance, and no similarity measure, you've described magic — and nobody hires a magician to debug retrieval quality.
"Precision vs recall — and when do you prefer which?"
Why they ask it: Metrics are where ML meets business decisions. Candidates who can only recite formulas get these two backwards under pressure; candidates who own a story never do.
A strong answer: Precision: of everything you flagged, how much was actually right? Recall: of everything that was really there, how much did you catch? Then anchor each with a story. A spam filter prefers precision — wrongly burying a real email (a false positive) hurts far more than letting one spam message through. Cancer screening prefers recall — missing a real case is catastrophic, while a false alarm costs one follow-up test. When you must balance both in one number, F1 is the harmonic mean of the two — and it's harmonic, not arithmetic, so a terrible score on either side drags it down hard. Bonus fluency: retrieval quality in a RAG system is literally a recall problem — "did the relevant chunk make it into the top-k?"
| Metric | The question it answers | Prefer it when… |
|---|---|---|
| Precision | Of what I flagged, how much was right? | False positives are expensive (spam filter) |
| Recall | Of what exists, how much did I catch? | Misses are expensive (cancer screening) |
Try it: On paper: 100 emails, 20 are spam, your filter flags 15 of which 12 are truly spam. Draw the confusion matrix and compute precision and recall by hand. Two minutes now saves you a blank stare later.
Follow-ups to expect:
- "Why the harmonic mean for F1, not the average?" Hint: it punishes imbalance — you can't hide a terrible recall behind a perfect precision.
- "How does moving a classification threshold trade one for the other?" Hint: raise the bar and precision rises while recall falls.
- "When is plain accuracy misleading?" Hint: imbalanced classes — 99% accuracy on 1% fraud means predicting 'not fraud' always.
Red flag: Definitions that could be swapped without you noticing. If you can't attach precision and recall to a story like the spam filter, the interviewer assumes flashcards — and flashcards crumble under one follow-up.
Five more terms to own in one sentence
You won't get grilled on these, but they will come up in passing, and a crisp one-liner keeps the conversation moving:
| Term | Your one sentence |
|---|---|
| Classification vs regression | Predicting a category versus predicting a number. |
| Supervised vs unsupervised | Learning from labeled examples versus finding structure in unlabeled data. |
| Epoch | One complete pass through the training data. |
| Inference | Using a trained model to make predictions — the part you pay for at runtime. |
| Distillation | Training a smaller model to imitate a larger one, trading a little quality for much cheaper inference. |
One part to go
That's the floor covered — RAG, agents, and now the classic ML minimum. In the finale, Part 7, we put every piece together in a full mock interview, walkthrough style: real questions, strong answers, and a debrief on why each one scores. See you there.