AI system design interview questions

About 40 questions interviewers ask about systems built on language models, with the points a strong answer covers.

Each answer is a skeleton, not a script. Ask about scale, quality bar and budget first, name the trade-off, and say how you would measure the result.

LLM basics and cost

What happens between sending a prompt and getting an answer? The text is split into ; the model reads the whole input at once (), then writes one token at a time (). Prefill sets ; decode sets how fast the rest streams.
Why do token counts matter so much? Cost, speed and the are all measured in tokens. Output tokens are slower and dearer than input tokens, so cap answer length and trim what you send.
When would you use a small model instead of the biggest one? Small models for narrow, high-volume jobs (sorting, extracting, short replies) where they pass your tests; large ones for open reasoning. Decide with your own , not leaderboards.
When is a reasoning model worth it? Multi-step maths, planning or code where a wrong answer is expensive. It thinks before answering, so it is slower and costs more; set a per route and keep plain models for simple turns.
Your bill is ten times the estimate. Where do you look? Break spend down per feature and per user (), then check for oversized prompts, runaway agent loops, retries, missing and one user or bot generating most of the traffic.
How do you cut cost without hurting quality? In order: cache stable prompt prefixes, send easy requests to cheaper models (), trim context, batch work that can wait, then consider for a high-volume task. Re-run evals after each step.

Prompting and context

How do you manage prompts in production? Treat them as code: stored in , reviewed, tested against a fixed set before release, and logged with every response so you know which version produced it.
How do you handle conversations longer than the context window? Keep recent turns word for word, summarise older ones (), and move durable facts into you retrieve when relevant.
Why not paste every document into a huge context window? Cost and wait grow with every request, and models miss facts buried mid-context (). Retrieval sends only what matters and respects who is allowed to see what.
When would you choose fine-tuning over prompting? Prompting and retrieval first. fits a fixed style or format, a narrow task at high volume, or shrinking prompts; it does not teach facts that change.
How do you get reliable machine-readable output? Ask for against a , validate every reply in code, retry once with the error message, then fall back to a safe default.

RAG and retrieval

Walk me through a production RAG system. Offline: load documents, split into , turn each into an , store in a with metadata. Online: rewrite the question, search, rerank, build the prompt, answer with .
When would you choose RAG over fine-tuning? when facts change, must be cited, or depend on the user's permissions. Fine-tuning for behaviour and format. Many systems use both.
How do you choose a chunking strategy? Split on the document's own structure (headings, paragraphs, table rows) rather than fixed lengths, keep a little overlap, and attach the title and section to each chunk. Test sizes against real questions.
What is hybrid search and when do you need it? Run and meaning-based search together and merge the lists. matters when users type product codes, names or error messages that meaning-based search blurs.
What does a reranker add, and when would you skip it? A rescores the top few dozen results with a slower, sharper model, which lifts precision. Skip it when the latency budget is tight or retrieval is already good enough on your tests.
How do you keep one customer's documents away from another's? Filter by tenant and permissions inside the database query, before results come back (), never after. Separate indexes for customers who need hard isolation.
The answers ignore the retrieved text. How do you fix it? Check retrieval first (is the right chunk there?), then the prompt: put the sources clearly, tell the model to answer only from them and to say when they don't cover the question, and check .

Serving and scale

What is the KV cache and why does it matter? While writing, the model keeps a working memory of every token so far (). It grows with context length and batch size, and usually decides how many users one can serve at once.
How do you serve many users on one GPU? : requests join and leave the batch at every step instead of waiting for the slowest one, with memory handed out in small pages. Cap queue length and shed load when full.
How do you make a chat reply feel fast? tokens as they come, reuse cached prompt prefixes, keep prompts short, route simple turns to a small model, and set a for each step.
How do you stay up when a model provider goes down? , a few retries with backoff, a , and a on a different vendor that has passed the same evals. Keep a degraded mode (cached or canned answers) for the worst case.
When would you host an open model yourself? When , steady high volume or deep customisation outweigh running GPUs, upgrades and on-call yourself. Below that, an API is simpler.

Agents and tools

What is the difference between an agent and a workflow? A workflow follows steps you wrote in code; an lets the model choose the next step. Prefer workflows when the path is known, since they are cheaper, faster and easier to test.
How does tool use work? You describe each tool with a name and input schema; the model replies with a ; your code checks it, runs it with the user's permissions and returns the result. The model never runs anything itself.
How do you stop an agent from looping forever? A , a spending cap, detecting repeated identical calls, timeouts per tool, and a . Log each step so you can see where it went round in circles.
How do you run agent tasks that take minutes or hours? Save progress after each step () in a , make tool calls safe to repeat with , and let the user leave and come back.
When are several agents better than one? When sub-tasks are truly independent and each needs its own large context. A coordinator hands out work to that return short summaries; otherwise one agent with good tools is simpler.
What is MCP and why does it matter? is a shared standard for connecting models to tools and data, so one tool server works with many apps. Each server is still code you trust: pin versions, scope credentials and review what it can reach.

Evaluation and reliability

How do you evaluate output when there is no single right answer? Write a , score samples with an checked against human grades, and add pairwise comparisons and user signals such as thumbs-down and escalations.
How do you evaluate a RAG system? Score retrieval and generation separately: did the right chunks come back (, ), and is the answer faithful to them and relevant to the question? Build the test set from real user questions.
How do you detect and reduce hallucinations? Ground answers in retrieved sources, require citations and check they support each claim, let the model say "I don't know", and sample production traffic for review. rate is a metric to track, not a bug to fix once.
How do you ship a prompt or model change safely? Run it against a in CI and block the release if scores drop, then try it on or a small before everyone gets it. Pin model versions so nothing changes silently.
What do you log and monitor? One trace per request with a per step (retrieval, model call, tool), plus tokens, cost, latency, model and prompt version. Alert on cost, errors, quality samples and drift.
A system scores 95% accuracy. What do you ask? 95% of what, measured on which data, by whom, and how does it compare with a simple baseline? Is the test set drawn from real traffic and kept apart from what was tuned on? What do the failures cost?

Safety and governance

How do you defend against prompt injection? Assume it will sometimes work. Keep instructions separate from user and document text, give the model the least access it needs, check tool calls in code, and require approval for risky actions. Treat from web pages and files as the bigger threat.
How do you make it safe to let an agent run code? Run it in a with no network by default (or an ), no secrets, limits on time, memory and disk, and a fresh environment each run.
How do you guard an agent that takes real-world actions? , limits on amounts and rates, human approval for anything costly or irreversible, an of every action, and a way to undo.
What does a responsible launch need? Clear notice that people are talking to AI, labels on generated media, on input and output, a written policy for wrong answers, and records of what data and models you use.

Design prompts

Design a customer support assistant. Retrieval over help articles, read-only tools to look up orders, human handoff with the full conversation, tight limits on refunds, and measured resolution and escalation rates.
Design search over a company's internal documents. Connectors that keep permissions, incremental indexing, hybrid search plus reranking, answers with citations, and access checks inside the query.
Design a document processing pipeline (invoices, forms). and layout first, then extraction into a schema, validation rules, confidence thresholds that send uncertain fields to a review queue, and a so spikes wait instead of failing.
Design a code review assistant. Read the diff plus the files it touches, comment only on high-confidence issues, never merge on its own, and measure how often developers accept its comments.
Design real-time content moderation. Fast cheap on everything, a model for borderline cases, humans for appeals and the hardest calls, and per-region policy rules.

Hint

A useful order for any of these: scope and clarify, rank the requirements, draw the simple version, go deep on the riskiest part, then cover how you'd measure it and what happens when it fails.