179 questions

A colleague argues that as instruction tuning and RLHF keep improving, models will get better at ignoring injected instructions, so this risk shrinks on its own over time. Do you agree?Security & SafetyMediumInstruction tuning and RLHF are designed to make models more responsive to instructions provided in the input. Because an indirect prompt injection attack relies on the model…A colleague proposes decomposing every incoming question with the same general-purpose chain-of-thought prompt you’d use for any reasoning problem, skipping a schema check because ‘the model is good at this.’ What’s wrong with that, and what does it cost you?RAGHardRelying on a general purpose prompt without schema validation allows the model to invent relations that do not exist in the graph. Because an empty result set from a hallucinated…A colleague wants to add more decisions to the router (rewrite, source selection, rerank depth). How do you adjudicate?System DesignVery HardAdding decisions to a classifier creates a multiplicative labeling bill. For example, 3 x 2 x 3 x 2 pipelines would require 36 classes, which is unsustainable for both labeling…A customer invokes their right to erasure. Can you delete their data from the model?Data, Privacy & LegalMediumA strong answer splits the object in the first sentence: the records in the datastore and logs are deletable and verifiable, while the memorized associations in the weights admit…A lab claims their MoE model has 70B parameters and matches a 13B dense model in quality. What would you ask before accepting that comparison?EvaluationMediumFirst, clarify if 70B is the total or activated parameter count. If it is 70B total with 7B activated, the training compute comparison is coherent, but the memory requirements…A normalization layer consumes 25% of wall-clock time but less than 1% of total FLOPs. How do you speed it up?Inference & ServingHardBecause normalization layers are memory bandwidth bound, techniques aimed at reducing FLOPs will not provide significant speedups. The most effective strategy is to reduce the…A publisher's license allows ingesting content into a retrieval index but forbids using it for training. How do you improve grounding without violating this?Data, Privacy & LegalHardThe constraint shifts the problem from quality to provenance. Using licensed text in fine tuning pairs puts that data into the model weights, which makes the most popular licensed…A staff engineer says the right fix is a nightly fine-tune, because retrieval adds a network hop and their p99 budget is 400 ms. Resolve it.RAGHardThis dispute represents a trade off between latency constraints and functional requirements. A nightly fine tune fails the primary operational requirement: reliably generating…A teammate benchmarked semantic chunking on our internal eval set and measured a gain of 8 nDCG points. Do you ship it?EvaluationMediumAsk how the evaluation corpus was created. If ground‑truth boundaries exist because passages were concatenated, the experiment measured separability rather than the chunker’s true…A user gets two different, conflicting answers for the same query in the same session. How do you triage this?RAGMediumBefore attempting to edit prompts or ranking functions, you must determine if the variance is due to the model's decoding process or the retrieval system. By pinning the index and…Adjudicate the debate between using RAG for small models versus generative retrieval for large models, and explain what changes if the corpus grows from 10^6 to 10^8 chunks.System DesignVery HardThe debate is resolved by the three store split: the payload, the index, and the model. RAG is effective because it keeps the payload outside the weights. Generative retrieval…An engineer says: we located this model’s induction heads, and on this answer the top one attends to chunk 4, so the answer is grounded in chunk 4. Do you accept that?EvaluationHardThe attention weight is not exclusive evidence of use. Because the final output is a margin over 1,024 head terms plus the direct path, one well behaved addend does not determine…Autoregressive generation is memory-bottlenecked at batch size 1. What are the fundamental architectural alternatives?Inference & ServingMediumAutoregressive (AR) generation is fundamentally limited by memory bandwidth because each token generation requires loading the entire model weight set. Diffusion models offer an…Beyond unit tests, what other kinds of tests belong in an ML CI/CD pipeline?EvaluationMediumA model can pass every unit test and still be unfit to ship, because unit tests check that code executes correctly, not that the resulting model behaves correctly. Data validation…Can you have an agentic control loop with no LLM calls...AgentsVery HardPro previewPreview only -- unlock to read the worked answer.Can you replace a 70B generator with a 7B model plus a vector store?RAGMediumTo evaluate this, split the problem: retrieval provides factual content, but the generator must still possess the reasoning competence to process the retrieved information. A…Compliance requires a document to be unreachable within fifteen minutes of a takedown notice. Our generative retriever compacts weekly. The ML lead says add the docid to a decode-time blocklist and we are done; the compliance lead says that is not deletion. Adjudicate.Data, Privacy & LegalVery HardWhile a blocklist satisfies the fifteen minute unreachability SLA for the retrieval path, it fails to remove the information from the model's parametric memory. The model could…Compute throughput scales 10,000x from K20 to H100 but HBM bandwidth scales only ~100x. What does this imply for algorithm design going forward?Inference & ServingHardBecause compute throughput has scaled much faster than memory bandwidth, the arithmetic intensity threshold for being compute bound has risen significantly. Operations that were…Derive HNSW from a skip list and determine the number of layers for ten million vectors.Retrieval & EmbeddingsHardTo derive HNSW from a skip list, you substitute the skip list's total order with a proximity graph structure. The halting test is modified such that the search stops when no…Design an AI system for a client with a strict overnight processing window.System DesignHardWork backwards from the SLA, not forwards from the model. Start with total volume, the pass through rate expected at each pipeline stage, the throughput each stage needs to…Design an eval suite that simultaneously detects regressions in safety and helpfulness after any data update.EvaluationHardA robust eval suite should include a fixed red team set of approximately 500 harmful prompts to measure safety recall and a fixed borderline set of 500 benign but risky prompts to…Does a high ablation score for a reranker prove it is the best area for investment?RAGHardIt is critical to distinguish between the integral (total contribution) and the derivative (marginal return). While removing the reranker entirely might cause a significant drop…Does semantic chunking (splitting on meaning boundaries rather than fixed size) actually pay off in practice, or is it overkill?Retrieval & EmbeddingsHardIt depends on the corpus, and treating it as a universal upgrade is a mistake. Semantic chunking — using embedding similarity or topic shift detection to decide chunk boundaries…Even with a clean protocol, what is the fundamental limit of human preference evaluation for assessing model quality?EvaluationVery HardThe fundamental limit of human preference is that it captures user sentiment rather than objective correctness. In tasks involving math, code, or factual retrieval, a confident…Explain LangGraph's core concepts: State, Nodes, and Edges.Agent Protocols & ToolsMediumThree concepts, and understanding how they compose is what lets you actually build something with LangGraph rather than just describe it: State — a typed object (a dict, in…Explain the KV cache and why generation has different GPU utilization characteristics than prefill.Inference & ServingMediumDuring prefill, the GPU processes all prompt tokens in parallel using large matrix multiplications, making it compute rooflined. During generation, the model produces one token at…Few-shot is reported to reduce prompt sensitivity, and FormatSpread reports sensitivity persisting as you add shots. Which is wrong?EvaluationMediumFew shot prompting reduces sensitivity by adding margin to the correct continuation, but it does not eliminate it. Because softmax attention normalizes across demonstrations, the…For a RAG product’s truthfulness evaluation, what processes should run on every pull request, what should run nightly, and why?RAGHardIn a Retrieval‑Augmented Generation (RAG) system, truthfulness comprises two aspects: faithfulness (the answer stays true to the retrieved content) and factuality (the answer…For an image captioning model trained on hundreds of millions of noisy, web-scraped image-caption pairs, what data cleaning and decoding choices matter most, and how should you evaluate caption quality?Multimodal & Generative MediaMediumData quality problems in scraped image caption corpora are not random noise you can average away — they're systematic biases that a model will happily learn if you don't filter…Given a FLOP budget of 6 × 10^23, how many parameters should your model have?Training & Fine-tuningMediumTo determine the optimal parameter count, we start from the compute formula C = 6ND. By defining the loss model L = E + A/N^α + B/D^β and substituting D = C/(6N), we can…How do agents communicate with each other in a multi-agent system?AgentsMediumThe mechanism should follow how tightly coupled the agents actually are, not be picked by default: Shared state — all agents read and write one state object. Simple, and fine for…How do you actually evaluate a RAG system end to end — what metrics matter?EvaluationMediumThe reason evaluation has to split into retrieval and generation is that both produce the same visible symptom — a wrong answer — for completely different underlying reasons, and…How do you adjudicate a disagreement between engineers regarding why a specific document is not being retrieved?Retrieval & EmbeddingsHardDo not accept proposed fixes until the failure is localized. By pulling the document by ID and checking the retriever's output, you can determine if the bug is a retrieval failure…How do you adjudicate a request to swap an 8B model for a 30B model to improve quality while maintaining a 500ms p50 latency target?Inference & ServingHardWhen evaluating model swaps, one must distinguish between quality and latency. Latency in a decode bound system is a function of bytes moved per token and the number of tokens…How do you adjudicate between using a prompted LLM reranker and training an instruction-aware cross-encoder?AgentsHardPrompting an LLM is excellent for measuring if intent disambiguation is valuable without the overhead of building a training set. However, it shifts costs from training to per…How do you calculate the FFN dimension for a SwiGLU model with a dmodel of 8192?LLM FoundationsMediumTo determine the FFN dimension for a SwiGLU model with dmodel=8192, multiply by 8/3 to get 21845.3. Rounding this to a multiple of 128 gives 21888. When finalizing this number,…How do you choose the few-shot examples in a RAG prompt?RAGMediumSimple similarity based retrieval is often insufficient because it does not account for the generator's specific performance. By using the generator to score candidates based on…How do you decide between using dense embeddings, BM25, or SPLADE for a large corpus with mixed query types?System DesignHardDo not arbitrate based on preference; scale the arithmetic. Dense embeddings at 768 fp32 are storage heavy and often underperform on rare tokens like part numbers. BM25 is…How do you design attention hyperparameters for a 70B model regarding head count, KV heads, and head dimension?LLM FoundationsHardThe head dimension is generally set to 128, which aligns with modern GPU warp sizes and Flash Attention block requirements. With a model dimension of 8192 for a 70B model, this…How do you do A/B testing on prompts in production?LLMOps & ProductionMediumPrompt A/B testing works the same way as any other experiment — the part people skip is defining the success metric first. Before writing either prompt variant, decide whether…How do you fix a generative retriever that returns document IDs that do not exist in the corpus?Retrieval & EmbeddingsHardThe problem arises because the decoder's search space includes many strings that do not correspond to real documents. Training alone cannot guarantee that the model will never…How do you handle a query that needs information from multiple documents?System DesignHardA single retrieval call only surfaces documents relevant to the query as a whole — it can't answer something that genuinely requires combining facts scattered across separate…How do you handle a requirement for near-real-time searchability when document expansion generation is too slow?Retrieval & EmbeddingsHardTo satisfy both the need for immediate searchability and the requirement for full vocabulary coverage, decouple the indexing process. Index new documents with their literal text…How do you handle the alignment problem in a production AI agent — making sure it reliably does what it's supposed to?Security & SafetyHardMost of what shows up as "misalignment" in a production agent isn't a deep values problem — it's a goal specified vaguely enough that the model fills the gap with something the…How do you implement multi-tenant access control in a RAG system?LLMOps & ProductionHardVector similarity search is permission blind by construction — it finds semantically similar content regardless of who's allowed to see it. Left unaddressed, that means a query…How do you measure whether an AI system you built is actually worth what it costs?LLMOps & ProductionMediumROI conversations about AI systems go wrong most often by counting the wrong side of the ledger — teams tally the sticker price API cost and stop there, ignoring engineering time…How do you pick the learning rate for a new 70B model?Training & Fine-tuningHardThere are two primary paths. The first is standard parametrization: fit a power law on the optimal learning rate versus width from a small scale pilot, extrapolate to the target…How do you respond to a colleague who suggests replacing CrAM with CAG after a security failure?RAGHardBefore swapping to CAG, you must isolate the failure point. CrAM does not compute credibility; it only acts on the provided scores. If a document was correctly scored low…How do you stop an agent looping forever?AgentsMediumAn iteration cap is necessary and not sufficient on its own — it just turns an infinite loop into a truncated failure with no useful output, which is better than a runaway process…How do you test prompts before deploying to production?LLMOps & ProductionMediumTesting a prompt change well means running it through several distinct layers, each catching a different kind of failure. Functional tests check the prompt against a labeled set…How does CLA interact with tensor parallelism and disaggregated KV caching?Inference & ServingHardWith tensor parallelism, each device holds a shard of the KV heads, and downstream layers in a group read the shared shard instead of computing fresh projections, reducing…How does prefix caching interact with continuous batching, and what are the memory management implications?Inference & ServingHardPrefix caching allows the system to reuse KV blocks for shared prompt prefixes, such as system prompts used across multiple requests. When integrated with continuous batching,…How does tiling improve matrix multiply performance on a GPU?Inference & ServingMediumTiling is a critical optimization technique for matrix multiplication on GPUs. By loading blocks of data into shared memory, the GPU can reuse these elements multiple times for…How much memory does your vector index need?Retrieval & EmbeddingsMediumThe interviewer is testing your ability to compute index memory rather than looking it up. For example, a million 768 dimensional vectors in FP32 require approximately 3.07 GB,…How would you audit a proposed training dataset for legal risk before a model release?Data, Privacy & LegalHardTo audit a dataset, you must first categorize every source by its license type. You should specifically flag shadow library or paywalled content that poses higher legal risks. It…How would you design a data curation pipeline that automatically identifies SFT examples likely to cause hallucination, at the scale of millions of examples?Training & Fine-tuningVery HardThe pipeline should utilize a self consistency filter. For each candidate SFT example, sample k completions (e.g., k=5) from the base model using only the instruction. Compute…How would you design an AI system a regulator could act...System DesignVery HardPro previewPreview only -- unlock to read the worked answer.How would you detect and fix length bias in a deployed RLHF model?LLMOps & ProductionMediumTo detect length bias, instrument production request logs to monitor output token counts by query category and compare them against the SFT baseline. If length is inflated,…How would you evaluate whether your post-training run actually improved the model?EvaluationMediumEvaluation must separate the proxy reward used during training from a held out evaluation harness that is harder to game. The harness should include both capability benchmarks and…How would you modify the Chinchilla analysis if you are training a mixture-of-experts (MoE) model?Training & Fine-tuningHardIn MoE models, the total parameter count differs significantly from the active parameter count. To perform a valid Chinchilla style analysis, you must replace total parameters…How would you support a query like ‘what happened right before the manager approved the budget’ against a video index?Retrieval & EmbeddingsHardSince standard embedding techniques often lose temporal ordering, filtering by a timestamp column after retrieval is insufficient because it may return topically relevant but…I swap two rows of the input table and your model’s answer changes. Is that a bug in the model or in your evaluation?EvaluationMediumA relation is a set of tuples, meaning row order should be invariant. The model's sensitivity to row swaps stems from the additive row ID embedding, which injects order dependent…If I tell you the model will serve users in 40 languages, what changes?LLM FoundationsMediumServing 40 languages requires a more robust tokenization strategy. If the tokenizer is trained on an English dominant corpus, non Latin scripts will suffer from high 'fertility'…If the next GPU generation doubles peak FLOPs but keeps memory bandwidth the same, does token generation speed improve?Inference & ServingMediumSince token generation is already memory bound, doubling peak FLOPs provides no benefit to throughput. The ridge point on the roofline model shifts higher, making the memory bound…If you increase the FFN dimension (dff) of a 7B model, what must change to keep the total parameter count constant?System DesignHardIncreasing dff from 11008 to 16384 adds approximately 66 million parameters per layer, totaling roughly 2.1 billion extra parameters for a 32 layer model. To maintain a 7B…Infra wants to freeze the graph index because a full re-extraction costs $2,850 and six hours; product wants same-day updates for a compliance SLA. Adjudicate.LLMOps & ProductionVery HardThe core issue is that the current design treats full reconstruction as the only update primitive. By moving to incremental, dual level indexing, you only update touched entities.…Is dividing the group advantage by the standard deviation in GRPO mathematically valid?Training & Fine-tuningHardWhile subtracting the group mean is a valid baseline, dividing by the standard deviation (σ) is problematic. When σ approaches zero, gradients explode. This normalization gives…Legal has ruled that no customer record may appear in a prompt at inference time. The ML team proposes fine-tuning on those records instead. Adjudicate.Data, Privacy & LegalHardWhile legal concerns about data in prompts are valid due to request logs and vendor retention, fine tuning is not a safer alternative. Facts embedded in model weights cannot be…Legal requires every retrieved chunk to carry a clause citation, while the platform requires uniform 512-token chunks to keep ANN latency predictable. How do you adjudicate?Retrieval & EmbeddingsHardThe requirement for uniform 512 token chunks to maintain predictable ANN latency is based on a misunderstanding of how vector search works. Every chunk, regardless of its original…Legal wants a hard, per-query, auditable fairness guarantee. Engineering wants to keep the doubly stochastic LP because it costs less accuracy. Adjudicate.System DesignHardDeterministic constraints provide an auditable floor but are more expensive in terms of aggregate accuracy compared to amortized LP approaches. The practical resolution is to use…Make the economic argument for "small model plus a datastore" over "one big model with everything baked into its weights" — and name its limit.Retrieval & EmbeddingsVery HardEvery fact baked into a model's parameters costs the same inference time compute to carry around whether that fact gets used once a year or a million times a day — a large model…Multimodal search performs well in evaluation, but employees report occasionally seeing snippets from documents they shouldn’t have access to. Where’s the bug?Security & SafetyHardThe defect lies in the pipeline architecture rather than the filter logic itself. Because permissions are checked after fusion and reranking, restricted documents influence the…One engineer argues RAG’s whole point is shipping a small model because knowledge lives outside the weights. Another says generative retrieval is where the field is going, and it pushes everything back inside. They can’t both be right. Adjudicate - then tell me what changes if the corpus grows from 10^6 to 10^8 chunks.System DesignVery HardThe two engineers are debating different parts of the storage stack. The core benefit of RAG is leaving the heavy document text (the payload) in an external database, allowing the…One team wants HNSW for a 200 M-chunk index, another wants IVF-PQ, and the RAM budget just got cut. Adjudicate.Retrieval & EmbeddingsHardAt 200 million vectors of 768 dimensions, raw float32 storage is 614 GB, and HNSW graph links add significant overhead. The choice between IVF and HNSW is secondary to the need…Our chat model is RLHF-aligned and refuses to output personal data. Is training-data PII still a risk?Security & SafetyMediumAlignment via RLHF changes the model's conditional distribution, but it does not delete the underlying parameters that encode memorized sequences. A refusal behavior is merely a…Our generative retriever returns document IDs that don’t exist in the corpus. What’s wrong and how do you fix it?Retrieval & EmbeddingsHardThe issue is structural rather than statistical. In generative retrieval, the decoder ranges over a large space of possible strings (k^L). If the number of possible generated…PQ made the index 30x smaller and recall@10 dropped six points. You add exact reranking to recover it. What just happened to your budget?Retrieval & EmbeddingsHardReranking requires full precision vectors to be available. Consequently, the 1.09 GB index acquires a 15.36 GB companion in float16, causing the real compression ratio to fall…Random Forest and Gradient Boosted Trees are both ensembles of trees — what actually distinguishes them in practice?Training & Fine-tuningMediumThey sit on opposite ends of the bagging/boosting split, and that shows up in every practical dimension. Random Forest grows deep, mostly independent trees on bootstrapped rows…Research wants a learned per-query stopper. Platform wants a hard cap of two rounds, because p95 is a contractual SLO and variance is the enemy. Adjudicate.RAGVery HardA fixed cap optimizes the tail of the distribution, while a learned stopper optimizes the mean but may widen the tail. If more than 15% of traffic requires three or more hops, a…Retrieval metrics improved after you shipped document-side query expansion, but a stakeholder insists users still can’t find internal documents they know exist. Two engineers disagree on the fix: one wants to regenerate predicted queries with a larger model; the other wants to change the generation prompt. How do you adjudicate?EvaluationHardBefore committing engineering resources to model or prompt changes, you must perform a diagnostic check to localize the failure. By pulling the specific document by its ID, you…Search wants depth 1,000 for long-tail queries. Platform wants 50 to hold p99. Both have data. Adjudicate.System DesignVery HardFind the variable they disagree about, which is the slope of their recall curve on the long tail slice. If long tail queries are served by an exact match heavy sparse path, their…Semantic chunking improved recall@5 on our evaluation set. Should we ship it?RAGMediumBefore shipping, verify that the evaluation set isn't biased; for example, concatenating unrelated documents can create artificial boundaries that flatter semantic chunking.…Should you fund sparse-autoencoder (SAE) features for interpretability or a cross-encoder reranker for performance?LLMOps & ProductionHardWhile SAEs are a principled way to address the oversubscription of the residual stream, they are fitted to a specific model and layer. A reranker is more robust because it…Someone swapped Document 1: for [1] in your context template and grounded accuracy moved four points on a 500-question set. Real, or noise?EvaluationHardTo determine if the change is real or noise, one must first calculate the noise floor. Given a sample size of 500 questions and an accuracy of approximately 64%, the standard…Template A scores 0.024 nats per token and template B scores 0.019. Ship B?EvaluationMediumSensitivity is evidence about accuracy, not a direct measurement of it. A lower sensitivity score (like 0.019 vs 0.024) suggests better stability, but it is a correlation. You…The data team wants tables in a warehouse for text-to-SQL; the search team wants everything in a vector index. How do you adjudicate?RAGHardPartition the strategy based on the question type. Lookup questions are answerable via retrieval if headers are repeated in rows. Aggregation questions, which require properties…The retrieval team just shipped a reranker taking recall@5 from 80% to 92%. One engineer wants to retrain with P = 0.92; another says P no longer matters, ship the always-golden model. Adjudicate.Training & Fine-tuningVery HardThe choice between an 'always golden' model and a RAFT trained model depends on the recall threshold. By solving for the intersection of the two models' accuracies, we find that…There are several ways to get an LLM to produce valid JSON. Rank them by reliability, and say which one belongs in a production pipeline.Agent Protocols & ToolsMediumThe five methods sit on a clear reliability versus control spectrum, and it's worth being precise about what each one actually guarantees rather than treating "JSON output" as one…Thirty percent of your tail queries return zero results. Where do you start?Retrieval & EmbeddingsMediumWhen a significant portion of tail queries returns zero results, the first step is to segment the query population to distinguish between different underlying issues that produce…Two RAG systems both score 66% end-to-end accuracy on the same benchmark. Are they equally good?EvaluationVery HardIt is a common trap to assume equal performance based on benchmark scores without verifying the evaluation methodology. Faithfulness against gold context is significantly easier…Users report the assistant gets 'how many' and 'total' questions wrong, but retrieval recall@20 is 0.94. What do you change?RAGHardThe core failure stems from evaluating aggregation questions ('how many', 'total') with standard top k retrieval metrics like recall@20. Recall@20 measures whether relevant…Walk me through how speculative decoding works and why it speeds things up.Inference & ServingMediumSpeculative decoding operates by having a draft model generate a sequence of K tokens, which are then verified by the target model in a single forward pass. Because the target…Walk me through how you would run an isoFLOP experiment to estimate the compute-optimal model size for your team’s next training run.Training & Fine-tuningVery HardTo conduct an isoFLOP experiment, first select 5 7 compute budgets spanning 2 3 orders of magnitude below your target. At each budget, train 5 8 model sizes ranging from…Walk me through the GRPO training loop.Training & Fine-tuningMediumThe process begins with sampling G responses per prompt to compute rewards. Next, advantages are normalized within the group by subtracting the mean and dividing by the standard…Walk me through what happens when a user hits your LLM API endpoint.System DesignEasyWhen a user hits an LLM API endpoint, the request first undergoes TLS termination and authentication at the gateway. It then passes to the router, which applies routing policies.…Walk through the main families of feature selection methods and when you'd reach for each.Python & DataMediumFeature selection methods split into three families based on how tightly they couple to the downstream model. Filter methods score each feature independently of any model —…Was training a 70B model on 15T tokens (over 200:1 ratio) wasteful compared to Chinchilla's 20:1 recommendation?Training & Fine-tuningMediumChinchilla's 20:1 ratio minimizes training loss per FLOP, whereas Llama 3's approach minimizes total lifecycle cost (training plus serving). For models serving billions of…We handle router uncertainty by escalating to the expensive path. Isn't that enough?Inference & ServingHardA single scalar for confidence conflates two different things: distance from the decision boundary and distance from the training data. An out of distribution query can be far…We replaced BM25 with a fine-tuned dense retriever last quarter. End-to-end quality moved two points and latency is unchanged. Where did the rest of the win go?RAGHardThe retriever shifted the recall curve up by roughly ten points at fixed k. Because depth was held constant, this showed up as a small ceiling improvement instead of a large…What are decorators in Python, and what do they actually get used for in production ML code?Python & DataMediumA decorator sits above a function and wraps it, adding behavior — timing it, retrying it, caching its result — without modifying the function's own code. The value is separation…What are multi-agent systems, and why are they useful?AgentsEasyA multi agent system splits a complex task across several specialized agents instead of asking one generalist to handle every part of it. Writing a research report might become a…What are the accuracy risks of CLA and how do you catch them in evaluation?EvaluationMediumCross Layer Attention introduces an expressiveness trade off regarding the value projections. To monitor for degradation, one should utilize retrieval heavy benchmarks such as…What are the three main failure modes of token-choice top-K routing in MoE models, and how does auxiliary-loss-free load balancing address them?LLM FoundationsHardThe three primary failure modes are: load collapse, where all tokens route to a single expert; token dropping, which occurs when an expert exceeds its capacity; and routing…What did the Chinchilla paper actually change about how people train LLMs, and why do production teams still often deviate from its recommendation?Training & Fine-tuningVery HardBefore Chinchilla, the field had been scaling parameter count faster than training data, on the assumption that was the compute optimal trade off. Chinchilla showed that was wrong…What do you predict when shipping a generative retrieval prototype from a 200k-document set to a 20M-document corpus?Retrieval & EmbeddingsHardAs the corpus grows, the bits available per document decrease. At 20M documents, the model has significantly fewer bits per document compared to the 200k set, likely falling below…What does it mean that LLaMA and PaLM both independently picked SwiGLU?LLM FoundationsMediumThe convergence of LLaMA and PaLM on SwiGLU acts as a cross validation of the architectural choice. Because these labs operated with different data, hardware, and model scales,…What does the original 2017 transformer do differently regarding normalization, and what problem does that cause?LLM FoundationsMediumThe original 2017 transformer utilized post norm, which places the normalization layer after the residual addition. This design forces the gradient back propagating to early…What happens to model quality if you skip language filtering?Training & Fine-tuningMediumWhen language filtering is omitted, non target language tokens consume a portion of the compute budget that would otherwise be used for the target language. This dilution results…What is a reranker, and when do you add one to a RAG pipeline?RAGMediumVector similarity is fast but approximate, and it measures a different thing than relevance: it tells you how close two pieces of text are in embedding space, not whether one…What is a Task in A2A, and what lifecycle states does it go through?Agent Protocols & ToolsMediumA Task is created the moment a client agent sends a request to a remote agent, and it moves through a fixed set of states over its life: submitted when the request first lands,…What is catastrophic forgetting during fine-tuning, and how do you defend against it?Training & Fine-tuningMediumCatastrophic forgetting happens because full fine tuning has no built in mechanism to protect what the model already knows — every gradient step nudges weights toward minimizing…What is CRAG (Corrective RAG) and how does it improve on standard Agentic RAG?RAGMediumStandard Agentic RAG retrieves and generates in sequence, with no check on whether the retrieved chunks were actually any good — if the vector search returns the wrong document…What is DPO, and what is its main advantage over PPO for preference alignment?Training & Fine-tuningMediumDirect Preference Optimization (DPO) works by reparameterizing the RLHF objective, showing that the optimal policy under a KL constrained reward maximization objective can be…What is RL from Verifiable Rewards (RLVR), and why does it produce more reliable training signal than a learned reward model?Training & Fine-tuningMediumRL from Verifiable Rewards (RLVR) utilizes tasks where correctness can be determined by a checker or unit test rather than a human labeled reward model. Because the reward is…What is self-consistency prompting, and why does sampling multiple reasoning paths actually improve accuracy?Training & Fine-tuningMediumSelf consistency is a decode time technique layered on top of chain of thought prompting: instead of generating one reasoning chain greedily, you sample several reasoning chains…What is the difference between C4 and FineWeb, and when would you use one over the other?LLM FoundationsMediumC4 is a 156 billion token dataset based on a single snapshot using heuristic only filtering, making it ideal for reproducibility and controlled ablations. FineWeb, by contrast,…What is the difference between token-choice and expert-choice routing?LLM FoundationsMediumIn token choice routing, each token independently selects the K best experts, which leads to variable load across experts but is highly compatible with standard inference…What is the ReAct pattern, and why is it the foundation of most modern AI agents?AgentsEasyReAct — reasoning plus acting — interleaves three things in a loop: a Thought, where the model reasons about what it needs next; an Action, where it calls a tool; and an…What is the risk of over-filtering for toxicity?Security & SafetyMediumAggressive toxicity filtering can lead to a model that is unable to process discussions regarding violence, discrimination, or explicit content, even when such discussions are…What makes a good instruction-tuning dataset?Training & Fine-tuningMediumA strong instruction tuning dataset prioritizes three core axes. First, it must demonstrate diversity in task types to ensure the model generalizes well. Second, it requires…What problem are state-space models (like Mamba) trying to solve relative to standard attention, and why haven't they fully replaced transformers?Inference & ServingVery HardStandard self attention has two related long context costs: the attention computation itself scales quadratically with sequence length (every token attends to every other token),…What's actually different between HNSW, IVF, and flat search in a vector database?Retrieval & EmbeddingsHardThese three answer the same question — how do you find nearest neighbors without literally comparing against every vector — with different tradeoffs. Flat (brute force) search…What’s wrong with using only Chatbot Arena or MMLU as your evaluation?EvaluationEasyRelying solely on Chatbot Arena or MMLU is insufficient for production grade evaluation. Chatbot Arena suffers from selection bias, as the user base is skewed toward tech savvy…When does an agentic design beat a deterministic pipeline?AgentsHardPro previewPreview only -- unlock to read the worked answer.When does the wrong choice between Stratified K-Fold and Group K-Fold break a model in production?EvaluationHardStratified K Fold and Group K Fold get reached for interchangeably by candidates who haven't hit the failure mode yet, but they solve genuinely different problems. Stratified K…When should an agent run synchronously versus asynchronously, and what does the hybrid streaming approach actually solve?System DesignMediumThe choice isn't ideological — it follows directly from how long the task actually takes, and getting it wrong shows up immediately as either timeouts or a frozen looking UI.…When would you choose pipeline parallelism over FSDP?Inference & ServingMediumFSDP's AllGather per layer pattern costs 3|θ| per step, which can stall on low bandwidth links like 25 GB/s InfiniBand. Pipeline parallelism's activation handoffs are one to two…When would you distill rather than build a funnel?Inference & ServingHardThey solve different shapes of problem. Distillation makes one expensive model cheaper everywhere it runs — the right choice when almost every request genuinely needs the full…Where do lambda, map(), and filter() actually earn their place in ML data processing?Python & DataEasyLambda is for a short function you're only going to use once, typically inline inside a Pandas call, a argument, or an sklearn . applies a function across every element of an…Where does the training data for a complexity classifier come from, given that nobody labels hop counts?RAGHardThe training data is derived by running your available pipelines on existing QA pairs and scoring them using exact match. You keep the cheapest configuration that yielded a…Why can’t you just do gradient ascent directly on the reward model output?Training & Fine-tuningMediumReward models are imperfect, finite capacity proxies for human preference. If you perform direct gradient ascent on the reward model output, the policy will quickly exploit the…Why can't you just put every knowledge source behind one vector index, and once you have several independently governed sources, how do you decide which ones to actually query per request?System DesignVery HardDistinct sources — legal contracts, HR policy, regional wikis, a compliance archive — often can't merge into one index for reasons that survive any amount of engineering effort:…Why did cutting the reranker's candidate pool fail to significantly improve p50 latency?Inference & ServingMediumLatency optimization must be guided by the distribution of compute costs. If the reranker accounts for only a few milliseconds of a multi second budget, reducing its candidate…Why did RoPE (rotary position embeddings) win out over learned absolute position embeddings?LLM FoundationsHardSelf attention has no inherent sense of order — it's permutation invariant unless you inject position information. Learned absolute position embeddings do this by giving every…Why do outliers appear more in larger models?LLM FoundationsHardThe appearance of outlier channels is an empirical phenomenon linked to model scaling. As models grow, they develop specific feature dimensions that function as gating or routing…Why does Adam make memory worse than SGD?Training & Fine-tuningEasyUsing BF16, SGD requires 4 bytes per parameter to store parameters and gradients. Adam, however, requires storing the first moment (4 bytes, FP32), the second moment (4 bytes,…Why does KV cache fragmentation hurt throughput?Inference & ServingMediumWhen memory is fragmented, the system cannot pack as many sequences into the GPU's memory as it could with contiguous allocation. Since LLM inference, particularly in the MLP…Why does RLHF outperform SFT for instruction following at scale?Training & Fine-tuningMediumSFT is bounded by the time and cost constraints of human generation. In contrast, pairwise preference collection allows for the exploitation of the generator validator gap, which…Why does the optimal learning rate decrease as you scale up a transformer?Training & Fine-tuningMediumIn standard transformer training, as the width n increases, the Adam update norm per layer scales at a rate of O(√n) when using a global learning rate. This results in larger…Why doesn’t sequence parallelism alone solve the 128K context memory problem?Inference & ServingMediumWhile SP splits the sequence for pointwise operations, each GPU must still iterate over all s keys during FlashAttention's tile loop. This means KV tensors must be accessible from…Why is autoregressive generation memory-limited rather than compute-limited?LLM FoundationsMediumAutoregressive generation is memory limited because, at a batch size of 1, the arithmetic intensity is far below the compute saturation point of modern GPUs like the H100.…Why is perplexity on a held-out set a poor proxy for downstream task performance, and what should you measure instead?EvaluationMediumPerplexity is a measure of a model's ability to predict the next token averaged over a reference corpus. It is inherently sensitive to tokenization, domain distribution, and…Why is pure vector search not enough for an enterprise...RAGMediumPro previewPreview only -- unlock to read the worked answer.Why is the naive PyTorch GeLU 7× slower than the fused version?LLMOps & ProductionMediumThe naive PyTorch GeLU implementation is slower because it consists of six separate CUDA kernels. Each kernel reads and writes the full tensor to HBM. For a 256 MB tensor at 2…Why might you choose to ship CAG over CrAM even if CrAM has a higher adversarial-injection score?RAGMediumA high adversarial injection score for CrAM indicates it is near Oracle for the specific, narrow threat it was calibrated against. However, this does not translate to performance…Why not train on all two million synthetic query-passage pairs generated by an LLM?Retrieval & EmbeddingsMediumTraining on unfiltered synthetic data introduces noise. A round trip check—training a preliminary retriever on the set and keeping only pairs where the retriever recovers the…Why use a synthetic key-value retrieval task in addition to multi-document QA to evaluate positional effects?EvaluationMediumThe multi document QA task is insufficient on its own because one could argue that middle chunks are simply harder to reason about. The synthetic key value task acts as a control…Why would you build a RAG system over a much larger closed-book model if you cannot guarantee higher accuracy?RAGMediumWhile accuracy is important, the primary value proposition of RAG is the ability to trace generated claims back to specific source documents. A closed book model, regardless of…Write the formula for nDCG@k from memory.EvaluationEasyThe formula for nDCG@k involves calculating the Discounted Cumulative Gain (DCG) and normalizing it by the Ideal Discounted Cumulative Gain (IDCG). The logarithmic discount is…You are evaluating a closed-weight model and the developer claims the test set was not in training data. You have no access to training data. How do you verify?EvaluationHardWhen you lack access to the training data of a closed weight model, you must rely on behavioral probes to detect contamination. An ordering probe checks if the model's preference…You deployed an instruction-tuned model, removed a single whitespace character from the prompt template, and benchmark accuracy dropped double digits. Why is the model this brittle to a change that doesn't alter the meaning of the text at all?Retrieval & EmbeddingsMediumCalling this a "grammar" or "formatting" issue undersells what's actually happening, because the model doesn't process characters or words the way a human reader does — it…You enabled Self-RAG. p50 latency moved four percent and the GPU bill tripled. Explain.Inference & ServingMediumBecause the K candidate continuations are independent, they run as one batch. Since decoding is memory bandwidth bound, batching K candidates costs roughly one step of wall clock…You have a 4,096-GPU budget. Walk me through your parallelism assignment.Inference & ServingVery HardFor a 4,096 GPU budget, the assignment strategy begins by setting tensor parallelism to t=8 within each NVSwitch node. Next, assess if the 512 nodes fit into a single non…You have five modalities and a labeled set for only two of them. How do you set thresholds for the other three?Retrieval & EmbeddingsHardWhen labels are sparse, you should prioritize borrowing thresholds from modalities that utilize structurally similar encoders, followed by manual spot checking for precision.…You have two million unlabeled domain sentences and one week. How do you build a domain sentence encoder and validate it?Retrieval & EmbeddingsHardStart by measuring an off the shelf model on your domain, as the evaluation set is the most valuable artifact. If you proceed with training, use a dropout recipe with specific…You need a real-time model to recommend new connections across a 500-million-user social graph, serving 50k requests per second. Why might reaching straight for a Graph Neural Network be the wrong first move?System DesignMediumThe pattern match is obvious — it's a graph, so use a Graph Neural Network — and that's precisely what makes it a trap. GraphSAGE or GAT is the theoretically appropriate…You switched to binary quantization and recall@10 fell eight points. Diagnose it.Retrieval & EmbeddingsHardLead with the mechanism: Hamming distance estimates θ/π with binomial variance. At d = 768, this variance is roughly ±0.027 on cosine, meaning any pair of candidates closer than…You wire up prefetching, but the profiler still shows the transfer sitting on the critical path and latency didn’t drop. What broke?Retrieval & EmbeddingsHardWhen prefetching fails to reduce latency, it is often because the two operations are not actually running in parallel. If the prefetch logic waits for the final query rewrite to…You're about to sign off on a large pre-training run. What's the actual checklist you walk through before submitting the job, beyond "the model architecture looks right"?System DesignVery HardWork backward from the two ways a run gets wasted: it doesn't fit the hardware, or it finishes and the result is quietly bad. On data, confirm exact and near duplicate…You're distilling a large ensemble into a single small model for low-latency serving. Why is training the student purely on the ensemble's winning label a waste of the ensemble's real value?Inference & ServingMediumIf you take the ensemble's top prediction and treat it as ground truth for standard cross entropy training, that's technically defensible — the ensemble really did predict "cat"…You're handed a compute budget and a target model size. Walk through how you'd actually assign data, tensor, pipeline, and — if the model is MoE — expert parallelism across the cluster.Training & Fine-tuningVery HardStart from what's true regardless of topology: tensor parallelism doesn't consume batch size the way data and pipeline parallelism do, so it's the right tool specifically for…You’re seeing attention heads collapse to attending a single position after 20B tokens. What’s happening and what do you do?Training & Fine-tuningHardQuery and key norms have grown unchecked, producing logits large enough for the softmax to concentrate near one hot. QK norm prevents this by normalizing Q and K before the dot…Your agent's tools return JSON. Where does that break?AgentsMediumTwo places, in practice: Schema drift. A tool's response shape changes upstream — a field gets renamed, a type changes from a number to a string — and the agent silently mis…Your assistant answers the first turn well and the third turn badly. Users say it forgets. What do you check first?RAGMediumA strong diagnostic approach involves inspecting the input at the retrieval stage to see if the query has been corrupted or diluted. Often, the issue is that the current question…Your BM25 index ranks a 40-page onboarding PDF above a two-sentence FAQ entry that answers the question exactly. Which constant do you turn, and which way?Retrieval & EmbeddingsMediumBefore adjusting constants, print the per term contributions to determine if the failure is due to term frequency or document length. If the document is winning because it is long…Your cost-per-query breakdown covers embedding, vector store, reranker, and generation tokens. What’s missing?LLMOps & ProductionMediumA complete cost analysis must account for externalities, specifically the infrastructure costs imposed on the sources you retrieve from. When your system performs a grounding…Your dense retriever scores well offline and badly on real user questions over a technical corpus, and you have no relevance labels. What is the bug and what is the cheapest fix?Retrieval & EmbeddingsHardThe asymmetry between document document training and question document inference is the root cause. You can diagnose this by comparing mean cosines of document document pairs…Your flat index is 30.7 GB and you have 1 GB of RAM for it. Same embedding model, same dimension. What do you do?Retrieval & EmbeddingsMediumFirst, convert the memory budget into bytes per vector. With 1 GB of RAM for 10 million vectors, you have a budget of 100 bytes per vector. Using Product Quantization (PQ) with…Your gradient norm alert fires, but when you inspect the documents at the data cursor position you find nothing obviously wrong. What else could be causing the spike?LLMOps & ProductionHardIf the data appears clean, the issue is likely infrastructure or numerical. First, check for silent data corruption by comparing gradients across replicas. Second, verify the…Your judge holds 78% macro-F1 on the in-distribution gold set, but drops to 61% on a new document domain, though accuracy is 89%. The PM wants to ship anyway. Adjudicate.EvaluationHardThe high accuracy figure is deceptive because it likely stems from the model defaulting to the majority label in a domain where it lacks sufficient exemplars. The significant drop…Your long-form answers are grounded for two sentences and then invent specifics. You already retrieve on the question. What do you change?RAGMediumA strong answer identifies the structural cause: the information need for later sentences is not expressible from the initial question, meaning no amount of initial top k…Your offline eval score dropped 8 points after a routine deploy. What is the first thing you check?EvaluationEasyBefore investigating the embedding model or the LLM, you should recompute retrieval recall@k for both the old and new pipelines. Since the regression could originate in any stage…Your pipeline prepends five 512-token chunks to every prompt and p50 latency is too high. Where is the time going, and what do you change first?Inference & ServingMediumA strong answer performs the arithmetic: 2,560 prefill tokens, 2N FLOPs per token, and approximately 287 ms on one A100 at 40% MFU. The primary goal is to keep the interface…Your prototype performs 50k embeddings in 3ms, but the product requires 20 million. What breaks and how do you change the approach?Inference & ServingEasyScaling to 20 million embeddings increases the memory footprint significantly. Because the arithmetic intensity is only 0.5 FLOP per byte, the bottleneck becomes memory bandwidth…Your RAG system cites multiple credible sources for a query, yet the user wanted a different entity than the one retrieved. What is broken?RAGMediumCredibility and diversity are independent axes. A system can be highly credible but fail to provide coverage for ambiguous queries. Simply increasing the number of sources or the…Your reranker keeps scoring the same handful of documents against new queries all day, but nothing about that computation is cacheable. Why not, and what would you change?Inference & ServingMediumIn a bidirectional cross encoder, the query and document are processed together, meaning no document only artifact exists to store. By switching to a causal decoder reranker, you…Your retriever has perfect recall on documents about the entity named in the question, and the answer is still wrong. What is happening?RAGMediumIn bridge chain questions, the second piece of necessary information is logically dependent on the output of the first. Since the original query does not contain the entity needed…Your search lead wants to retire BM25 for a dense bi-encoder to end vocabulary mismatch. On-call objects that head queries will regress. Thirty percent of traffic is exact part numbers. Adjudicate.Retrieval & EmbeddingsVery HardThe proposal to replace BM25 with a dense bi encoder must be scrutinized for its actual utility. While dense retrieval is effective at absorbing synonymy, it does not structurally…Your target bucket improved by 15 points on n = 100, but your control bucket - queries the new component should never touch - also moved by 4 points. Do you ship?EvaluationHardIt is critical to distinguish between signal and noise. When the control bucket shows movement, it suggests that the observed changes might be due to sampling variance. Performing…Your team replaced a chain-of-thought prompt with a four-stage pipeline and accuracy went down. What do you look at first?LLMOps & ProductionMediumWhen a pipeline change degrades performance, you must isolate the failure point. Check if each stage is producing valid output and whether the correct candidate is being filtered…Your team wants to enable FP8 training end-to-end on H100s to improve throughput. What is your rollout plan and what failure modes do you specifically instrument?Training & Fine-tuningHardThe rollout plan involves: (1) starting with FP8 only in feed forward matmuls, keeping attention and layer norm in BF16/FP32; (2) instrumenting per tensor amax history to detect…Your toxicity classifier was trained on Wikipedia talk-page comments. How does that affect its performance on Reddit or Twitter-style text?Security & SafetyHardDomain shift is the central challenge when moving from Wikipedia talk pages to social media. Because Wikipedia text is generally more formal and less abbreviated than social…μP is derived for SGD on a deep linear network. Why does the Adam rule differ?Training & Fine-tuningVery HardBecause Adam normalizes gradient elements, the effective operator norm of the weight update becomes O(η√nlnl 1) regardless of the raw gradient scale. When you re derive the update…
← All questions
LLMOps & ProductionMedium· LLM Monitoring & Observability

How do you track and optimize LLM costs in production?

Cost tracking only becomes actionable once it’s granular enough to point at a specific decision. Logging total spend tells you the bill is high; logging spend by model, by feature, and by user tells you which feature is eating the budget and which model tier is driving it, which is what actually lets you prioritize where to optimize first rather than cutting costs uniformly and hoping it helps.

Once you know where the spend concentrates, the levers themselves are fairly standard: route more of that feature’s traffic to a cheaper model if quality allows it, add semantic caching if the queries repeat often, compress the prompt and any retrieved context to cut input tokens, cap output length where verbose responses aren’t needed, and take advantage of provider-side prompt caching for repeated system prompts. Combined, these can meaningfully cut a feature’s cost without touching the model it uses at all — but knowing which feature to target first is what the granular tracking actually buys you.

Code playground (Python)