Key Takeaways
Build products around foundation models, rather than starting by training one
Start with a capable base. AI engineering means building applications with foundation models that already exist, rather than creating a model from scratch. Self-supervised learning let language models absorb vast amounts of text without hand-labeling every example. Their capabilities expanded as models grew and began working across images, audio, and other data. Model APIs then made these systems accessible to teams without enormous training budgets. The engineering challenge shifted: adapt a general model to a real task through instructions, relevant information, evaluation, and, when needed, further training.
Ship a product, not a demo. A weekend prototype can look impressive while leaving months of work in reliability, safety, cost, and user experience. LinkedIn reported reaching 80% of its desired experience in one month, then needing four more months to exceed 95%. Start with the product and its users, measure whether it works, and invest in more complex model work only when evidence justifies it.
What's striking is how the center of gravity moves from algorithm invention to system judgment. That resembles the history of other general-purpose technologies: access creates opportunity, but useful products depend on fitting capabilities to specific institutions and users. The accessible API is not the finished product; it is closer to a new material that engineers must shape. The 80-to-95% gap is especially important because demos conceal operational demands. Still, the product-first approach has limits: where errors have severe consequences, careful validation must precede exposure to real users.
Build evaluation before choosing a model or polishing a convincing demo
Define useful first. Evaluation-driven development means deciding how success and failure will be measured before building the application. A customer support bot, for example, might need to answer accurately, keep responses safe, and resolve a defined share of requests. Those are different criteria, and a strong score on one cannot stand in for the others. Test the full application and its components: if a system extracts an employer from a resume, separately check text extraction and employer identification.
Trust local evidence. Public benchmarks help eliminate weak candidates, but they may omit your task, use unclear scoring, or contain examples that appeared in a model's training data. Build a representative test set from real use, include difficult and out-of-scope cases, and track results by user group or input type. A 100-example test may also be too small to distinguish a real improvement from noise.
This approach connects naturally to measurement theory: a metric is an instrument, not the underlying value. A chatbot's factual score matters only insofar as it predicts customer outcomes, and even a carefully chosen measure can overlook harms concentrated in a subgroup. Component-level tests help diagnose causes, while end-to-end tests preserve the actual user goal. The tension is cost. Extensive evaluation can delay learning, so a sensible practice is staged testing: begin with cheap checks, then reserve expert review for decisions with meaningful risk.
Treat each model response as a probability, not a promise
Generation involves choice. A language model estimates probabilities for possible next tokens, then samples from them to produce a response. That makes identical queries capable of producing different answers. Temperature adjusts the balance between predictable and less common choices; a low setting generally makes responses more consistent, while a higher one can increase variety. Sampling several candidate responses and selecting one with a verifier can improve performance, but it costs additional computation.
Plausibility is not proof. A model can continue an unsupported claim it has already generated, producing a cascade of invented details. It can also imitate knowledge in training examples without reliably knowing when that knowledge applies. Concise answers, fixed generation settings, retrieval, and verification can reduce risks, but no single prompt guarantees factuality. For high-stakes outputs, check claims against trusted evidence rather than treating confidence or fluent writing as evidence.
The probability framing is useful because it replaces the misleading picture of a database returning stored facts with a process that constructs likely continuations. It also explains why variation is a design choice: creative work may benefit from surprise, while a classification pipeline usually values stability. Yet probabilistic generation alone does not explain every hallucination; training incentives, missing evidence, and system design matter too. The practical implication is not to distrust every response equally, but to calibrate verification to the consequence of being wrong.
Retrieve evidence the model needs instead of flooding it with context
Add query-specific knowledge. Retrieval-augmented generation, or RAG, retrieves relevant material from an external source and supplies it to a model as context for generating an answer. It is useful when facts are private, current, or too extensive to fit into the model's stored knowledge. A product support system might retrieve the instructions for a particular device before answering a customer's question.
Retrieval is its own product. Documents are often divided into chunks so the system can retrieve useful passages rather than entire files. Term-based methods such as BM25 are fast and provide a strong baseline; embedding-based retrieval can match meaning beyond exact wording. Combining them, then reranking candidates, can improve results. Measure retrieval quality as well as answer quality. Longer context alone is not a cure: irrelevant passages consume resources and can distract the model.
RAG makes a model less like a self-contained expert and more like a reader working with a prepared reference shelf. That distinction helps separate memory from evidence: a retrieved source can be inspected, updated, and removed without retraining the model. It also creates a new failure boundary. A correct generator cannot answer from a source it never received, and a relevant-looking passage may still be wrong or outdated. Retrieval quality therefore deserves independent testing, not merely a vague belief that adding search makes answers reliable.
Write prompts like specifications, and treat every input as untrusted
Make the request testable. Prompt engineering is the practice of shaping a model's behavior through its input rather than changing its weights. Clear instructions, relevant context, examples, and explicit output requirements make expectations easier to follow. For a complex customer request, separating intent classification from response generation can make each step easier to test and debug. Version prompts and evaluate changes against the same examples, because a clever wording change can improve one case while harming the whole application.
Instructions are attack surfaces. A prompt injection is hostile text that tries to override the application's intended instructions, potentially through a user message, retrieved document, or tool output. Treat retrieved content as untrusted, limit tool permissions, filter sensitive inputs and outputs, and require human approval for consequential actions. A prompt that says not to reveal private information is useful, but it is not a security boundary by itself.
The specification analogy is powerful because it encourages observable requirements instead of mystical prompt tricks. It also links prompt design to security engineering: data that enters a system can carry instructions, not just facts. The crucial nuance is that clearer language improves reliability but cannot establish authorization. If a model can send money, execute code, or expose records, safeguards belong in the surrounding system, where permissions can be enforced independently of the model's interpretation.
Use retrieval for missing facts and finetuning for repeatable behavior
Diagnose the failure first. Finetuning continues training a previously trained model so it behaves better on a target task. If the model gives wrong or outdated facts, RAG can provide current evidence. If it has the information but repeatedly ignores the requested format, style, or task procedure, finetuning may help. Prompt experiments should come first: they are cheaper, reveal failure patterns, and help establish evaluation and data standards.
Adapt efficiently. Parameter-efficient finetuning changes a small portion of the trainable parameters rather than updating the entire model. LoRA, or Low-Rank Adaptation, adds small trainable matrices whose updates can be combined with the original weights. It can make adaptation more accessible and support multiple task-specific variants, though it may not match full finetuning in every case. Finetuning also needs good examples; poor annotations can teach the wrong behavior.
This distinction is a practical form of diagnosis: missing information calls for an information channel, while a repeated behavioral failure may call for changing the model. It prevents the common mistake of using expensive training to solve a retrieval problem. The boundary is not absolute, since RAG can alter behavior by changing context and finetuning can affect factual reliability. A disciplined team tests interventions against the same task-specific evaluation set, rather than assuming that a fashionable technique is the right treatment.
Give agents tools gradually, and keep consequential actions reviewable
Tools expand the action space. An agent is a system that perceives an environment and acts within it. A model can use tools such as search, a calculator, a database, or a code interpreter to gather information and accomplish tasks. A useful agent must plan, execute, inspect results, and revise its approach when needed. The ReAct pattern interleaves reasoning, actions, and observations so the system can respond to what its tools return.
Small errors compound. If each of ten steps succeeds 95% of the time and failures are independent, the chance of every step succeeding is about 60%. More tools can improve capability but also create new ways to fail, including invalid parameters, mistaken plans, and unsafe actions. Start with read-only tools, test tool calls and outputs, and place approval gates around actions that alter records, send messages, or spend money.
Agents bring together planning theory and software permissions. The more consequential the action, the less reasonable it is to rely on a model's self-assessment alone. Human approval is not simply a temporary obstacle; it can serve as an explicit authority boundary. The multiplication example assumes independent step failures, which real systems may not satisfy, but its lesson survives: long chains need stronger monitoring and recovery than single responses. A modest agent that reliably completes a bounded task can be more valuable than a dazzling general-purpose demo.
Curate for quality and coverage, not merely a mountain of training data
Data teaches the target behavior. A useful dataset needs quality, coverage, and enough examples for the task. High-quality examples should be relevant, consistent, correctly formatted, and compliant with privacy and licensing rules. Coverage means representing the languages, topics, formats, and difficult cases users actually bring. Common Crawl, for example, is vast but reflects internet imbalances: English accounts for about 46% of its data, while some widely spoken languages are barely represented.
Synthetic data needs inspection. Models can generate instructions, examples, and simulated tool interactions, expanding scarce datasets or filling gaps. The resulting data should be verified, deduplicated, and tested for factual errors and bias. Repeated training on generated material may cause models to lose rare patterns, a risk called model collapse. Combining synthetic examples with reliable human data and tracking data sources helps preserve quality and lineage.
Data curation is a form of governance as well as optimization. Decisions about which languages, communities, and edge cases appear in a dataset influence who receives competent service. Synthetic data can help address scarcity, but it may reproduce the blind spots of its generating model and make those blind spots look more abundant than they are. The useful question is not whether data is human-made or machine-made in isolation, but whether its provenance, coverage, and downstream effect can be examined.
Measure quality, latency, and cost together in the running system
Performance is a product property. For language models, time to first token measures how long users wait for the response to begin; time per output token measures the pace of what follows. Batching can lower cost by serving requests together, while caching can avoid repeated work. Quantization reduces the memory used by model weights, and speculative decoding uses a smaller draft model to propose tokens for a larger model to verify. These methods involve trade-offs, so compare them under realistic workloads.
Close the learning loop. Log prompts, model versions, retrieved sources, tool calls, and intermediate results so failures can be traced. Monitor user feedback, but interpret it cautiously: users may reward verbosity, click randomly, or provide feedback mainly when unhappy. A system's popularity signal is not automatically evidence of accuracy or benefit. Protect user privacy and connect product measures to the outcomes the application is meant to improve.
This systems view connects AI engineering to reliability engineering: performance depends on interactions among components, not only the model's benchmark score. Latency and cost are not secondary details when they determine whether a feature can be used at all. Feedback adds a further complication because the product shapes the data it collects; recommendations influence clicks, and clicks then influence recommendations. Monitoring should therefore pair user signals with independent quality checks, especially when incentives reward engagement more readily than correctness.
Analysis
The book's central argument is that foundation models change who can build AI, but they do not repeal the disciplines that make software dependable. The cost and expertise of pretraining concentrate model creation in a few organizations. APIs and open-weight models distribute application building much more widely. As a result, competitive work moves toward adaptation: choosing a model, constructing its context, setting its behavior, testing its failures, and fitting it into a product.
Its strongest organizing principle is diagnostic rather than prescriptive. A weak answer may reflect missing evidence, poor instructions, a limited model, defective retrieval, bad training examples, or a serving bottleneck. These causes require different interventions. RAG cannot teach a model a reliable response format by itself, and finetuning cannot supply tomorrow's changing facts. Evaluation is the mechanism that helps distinguish these cases. This makes measurement an engineering input, not merely a report at the end.
The book also treats deployment as a chain of trust boundaries. Inputs can contain malicious instructions, retrieved material can be wrong, tools can execute harmful actions, and user feedback can encode bias. Each additional capability expands both usefulness and the system's possible impact. The sensible response is not to make every system maximally autonomous, but to grant abilities in proportion to verified performance and consequence.
A productive tension remains between speed and assurance. Fast iteration exposes a product to actual use, while rigorous evaluation reduces avoidable harm. The strongest synthesis is staged deployment: begin with narrow tasks, create representative tests, observe real use, and widen autonomy only as evidence accumulates. That approach is less glamorous than a universal agent, but it better matches the book's central lesson: a capable model is only one component in a reliable application. The durable expertise lies in making the whole system useful, measurable, and maintainable as models and user needs change.
Review Summary
AI Engineering receives overall positive reviews, with readers praising its comprehensive coverage of AI application development using foundation models. Many appreciate the book's practical insights, clear explanations, and focus on evaluation pipelines. Readers find it valuable for understanding the current state of AI engineering, though some note its rapid evolution may lead to partial obsolescence. The book is recommended for both beginners and practitioners, offering a balanced mix of broad overview and in-depth exploration of key concepts.
Glossary
AI engineering
Building applications with foundation modelsThe process of building applications on top of readily available foundation models. It emphasizes adapting and evaluating existing models through prompts, context, retrieval, tools, and finetuning, rather than primarily developing a model from scratch. It also includes product interfaces, deployment, inference efficiency, monitoring, and feedback.
Foundation model
A broadly adaptable pretrained modelA large pretrained model that can support many downstream tasks rather than one narrowly defined task. The book uses the term for both large language models and large multimodal models. Developers adapt these models to particular applications with instructions, additional context, tools, or further training.
Evaluation-driven development
Define success before buildingAn approach to AI application development that establishes evaluation criteria before implementation. The team specifies what useful and harmful outputs look like, chooses methods and data for measuring them, and uses the results to guide model and product decisions. Evaluation is applied to components, end-to-end tasks, and production behavior.
Retrieval-augmented generation
Retrieve evidence before generatingA system pattern that retrieves information from an external source and provides it to a generative model as context for a response. The retrieved material may come from documents, databases, or other sources. Its effectiveness depends on retrieving relevant information and assessing both retrieval quality and the generated answer.
AI as a judge
Use a model to evaluate outputsAn evaluation method in which an AI model scores, compares, or classifies responses from another model. It can assess subjective qualities without requiring a complete set of reference answers, but results depend on the judge model, prompt, and scoring criteria. Bias, inconsistency, cost, and changing judge versions require oversight.
LoRA
Low-rank model adaptationLow-Rank Adaptation is a parameter-efficient finetuning method. It represents an update to a model weight matrix using two smaller trainable matrices, while keeping the original weights fixed. The smaller updates can be combined with the original weights for inference, allowing task-specific adaptation with fewer trainable parameters than full finetuning.
ReAct
Interleave reasoning and actionsAn agent pattern that alternates a model's reasoning, tool actions, and observations from the environment. The model uses each observation to decide what to do next, continuing until it judges that the task is complete. The cycle supports multi-step work, but its plans, tool calls, and stopping decisions must be evaluated.
Chinchilla scaling law
Relate model size to training dataA compute-optimal training rule derived from experiments with language models. Under the assumptions discussed in the book, a model should receive roughly 20 training tokens per parameter, with model size and training-token quantity increasing together as compute grows. The rule is not universal for every architecture, data source, or deployment objective.
Model collapse
Degradation from recursive synthetic trainingA potential failure in which models trained repeatedly on generated data lose information about the original data distribution, particularly rare patterns. The risk depends on how synthetic data is generated and mixed with real data. The book presents it as a concern in research, not as an inevitable outcome of all synthetic-data use.
Goodput
Requests meeting service objectivesA serving metric that counts only requests meeting specified performance objectives, such as limits on time to first token and time per output token. Unlike raw throughput, goodput discounts requests that complete but violate the user-facing service target. It therefore captures whether a system is both productive and acceptably responsive.
Speculative decoding
Verify draft tokens with a larger modelAn inference technique in which a faster draft model proposes several tokens and a target model checks them in parallel. The target accepts an agreed sequence and generates the next token, then the process repeats. It can reduce generation latency when enough proposals are accepted without changing the target model's intended output distribution.
Download PDF
Download EPUB
.epub digital book format is ideal for reading ebooks on phones, tablets, and e-readers.