Before you build with AI.

What an LLM is

A large language model (LLM) is a neural network trained to do one thing: given some text, predict what comes next. It scores every possible next piece of text, one is chosen, appended, and the model predicts again. Answering questions, summarizing, translating and writing code all come out of this single repeated step.

How a model gets trained

Pretraining runs the model over a huge amount of text so it learns language, facts, reasoning patterns and code. The result predicts text well but isn't yet a helpful assistant. Post-training then tunes it, using example conversations and human and AI feedback, to follow instructions, use tools and decline harmful requests.

Three consequences shape everything you build:

• Knowledge cutoff: the model knows nothing that happened after its training data was collected.
• No private knowledge: it has never seen your company's documents, database or tickets.
• No learning while you use it: the weights are fixed. Anything it needs for a task has to be in the request itself.

Getting the right information into the request is most of AI engineering, and it's what retrieval (RAG) and tools are for.

Tokens and the context window

Models don't read characters or words; they read tokens, which are word fragments. Limits and prices are all measured in tokens. The context window is the maximum number of tokens one request can hold, and everything counts toward it: your instructions, the conversation so far, tool definitions, any documents you include, and the answer being generated.

Two practical effects: long conversations and big documents cost more on every request, because the whole context is sent each time; and anything that doesn't fit simply isn't seen by the model.

Strengths and failure modes

LLMs are strong at language: explaining, summarizing, rewriting, extracting structure from messy text, writing code and reasoning over material you provide. Their typical failures are just as predictable:

Prompts

A request has a system prompt (who the model is, rules, output format) and a list of messages alternating between the user and the assistant. Good prompts read like a brief to a smart colleague who has no context: say what you want, give the material, say what to do when unsure.

System: You are a support assistant for the FastAPI documentation.
Answer only from the provided documentation and cite the page.
If the answer isn't in it, say you don't know.

User:
<docs>
...passages found by search...
</docs>
<question>How do I run code after the response is sent?</question>

Tags like <docs> have no special meaning to the API; they just make it unambiguous where material starts and ends. Examples of good answers, included in the prompt, are one of the most effective ways to steer output.

Claude and Anthropic

Claude is a family of LLMs built by Anthropic. People use it through apps (claude.ai on web, desktop and mobile), through coding tools like Claude Code, and through the Claude API, which is how your own software talks to it. The AI engineering guide uses the API.

Models come in tiers that trade capability against speed and cost: Haiku is the fastest and cheapest, Sonnet balances capability and cost, and Opus is the most capable of the three. Anthropic's newer Mythos tier sits above Opus. Model names and versions change regularly, so treat the models overview in the Claude docs as the source of truth, and keep model ids in configuration rather than code.

Anatomy of an API call

Every interaction is one HTTPS request to the Messages API with your API key. You send the model id, an output limit, an optional system prompt and the messages; you get back content blocks, a stop reason and token usage. The official SDKs (Python, TypeScript and others) wrap this in function calls.

Accounts, keys and cost

• API keys live in the Claude Console at platform.claude.com, under Settings → API keys. A key starts with sk-ant- and is shown only once, when you create it.
• API billing is separate from a claude.ai subscription. A Pro or Max plan doesn't include API usage; the API is paid per token from its own balance.
• Keys are secrets. Keep them in a .env file that's in .gitignore. A key pushed to a public repository gets found and abused quickly; if that happens, delete it in the Console and create a new one.
• Set limits. Keys belong to a workspace, and workspaces can have their own limits. Use a small budget while learning.
• Embeddings need a second account. Anthropic doesn't offer an embedding model; the guide uses Voyage AI, which has its own key and a free allowance (check its pricing page).

Other providers

OpenAI, Google (Gemini) and others offer similar APIs, and open-weight models can run on your own hardware with tools like Ollama. The concepts in this crash course apply to all of them; request formats and feature names differ in the details. Frameworks such as LangChain hide some of those differences behind one interface, which is one of their main selling points.

Structured output

Models produce text, but programs need data. With structured outputs you supply a JSON schema, usually generated from a Pydantic model, and generation is constrained so the reply always matches it. Instead of parsing prose, you get a validated object: a page summary with title, summary and topics, or a verdict with grounded: true.

This is the bridge between a language model and ordinary code: once output has a type, you can store it, test it and branch on it.

Tool use

A model can't do anything by itself: it can't search your database, call an API or read a file. Tool use lets you describe functions (a name, what it does, what arguments it takes). When the model decides it needs one, it replies with a request to call it. Your code runs the function and sends back the result, and the model continues with that information.

Embeddings

An embedding model turns a piece of text into a fixed-length list of numbers, a vector (for example 1024 numbers). It's trained so that texts with similar meaning get vectors that point in similar directions, even when they share no words: "How do I test my API?" lands near a passage about TestClient.

Embedding models are separate from chat models: smaller, cheaper, and they only produce vectors. The same model must embed both your documents and the incoming questions, or the vectors aren't comparable.

Measuring similarity

The usual measure is cosine similarity: it compares the direction two vectors point in. 1 means the same direction (very similar), around 0 means unrelated. Distance is the flip side: smaller distance, more similar.

import math

def cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

cosine([0.9, 0.1, 0.0], [0.8, 0.2, 0.1])   # 0.98: very similar
cosine([0.9, 0.1, 0.0], [0.0, 0.2, 0.9])   # 0.02: unrelated

Real vectors have about a thousand dimensions instead of three, but the arithmetic is the same.

Vector databases

A vector database stores vectors next to their source text and metadata, and answers one question fast: which stored vectors are closest to this one? Comparing against every vector works for thousands of items but is too slow for millions, so vector databases build approximate nearest neighbour indexes (HNSW is the common one) that skip most comparisons in exchange for occasionally missing a close match.

Keyword search still matters

Vector search finds meaning but can miss exact strings: a class name like BackgroundTasks, an error code, a version number. Classic full-text search is the opposite. Production systems usually run both and merge the results, which is called hybrid search. Postgres has full-text search built in, so pgvector plus Postgres covers both.

Why RAG exists

Your model doesn't know your documents, its knowledge has a cutoff, and it fills gaps with plausible guesses. There are three ways to give it your knowledge:

The two pipelines

RAG is two separate flows. Ingestion runs ahead of time: split documents into chunks, embed each chunk, store the vectors. Answering runs per question: embed the question, find the nearest chunks, and give them to the model with instructions to answer only from them and cite them.

Where RAG goes wrong

• Bad chunks: pieces too small to make sense alone, or too big to be specific.
• Wrong passages retrieved: the answer exists but search didn't find it. No prompt can fix this.
• Right passages, wrong answer: the model ignores them or adds claims they don't support.
• Stale index: documents changed but weren't re-ingested.

Each failure has a different fix, so you need to measure retrieval and answers separately. That measurement is what evals are, covered next.

Workflows and agents

Anthropic draws a useful line between two kinds of systems. In a workflow, your code decides the steps and the model fills them in. In an agent, the model decides the steps itself, calling tools in a loop until it judges the task done.

Anthropic's guidance is to start with the simplest thing that works, often a single well-prompted call, and add steps or autonomy only when it measurably helps.

MCP: a standard plug for tools

Without a standard, every AI app needs custom code for every tool it connects to. The Model Context Protocol is an open standard that fixes this: write a tool integration once as an MCP server, and any compatible host can use it. The official docs compare it to USB-C: one connector for many devices.

A server can offer tools (actions the model calls), resources (data the app can read) and prompts (templates the user picks). If you know REST APIs, think of an MCP server as an API designed to be discovered and called by AI applications.

Evals: testing when outputs vary

A unit test expects one exact output. LLM systems give varied outputs, so testing them means running a fixed set of realistic questions, the golden dataset, and scoring the results:

• Deterministic checks where possible: did the right page come back, does the answer mention the required term, are citations valid.
• LLM-as-judge for what rules can't check, such as "is every claim supported by the sources?". A second model grades the answer against a rubric.
• Scores over time: the point isn't pass or fail on one run, it's seeing whether a change to the prompt, model or chunking made things better or worse.

Frameworks

Everything above can be built with a provider SDK and ordinary code, and the AI engineering guide does that first so you see the mechanics. Frameworks package common patterns:

The trade-off is always the same: less code to write, but another layer between you and the model to understand and debug.

The stack at a glance

Before you start

1. Python 3.12 or newer and uv, plus the FastAPI, Postgres and pytest basics from the first guide.
2. Docker, to run Postgres (with pgvector) and Redis locally.
3. A Claude API key from the Claude Console, a small prepaid balance, and a spending limit.
4. A Voyage AI API key.
5. A corpus to work with. FastAPI's documentation is Markdown in its GitHub repository: git clone https://github.com/fastapi/fastapi, then look in docs/en/docs.

Then check your key works with one call (put ANTHROPIC_API_KEY in your environment or .env first):

# hello_claude.py   run with: uv run --with anthropic python hello_claude.py
import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-haiku-4-5-20251001",
    max_tokens=100,
    messages=[{"role": "user", "content": "Say hello in five words."}],
)
print(next(block.text for block in response.content if block.type == "text"))
print(response.usage)

If it prints a greeting and a token count, you're ready for the AI engineering guide.

Glossary