16. RAG, tools and agents

A trained model's knowledge is fixed at its data cutoff, cannot include your private documents, provides no sources, and cannot be updated without retraining. Retrieval-augmented generation addresses all four by changing the input rather than the model.

The pipeline:

1. Chunk your documents into passages.
2. Embed each chunk into a vector with an embedding model and store it in a vector index.
3. At query time, embed the question and find the nearest chunks by cosine similarity (Lesson 3's geometry, at production scale).
4. Insert the retrieved chunks into the prompt with an instruction to answer only from them.
5. Generate, ideally with citations back to the chunks.

Note the division of labour: retrieval supplies the facts, the model supplies the language and synthesis. Failures in the answer are very often retrieval failures wearing a costume.

The tutorial version — fixed 512-token chunks, top-5 cosine similarity — underperforms badly on real corpora. The standard remedies:

• Chunking. Fixed-size splits cut sentences and separate a claim from its qualifier. Split on structure (headings, paragraphs), overlap chunks, and prepend document/section titles to each chunk so it carries its own context.
• Hybrid search. Embeddings capture meaning but miss exact strings — error codes, product SKUs, surnames. Combine dense vectors with keyword search (BM25) and fuse the rankings. This is usually the single biggest quality win.
• Reranking. Retrieve 50 candidates cheaply, then score each against the query with a cross-encoder that reads both together. Far more accurate than comparing two independently computed vectors.
• Query rewriting. "What about the second one?" is unretrievable. Rewrite follow-ups into standalone queries using the conversation history.
• Order in the prompt. Given the lost-in-the-middle effect, put the strongest chunks at the beginning and end.

There is less magic here than the term suggests. The model cannot execute anything. What happens:

1. You describe available tools — names, descriptions, parameter schemas — in the request. The API formats them into the context.
2. The model, having been fine-tuned on this format, generates structured text naming a tool and its arguments.
3. Generation stops. Your code parses that, decides whether to run it, and executes the actual function.
4. The result is appended to the conversation as a new message.
5. The model is called again, now with the result in its context, and continues.

Everything outside step 2 is your program. The model is a text generator that has learned a convention for asking. That framing matters for security: the model's request to call delete_records is a suggestion, and your code is the only thing that decides whether it happens.

Practical notes: tool descriptions are prompts and deserve the same care; too many tools degrade selection accuracy (group them or filter by context); and constrained decoding — masking the logits so only tokens valid under the schema can be sampled — is how providers guarantee syntactically valid JSON.

An agent is the tool-calling loop run repeatedly toward a goal: observe, decide, act, observe the result, repeat until done or out of budget.

The characteristic failure modes are worth memorising, because they are all structural rather than incidental:

• Compounding error. At 95% reliability per step, a 20-step task succeeds about 36% of the time. Long autonomous chains need either much higher per-step reliability or verification between steps.
• Context growth. Every observation is appended. Long runs fill the window, get slow and expensive, and start losing early information.
• Loops. Retrying the same failing action indefinitely. Needs explicit step budgets and loop detection.
• Injection via tool results. Retrieved pages and API responses enter the context as text. A hostile web page can carry instructions. Everything from Lesson 15 applies here with higher stakes, because the model now has the ability to act.

What makes agents work in practice: narrow scope, tools that are hard to misuse, verification steps (run the tests, check the schema), explicit budgets, and a human confirmation gate on anything irreversible.