Lecture 12: Models — LLM Agents

Definition

What is an agent?

• An agent is a model that interacts with the world — it combines some internal processing (reasoning, memory) with external interaction (tools, APIs, environment).
• Core components: an LLM (the brain), memory (short-term prompt context, long-term database), tools (APIs, search, calculators, code), and an environment (the world it senses and changes).
• Examples: self-driving cars, chatbots, game AI, robots — agents predate LLMs, but LLM agents differentiate themselves by doing reasoning on top of acting.

RecallWhat two ingredients turn an LLM into an agent?

Context

Historical language-based agents

• Text only: ELIZA (1966) — rule-based reflection of user input as a Rogerian therapist; no real understanding.
• Virtual world interaction: Siri, Alexa — speech in/out, can call a fixed set of services.
• Robot control: SayCan — grounding LLM suggestions in robot affordances.
• All of these resemble reflex actions / system-1 behaviour — pattern match input to action with little or no explicit reasoning.
• The shift with modern LLM agents is adding deliberate reasoning (system-2) into the loop.

RecallWhat was missing from ELIZA / Siri that LLM agents now add?

Think first

Reasoning: Chain-of-Thought and variants

• Reasoning in LMs = intermediate generation — text output that is not part of the final answer shown to the user, but helps the model arrive at it.
• Chain-of-Thought (CoT) prompting: instruct the model to show its working, e.g. "Let's think step by step." Few-shot exemplars of step-by-step solutions also work.
• Self-consistency: sample many CoT traces, take the majority answer — voting smooths over noisy reasoning chains.
• Analogical reasoning: ask the model to recall a similar problem first, then solve.
• Reflexion: include statements that reflect on past outputs / failures in the prompt, so later attempts learn from mistakes.
• Tree-of-Thoughts: branch and explore multiple reasoning paths, prune bad branches.
• Planning is a related (open) challenge: when evaluated carefully, LLMs still struggle to produce reliable multi-step plans.

RecallWhat is the difference between CoT and self-consistency CoT?

Reason + Act

ReAct: reasoning interleaved with acting

• ReAct (Yao et al., 2023) makes reasoning itself one of the actions the model can take, alongside acting in the world.
• Two natural patterns: (a) alternate reason / act / observe steps, or (b) let the model choose any action at each step (including a "Thought" action).
• No training required — ReAct is purely a prompting scheme with human-written few-shot examples showing the Thought → Action → Observation pattern.
• Result: the agent can update its beliefs based on observations, then plan the next action.

RecallDoes ReAct require fine-tuning? How is it implemented?

Retrieve then generate

RAG: Retrieval-Augmented Generation

• RAG is the simplest and most widely used form of acting: before generating, the LM retrieves relevant documents from a knowledge base and conditions output on them.
• Pipeline: input → retriever → top-k passages → prepend to prompt → LM → output.
• Usefulness depends on (a) the knowledge base you retrieve from, and (b) the quality of your retrieval method.
• Sparse retrieval (BM25): bag-of-words tf–idf scoring; exact lexical match.
• Dense retrieval: encode query and documents into vectors with an embedding model; rank by cosine similarity in vector space — captures semantic match.
• RAG also doubles as long-term memory: write = append new documents, read = retrieve.

RecallWhat two factors determine whether a RAG system is useful?

Acting

Tool use

• An LLM "uses" tools by producing special symbols in its output (function calls, JSON, API names) — a controller parses them, executes the tool, and feeds the result back in.
• Toolformer (Schick et al., 2023): find natural places in existing text to insert tool calls, then fine-tune the LM to predict them.
• Common tool categories: calculators, web search, code execution, knowledge-base lookup, browser control, computer use (Anthropic) — agents that drive a whole desktop.
• For simple APIs (calculator, search), creative prompting alone often suffices. For complex tools (entire computer), there are many active efforts including RL training (e.g. WebGPT).
• Writing and executing code is itself a powerful tool — the LM emits a Python snippet, it runs in a sandbox, the output is observed.

RecallHow does an LLM "call" a tool at the token level?

Many LLMs

Multi-agent systems

• An agent need not be a single LLM call — it can be a system built from many LLM calls plus glue code.
• AutoGen (Wu et al., 2023): a collection of LLMs that talk to each other, e.g. a planner and a coder agent collaborating on a task.
• DyLAN (Liu et al., 2024): a collection of LLMs that we choose between dynamically based on the sub-task.
• DSPy (Soylu et al., 2024): a framework that lets you define computation graphs over LM calls and have a compiler optimise prompts and / or fine-tune the modules automatically.
• Multimodal and physically embodied agents extend the idea to images, drones, and robots — language is still doing the heavy lifting.

RecallName two ways multiple LLMs can be composed into a single agent.

How do we measure progress?

Evaluation and risks

• Agent evaluation requires new benchmarks that approximate the real world, often with simulators so the agent can interact:
  • WebArena — multi-step browsing tasks in a sandboxed web.
  • SWE-Bench — fix real GitHub issues end-to-end.
  • PrivacyLens — privacy-respecting behaviour in messaging-style tasks.
• Risks:
  • Hallucination — confident wrong answers, amplified when the agent acts on them.
  • Prompt injection — malicious content in retrieved pages / tool output hijacks the agent's instructions.
  • Runaway actions — agents can take many real-world steps quickly; loops or misuse of paid APIs / write actions are costly.
  • Evaluation difficulty — open-ended tasks have no single ground truth; success depends on the simulator.

RecallName a sandboxed benchmark for agents, and one safety risk it does not directly test.

Lab

Workshop & materials

• chapters/chapter12/materials/lecture-12.pdf — full slide deck (Lecture 12, 2025).
• The lecture ends with a workshop preview for the agent assignment — build a ReAct-style loop with a tool of your choice.

Quick practice