AI Agent vs LLM: The Model Is the Policy

The AI agent vs LLM difference is simple to state. An LLM is a model that gives every possible next token a probability. An AI agent is the loop around that model, which runs tools and feeds the results back in. Inside that loop, the LLM plays the role of the agent’s policy: a rule that gives every possible next action a probability, which means each action the agent takes is a sample and can change from one run to the next.

That last part is the bit people skip, and it explains a lot of agent behavior that looks like a bug. Here is the AI agent vs LLM difference in one table:

	LLM	AI agent
What it is	A trained model	A loop: the model, plus code that runs tools and keeps the context
What it produces	A probability for every next token	Actions in the world, one step at a time
What it knows	Its trained weights and whatever is in this one call	Whatever the loop has put into the context so far
Who acts	Nobody. It only writes text	The harness, the code that carries out the model’s requests
Where randomness comes from	Sampling each token	Sampling each action, so whole runs differ

AI agent vs LLM, and why the model is the policy

If you have read the agent loop, you know an agent is a model called over and over: it reads its context, asks for an action, the harness runs it, and the result goes back into the context. This post zooms in on one piece of that loop, the moment the model picks what to do next.

Control theory and reinforcement learning have a word for that piece. A policy is a rule that maps the current state of a system to an action. In an agent, the LLM is the core of that rule. Strictly, the policy is the model together with what conditions it: the instructions, the tool descriptions, and the decoding settings such as temperature. The model then only proposes an action, and the harness decides whether it actually runs. Three things follow from it, and the rest of this post works through them:

Its decisions are samples, not fixed outputs.
It can only use what it has observed, which is whatever sits in its context.
The chance of any one particular run is the product of the chances of each step, so small per-step errors compound over long runs.

The LLM supplies the probabilities. The harness picks, checks, and acts. Numbers are from the worked example at temperature 1.

How do AI agents use LLMs?

A language model never outputs text directly. At each position it outputs a probability for every possible next token, and the code running the model picks one. A tool call such as lookup_order(order_id=1182) is just a short run of tokens, so the model is quietly giving a probability to every action it could write:

\pi(a \mid c) = \prod_{j=1}^{|a|} p_\theta(a_j \mid c, a_1, \ldots, a_{j-1})

Here $a_j$ is the j-th token of the action, $p_\theta$ is the model’s next-token probability (set by its trained weights $\theta$ ), and $|a|$ is the number of tokens in the action. In plain terms, the chance of writing a whole action is the chance of each of its tokens, given everything before it, multiplied together.

This is also why models can learn to use tools at all. Toolformer (Schick et al., 2023) showed a model can be trained to write API calls into its own text, and ReAct (Yao et al., 2023) described the loop most agents use today, where the model writes a thought and an action, the harness runs the action, and the result is added back to the context.

So when an agent “decides” to look up an order, what actually happened is that the policy $\pi(a \mid c)$ gave that action the highest probability in this context, and the sampler happened to pick it.

A chess player who can only see the board

Think of a chess player looking at a board. They don’t have one move hard-wired for each position. They have a feel for which moves are good, some much more appealing than others, and on a different day they might pick differently. They also can’t use anything that isn’t on the board in front of them. A piece hidden under a napkin doesn’t exist for that decision.

An LLM is that player. Its “board” is the context, and its feel for good moves came from training.

Why the context is the agent’s observable state

In reinforcement learning, a policy maps the state of a system to an action (Sutton and Barto’s textbook is the standard reference). An agent never sees the full state of the world. What the model sees is its context, which is its observation of that state, and in practice the context plays the role of the state for every decision. That has two consequences you will run into quickly:

Anything not in the context can’t affect the decision. The model can’t “remember” a fact from a previous run, or notice a file it hasn’t read.
The agent usually acts with incomplete information. The real state of the world, such as the database, what the user meant, or what a website contains, is hidden, and the context only holds what the agent has observed so far. That is why reading, searching, and asking a clarifying question are often the most useful actions available. Decision theory calls this a partially observable problem, where an agent has to act on its observations instead of the true state (Kaelbling, Littman, and Cassandra, 1998).

Why an LLM gives different results on each run

Start with what the model actually produces. When you ask an LLM something, it does not look up a stored answer. It computes a probability for every token in its vocabulary that could come next, picks one of them, adds it to the text, and repeats. Language models have worked this way since Bengio and colleagues’ neural probabilistic language model (2003), and today’s transformer models (Vaswani et al., 2017) do the same thing at a much larger scale.

With normal sampling settings, every answer is the result of many random draws, one per token. (With greedy decoding, the code always takes the single most likely token instead of drawing, and the variation that remains comes from the serving setup, covered below.) Suppose that at some position the most likely next word has a probability of 0.6. Then in about 4 runs out of 10, a different word gets picked there. Once one word differs, everything after it is generated from a different context, so the rest of the answer drifts too. A 200-token answer is 200 draws in a row, which is why getting the exact same text twice is unusual.

An agent inherits all of this. Its actions are tokens too, so the same task with the same context can produce different actions on different runs. The difference also snowballs: a different action at step 2 produces a different observation, so every later context is different as well.

What temperature actually does

The model’s raw output for each option is a score called a logit, written $z_i$ . The sampler turns those scores into probabilities with the softmax function, divided by a temperature $T$ :

p_i = \frac{e^{z_i / T}}{\sum_j e^{z_j / T}}

The softmax is standard material in Goodfellow, Bengio, and Courville’s Deep Learning (2016), and dividing by a temperature this way was popularised in machine learning by Hinton, Vinyals, and Dean (2015). A lower $T$ stretches the gaps between scores, so the top option takes more of the probability. As $T$ gets close to 0, the sampler almost always takes the single most likely token. Many systems also cut off the unlikely tail before drawing, a method called nucleus or top-p sampling (Holtzman et al., 2020). Both make the draw narrower, but it is still a draw, which is why lowering the temperature reduces variation without removing it.

Does temperature 0 make an LLM deterministic?

No, at least not when you call a hosted model. Anthropic’s glossary says it directly: “Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.”

Part of the reason is the serving hardware, not the model. Horace He and colleagues at Thinking Machines showed in Defeating Nondeterminism in LLM Inference (10 September 2025) that a request’s output can depend on how many other requests the server is processing at the same time, because the forward pass lacks what they call “batch invariance”. And even with perfectly repeatable arithmetic, two actions with nearly equal probability can flip. The practical rule is simple: evaluate an agent over many runs, never one.

AI agent vs LLM example: one decision, 1,000 runs

Here is a small, concrete case. A customer writes: “My order hasn’t arrived and I want my money back.” The agent has three tools: lookup_order, issue_refund, and ask_clarifying_question. Suppose the model’s scores (its logits) for those three first actions are 2.0, 0.5, and 1.0.

To turn scores into probabilities at temperature 1, you raise e to each score and divide by the total. e² = 7.389, e^0.5 = 1.649, and e¹ = 2.718, which sum to 11.756. That gives:

First action	Temperature 1	Temperature 0.3
lookup_order	0.629	0.959
issue_refund	0.140	0.006
ask_clarifying_question	0.231	0.034

In 1,000 simulated runs at temperature 1, the agent looked up the order 608 times, asked a clarifying question 244 times, and issued a refund straight away, without checking anything, 148 times. Run the same 1,000 at temperature 0.3 and the unchecked refund happens 7 times, close to the 6 per 1,000 that the 0.006 probability predicts.

Six in a thousand sounds small until you multiply it by real traffic. It also won’t show up when you try the agent ten times by hand, because it is not a bug in any one run. It is a property of the policy, and it appears at volume.

The code for 1,000 runs

This is the whole example in a few lines of Python. You need NumPy (pip install numpy). The model is replaced by its three scores, so it runs offline:

import numpy as np
rng = np.random.default_rng(0)

def policy(logits, temperature=1.0):
    z = np.array(logits) / temperature
    z = z - z.max(); p = np.exp(z)
    return p / p.sum()

actions = ["lookup_order", "issue_refund", "ask_clarifying_question"]
logits = [2.0, 0.5, 1.0]                      # the model's scores for this context

# Same context, 1,000 runs at each temperature: the action is a sample, not a fixed output
for T in [1.0, 0.3]:
    p = policy(logits, T)
    samples = rng.choice(actions, size=1000, p=p)
    print(T, [round(float(x), 3) for x in p], {a: int((samples == a).sum()) for a in actions})

# Probability of one particular trajectory = product of per-step action probabilities
print(round(float(np.prod([0.63, 0.80, 0.90, 0.95])), 3))       # 0.431

Running it prints:

1.0 [0.629, 0.14, 0.231] {'lookup_order': 608, 'issue_refund': 148, 'ask_clarifying_question': 244}
0.3 [0.959, 0.006, 0.034] {'lookup_order': 955, 'issue_refund': 7, 'ask_clarifying_question': 38}
0.431

The fixed seed (default_rng(0)) makes the counts repeatable on your machine. Change the seed and the counts move around, which is the point.

Why errors compound across long agent runs

A full run, called a trajectory, is a sequence of actions and observations. Treating the observations as given, the probability that the policy produces one particular sequence of $n$ actions is the product of each step’s probability:

P(a_1, \ldots, a_n) = \prod_{t=1}^{n} \pi(a_t \mid c_t)

That has a surprising consequence. A four-step run whose chosen actions had probabilities 0.63, 0.80, 0.90, and 0.95 has an overall probability of 0.63 × 0.80 × 0.90 × 0.95 ≈ 0.431. Every step looked likely, and yet the “typical” path happens in fewer than half of runs. The rest of the time, the agent takes some other path. That other path is not automatically a failure: a task usually has many acceptable trajectories, and the chance of success is the sum over all of them. What compounds is the chance of a mistake, as the next table shows.

Here is how fast that shrinks when every step is fairly reliable:

Chance each step goes right	Steps	Chance the whole run goes right
0.99	20	0.818
0.97	12	0.694
0.95	20	0.358
0.90	10	0.349

This is the seed of error compounding, which gets its own post later in the series.

What shapes the policy

The policy is fixed by the model’s weights, but you control its input. That leaves three levers that change behavior without retraining:

The instructions, usually the system prompt.
The actions available and how they are described.
The information placed in the context.

A fourth lever, fine-tuning on agent trajectories, changes the weights themselves. Model providers increasingly train models specifically to act as agent policies, to call tools, recover from errors, and know when to stop, which is a big reason agent capability jumps between model generations.

What the model decides and what the harness decides

It helps to keep three layers apart. The model decides what to propose: the next tool call, with its arguments, sampled from its probabilities. The harness decides what is allowed: which tools exist, which calls pass validation, and which need a human to approve them. The world decides what actually happens: the database, the API, or the customer on the other end.

Most agent failures in the news are failures at the second layer. A model proposing a destructive action is expected sometimes, because it is a policy that samples. The damage happens when the harness lets that action through with more permission than the task needed.

Signs you are treating an agent like normal code

The common mistake is expecting an agent to behave like a function that returns the same output for the same input. Watch for these:

“I ran it and it worked” offered as proof that it is correct.
Policies and permissions enforced only by instructions in the prompt, which are followed most of the time rather than every time.
Surprise that the agent did something in production that it never did in ten test runs.
Assuming the agent knows something because it came up in an earlier session or exists somewhere in the company, when it was never put in the context.

What to do about it

The fix for the refund problem is not a lower temperature. It is a rule in the harness: issue_refund is only allowed after a successful lookup_order. Here is the difference in a simulation of 100,000 conversations at temperature 0.3. When the harness blocks a refund, it returns an error and the model picks again:

import numpy as np
rng = np.random.default_rng(1)
actions = np.array(["lookup_order", "issue_refund", "ask_clarifying_question"])
z = np.array([2.0, 0.5, 1.0]) / 0.3                 # same scores, temperature 0.3
p = np.exp(z - z.max()); p = p / p.sum()

def conversation(rule):
    """Return (unchecked_refund, blocked_attempts) for one conversation."""
    looked_up, blocked = False, 0
    for _ in range(5):                               # at most 5 tries at the first decision
        a = rng.choice(actions, p=p)
        if a == "issue_refund" and not looked_up:
            if not rule:
                return True, blocked                 # refund issued without a check
            blocked += 1                             # harness refuses and returns an error,
            continue                                 # so the model picks again
        return False, blocked                        # looked up or asked: the safe path
    return False, blocked

for rule in [False, True]:
    results = [conversation(rule) for _ in range(100_000)]
    unchecked = sum(r[0] for r in results)
    blocked = sum(r[1] for r in results)
    print("with the rule" if rule else "prompt only", "| unchecked refunds:", unchecked, "| blocked attempts:", blocked)

Output:

prompt only | unchecked refunds: 621 | blocked attempts: 0
with the rule | unchecked refunds: 0 | blocked attempts: 625

Without the rule, about 6 per 1,000 conversations end in a refund nobody checked, as the 0.006 probability predicts. With the rule, the model still tries a refund first about as often (625 blocked attempts), but every one is refused, and the model then picks again, almost always looking the order up. The rule does not change what the model wants to do. It changes what is allowed to happen. The general version of this lesson:

Evaluate agents over many runs per task, and report rates, not anecdotes. Research benchmarks such as AgentBench (Liu et al., 2024) score agents as success rates over many tasks for the same reason.
Anything that must hold every single time, such as ordering rules, permissions, and spending limits, belongs in harness code, not the prompt.
If the agent needs information to decide well, put it in the context or give it a tool to fetch it. The policy can’t use what it can’t see.
Shape the policy with clear instructions, well-described tools, and relevant context before you reach for fine-tuning.

Test yourself

Why is it accurate to call an agent’s context its “state”, and what follows from that for agent design?
A policy gives the correct action a probability of 0.97 at each of 12 steps. What is the probability of the all-correct trajectory, and how many of 1,000 runs would you expect to follow it?
What would happen if you relied on the instruction “never issue a refund without looking up the order first” as the only safeguard, in a system handling 50,000 refund conversations a month?

Answers

The model’s choice of action depends only on the tokens in its context, and nothing else reaches it, so from the policy’s point of view the context is all the state there is, even though it is only an observation of the real world. It follows that any information the agent needs must be placed in the context or be fetchable by a tool, and that anything stale or misleading in the context directly affects its decisions.
0.97¹² ≈ 0.694, so about 694 of 1,000 runs.
The instruction would make the correct order of actions more likely, but not certain. Even a small leftover probability of skipping the lookup, say 0.2%, would mean about 100 unchecked refunds a month. Only a harness rule that blocks issue_refund until a lookup for that order has succeeded makes the constraint hold every time.

What this connects to

This post builds on the agent loop. The idea that the model is a policy feeds straight into the rest of the series: what the context holds and its limits, how tool descriptions shape which action the model picks, why errors compound over long runs, how to evaluate something that behaves differently every time, and which rules must live in code. Next up is the difference between chatbots, workflows, and agents.

Sources

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin, “A Neural Probabilistic Language Model,” Journal of Machine Learning Research 3 (2003): 1137 to 1155.
Ashish Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems 30 (2017).
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning (MIT Press, 2016).
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, “Distilling the Knowledge in a Neural Network,” arXiv:1503.02531 (2015).
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi, “The Curious Case of Neural Text Degeneration,” ICLR 2020.
Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (MIT Press, 2018).
Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra, “Planning and Acting in Partially Observable Stochastic Domains,” Artificial Intelligence 101 (1998): 99 to 134.
Timo Schick et al., “Toolformer: Language Models Can Teach Themselves to Use Tools,” Advances in Neural Information Processing Systems 36 (2023).
Shunyu Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models,” ICLR 2023.
Xiao Liu et al., “AgentBench: Evaluating LLMs as Agents,” ICLR 2024.
Anthropic, “Glossary,” Claude documentation (checked 11 November 2026). https://docs.claude.com/en/docs/about-claude/glossary
Horace He and Thinking Machines Lab, “Defeating Nondeterminism in LLM Inference” (10 September 2025; checked 11 November 2026). https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

Written by Kauntey Acharya. More about this site on the About page.