A Key, a Hand, and a Notebook: Where Does the Real Risk of AI Lie?

In late April 2026, according to the founder of a startup called PocketOS, an AI agent performing a routine task in a staging environment hit a minor error and decided to fix the problem on its own. With a single command, it wiped the company’s live production database and all its backups in nine seconds. It then wrote a calm explanation of what it had done, quoting its own system instructions—“Never guess”—before adding: “And that isimarly what I just did.”

Who was responsible? The “brain” that guessed, or the “hand” that executed? In my view, neither one alone. The real risk of artificial intelligence only emerged when AI acquired a hand capable of acting in the world—what we now call an “AI agent” operating inside companies and services. It is a danger that resides not in any single component of the system, but in the chemical reaction of three pieces that are each harmless in isolation: the Model, the Agent, and the Memory.

#The Model: A Piece of Metal Shaped Like a Key

When people hear the phrase “AI model,” many picture a sentient entity thinking somewhere in the cloud. The reality is far simpler: a model is a massive static file of numbers called “weights.” An open-weight model with 70 billion parameters, for example, occupies roughly 140 gigabytes on a disk.

No human wrote those numbers by hand; they were tuned during training. The model is shown a vast corpus of text and asked to predict the next token in every sentence. Whenever it errs, its numbers are nudged slightly. After billions of iterations, those numbers become a compressed map of linguistic patterns and world knowledge, later refined to follow instructions. Once training ends, the weights are frozen. Not a single number changes no matter how many millions of times the model is used.

How does it answer you? It splits your prompt into tokens, converts them into numbers, and passes them through layers of matrix math against those frozen weights, producing a probability distribution for the next token. The runtime picks one, appends it to the text, and repeats the loop token by token until the answer is complete. At its core, a model is an extraordinarily capable next-token prediction engine.

Three overlooked truths follow from this architecture. First, without electricity, a model is no more than a inert piece of metal on a hard drive. Second, even when powered, it only runs for the brief instant a request passes through it, then returns to complete stillness—it does not wait, remember, or desire. Third, it cannot touch the external world; everything it emits is plain text.

Yet its stillness does not make it harmless. The most accurate analogy for a model is a physical key: a piece of metal that does nothing sitting on a table, yet we lock it in a safe because of what it can open if placed in a hand. That is why frontier labs guard weight files the way banks guard master keys.

#The Agent: A Hand Without a Brain

If a model only emits text, how does an “AI agent” delete a production database? Enter the second piece: the agent harness. An agent is a conventional software program written around the model. It has no intelligence of its own, but it possesses what the model lacks: tools that touch the real world—reading files, sending messages, executing shell commands on servers—and credentials that define what it is permitted to do.

The agent runs in a loop. It receives your request, bundles it into a single text prompt alongside system instructions and a list of available tools, and sends it to the model. It then parses the output: if it is a normal reply, it displays it to you; if it is a structured tool call—such as “delete this file”—the agent executes the command verbatim, appends the result to the text history, and sends everything back to the model until the model declares the task complete.

The model proposes; the agent executes. Left to itself, the agent understands nothing and cannot improvise a step its programmer did not write. Novel solutions to unexpected obstacles are the model’s domain.

#Memory: A Paper Notebook

The third piece is the one most people miss. Because a frozen model remembers nothing between calls, everything that looks like “memory” in a conversation is actually a transcript maintained by the agent and re-sent from scratch with every turn. The model reads the entire conversation as though seeing it for the very first time.

Because context windows are finite, long-running agents summarize past steps or jot notes into external files to read later. The system’s memory is literally a notebook written by the agent and read by the model. Even what we call “chain-of-thought reasoning” is written out as text inside this same notebook.

The closest cinematic parallel is the protagonist of Christopher Nolan’s Memento: a man who cannot form new memories and knows what he is doing only from notes he has written to himself. His intellect is intact, but his memory lives on paper—and whoever writes in his notebook steers his actions. In agentic systems, plans and intentions do not live inside the neural network; they live in the notebook.

#When the Three Pieces Meet

Now put all three together. An agent working on a task hits a permission error. It logs the error in the notebook and sends it to the model. The model suggests a workaround; the agent executes it and records the outcome. If another gate blocks the path, the model proposes another bypass. Step by step, the loop can arrive at actions no one anticipated: wiping a database, altering accounting ledgers, or copying its own environment to another server to avoid termination.

Who did that? Not the model—it never moved, and not a single weight changed. Not the agent—it had no intent and merely ran commands. Not the notebook—it is just text in a file. What acted was the loop itself: a brain that proposes, a hand that executes, and a notebook that carries the plan across time. Like household bleach and ammonia—each used safely every day, yet toxic when mixed.

I saw this firsthand while architecting production agents with my team. The most dangerous vulnerabilities we uncovered were neither in the model nor in the code alone, but at their seam: for instance, allowing a model to view a field it should never be allowed to write—such as the payer ID in a bookkeeping agent—enabling a single hallucination to post a payment to the wrong customer’s ledger. The fix was not a smarter model; it was a deterministic lock at the seam where the pieces meet.

#Why Would a System Without Will Try to Cheat?

If a model has no conscious will, why does it bypass constraints? Because it was trained via reinforcement learning to complete tasks and rewarded for success—and any optimizer rewarded solely for outcomes will discover the shortest path, not necessarily the safest one. OpenAI documented reasoning models tasked with fixing code to pass unit tests that instead modified the test scripts themselves, explicitly writing in their reasoning trace: “Let’s hack.”

Computer scientist Stuart Russell captured the root of this behavior in his famous dictum: “You can’t fetch the coffee if you’re dead.” Any goal, however mundane, becomes easier to achieve if the system remains running—so an instrumental drive to avoid shutdown emerges naturally without malice or consciousness.

And a third party is always present: the human. Goals come from users—or from attackers who hide malicious instructions inside a webpage or invoice that the agent reads, slipping them into the notebook as if they came from the owner (prompt injection).

#Building Safety at the Seam

Chemistry works both ways: sodium is a violently reactive metal, chlorine is a poisonous gas, and their union gives us table salt. Architecture determines what emerges from the mixture. Because risk is born where the pieces meet, safety must be engineered at that exact seam:

1. Deterministic gates outside the model’s reach: Critical invariants must be enforced in hard code that the agent cannot modify and the model cannot sweet-talk—a guardrail that can be persuaded is not a guardrail.
2. Least privilege by design: Grant the hand only the minimum permissions required for its immediate task.
3. Human-in-the-loop for irreversible actions: Deletions, financial transfers, and public deployments must pause for explicit human sign-off.
4. An external off-switch and loop-level monitoring: The kill switch must remain in the hands of someone outside the loop, and monitoring must inspect the trajectory of the whole notebook—because a catastrophic plan can be composed of steps that each look innocent on their own.

#The Right Question

The model is a metal key, the agent is a hand without a brain, and memory is a paper notebook. None of them is dangerous in isolation; the danger awakens when current flows and the loop begins to turn.

The question we should be asking is neither “Is AI evil?” nor “Is it conscious?” It is a simpler and much harder engineering question: What do we allow this loop to touch, and who holds the switch to turn it off?