0:00 / 0:00

Every Agent Is An LLM Plus A Harness

From The Great AI Catchup For Leaders

An AI agent is an LLM with a harness around it. The LLM takes tokens in and puts tokens out, and the harness is the application around it, usually code, that gives that output somewhere to run and supplies memory, context, skills and safeguards. Add a UI and a runtime and you get every AI app, from ChatGPT to the agents inside Notion, Granola and Linear.

Transcript

And then the third thing I would say is, you know, LLMs still are not perfect, but a good enough LLM, or a great enough LLM now, candidly, with a harness around it, and I'll tell you what a harness is, is extremely useful in the real world, right? So if you just pair those two things, you actually get agents that can do so much cool stuff. And if you had to really break it down, an agent is really just an LLM, which is the base mathematical model. That has pretty much stayed the same from a mechanistic perspective. It inputs tokens, applies a neural network, or a series of neural networks, and outputs a series of tokens, right? So the way I think of it is it produces intelligence via words, and then the harness is really the application around it. So if your model is, you know, Fable or Opus or Astra or Haiku or whatever you hear about, the harness is what actually takes the raw output from a model. In most cases, it is code and makes it useful by giving it a place to execute. It also supplies the model with the memory, it supplies the model with the safeguards, the context, the skills, and all of those things that we just discussed, those are all elements of the harness. And so if you then extend that equation, and you say, okay, now I have an agent, if I give it a UI, meaning ChatGPT looks different, Grok looks different, any agent that you build custom looks different, and supply it with the runtime, which is, again, just where it's hosted and how it runs and executes code, you basically get every AI app ever. So any AI app that seems like transformative, breakthrough, etc., is really just a combination of these three things. It applies to the ones that we know. It also applies to, you know, this proliferation of agents that I'm now seeing, you know, you've got these personal agents, which are really, again, a harness that prioritizes memory and a lot of cron jobs that repeatedly start and stop loops. So that is why it feels like it's doing a lot of work on its own. There's no magic there. Now, ChatGPT and Claude and Codex, what I'm seeing is they want to really own the workspace. And so a lot of those apps are now building the ability to create slides and docs and inbuilt artifacts, which I think are pretty much the coolest feature coming out of Claude and OpenAI these days. And then the third type of agent is typically embedded inside software. So Notion has its own agent. Granola has its own agent. Linear, which is a software tool, has its own agent, really, I mean, hundreds and thousands of them, right? But again: LLM, harness, runtime, UI. That is every AI app.