How an LLM Agent Works: Context, Tools, and the Feedback Loop

An LLM agent differs from a generative model through its feedback loop. Where a chatbot simply generates text, an agent calls tools, receives results, and refines its response. This is made possible by a special architecture layered on top of the LLM.

Context and the basics of operation

The LLM receives a block of text as input, called the context. Its size is limited: in the early days of transformers, 32K tokens was considered a breakthrough; now 1M is the industry standard. Based on the context, the model generates a continuation—a logically complete fragment that ends with a special <stop> token. The context is then extended with the new fragment, and the cycle repeats.

Calling tools and describing them

The model is trained to use special formatting to call external functions. Each provider implements this differently: Anthropic uses its own format, while Ollama and vLLM replicate the OpenAI Tools API. For the model to know which tools are available to it, they must be described in advance within the context (in the "system prompt") according to the JSON Schema standard—specifying parameters, their types, descriptions, and whether they're required.

Here's an example of a delivery-status-check tool description (get_delivery_status): a name, a description of its purpose, and a schema with the fields order_id (required, string) and detailed (optional, boolean). With such descriptions in context, the model generates calls in a specific format (for example, Anthropic's).

The set of available tools is called a harness. To describe tools not built directly into the agent itself, developers use MCP (Model Context Protocol)—an industry standard for IPC between an agent and third-party systems. Communication happens over JSON-RPC 2.0, with transport either via a local process's stdin/stdout (analogous to FastCGI) or HTTP for external servers.

The agent loop, feedback, and context management

The LLM provider's API returns typed JSON objects. The format varies: the OpenAI Tools API is the most widespread, with Ollama and vLLM replicating it, while Anthropic has its own. In a response (from the claude-sonnet-4-5 model), a tool call arrives as a separate block within the content array, containing an id, a name, and a parsed input object. The marker of an unfinished turn is stop_reason: "tool_use".

The execution result comes back as a new message with the role user and a tool_result block that references the call's id. If the tool fails, an is_error flag is returned along with a trace. An error doesn't break the loop: the agent sees what it did and corrects itself on the next turn. Previously, the model wrote code blindly; now it sees the feedback.

Context has two limits: a formal one (exceeding it causes the API to return an error) and a working one (the volume at which quality degrades). In a long context, the beginning and end remain in focus while the middle gets lost. Two techniques are used to combat this:

• Eviction to files — only 10–20 lines of output are placed in the context, with the rest written to a temporary file with a path nearby; the model can retrieve what it needs via a separate tool call if necessary. • Compaction — replacing the context with a concise summary generated by the LLM using a special system prompt that captures the main request, key concepts, technical details, and architectural decisions.

That covers the basic architecture of an agent. It's worth practicing by writing an agent against the raw API: you can fit it into 200–300 lines of code and get a tool that's not far behind Claude Code or Codex. An agent's power comes mostly from the strength of the underlying model, not from its set of tools.