Implementing the Spilling Concept for Managing AI Agent Context

If you frequently build or run coding agents with terminal access (like an exec tool), you have probably seen them start sharp and responsive, only to hallucinate, repeat themselves, or crash with context length exceeded around turn 20 or 30.

The culprit is almost always unmanaged tool I/O, rather than the model itself.

A single unconstrained terminal command—such as docker logs, a broad git diff across a monorepo, or a build script dumping thousands of stack trace lines—can easily flood the context window with raw text. Dumping 4 MB to 6 MB of logs directly into the prompt payload triggers several cascading issues:

The context window is immediately maxed out.
Token input on subsequent turns balloons, increasing latency and operational cost.
The model suffers from attention drift. Essential instructions and task context established early in the session get buried under thousands of lines of terminal noise.

When building Nouride (a lightweight multi-agent daemon for homelab workflows), resolving this was an early priority. I needed an agent like Nouva to pair-program across dozens or hundreds of turns without crashing midway.

Here are a few approaches I tested, and the solution that stuck: Tool Output Spilling.

Initial Approaches That Fell Short

Before finding a workable pattern, I tried several common approaches that proved troublesome in practice:

1. Instructions in the System Prompt

I initially thought a prompt instruction would suffice: "If a terminal output is expected to be long, redirect it to a file and provide the file path instead."

This rarely worked consistently. An LLM cannot predict the output volume of a command before running it. When executing journalctl or npm test, it cannot foresee whether the output will be 5 lines or 10,000 lines. Leaving memory management to model intuition before execution is too fragile.

2. Naive Head Truncation

Next, I placed a hard character limit on the backend. If the output exceeded a threshold, it was truncated to keep only the first 2,000 characters.

This introduced a worse problem. In execution logs, the critical signal almost always lives at the tail: the final error line, compiler summary, or exit failure code. Keeping only the head caused the agent to see successful initial setup lines, assume everything succeeded, and report success while the process actually failed at line 5,000.

3. Short Marker: "Run Again If Needed"

Another common pattern is returning a short stub: [output 5MB truncated, re-run command if you need the details].

This makes little sense when the command is a 2-minute build or test suite. Forcing the agent to wait another 2 minutes just to peek at intermediate lines is inefficient.

The Solution: Filesystem Spilling

Eventually, I adapted a concept operating systems have used for decades: memory spilling / paging to disk. When fast memory (RAM) is constrained, the OS swaps unused pages to disk rather than terminating the process.

In Nouride, the daemon harness sits squarely between tool execution results and the outgoing prompt payload sent to the LLM. The pipeline runs deterministically:

1. Capping by Turn Budget

A single tool result must not exhaust the context window. In Nouride, each tool result is capped at a maximum of 25% of the total conversation budget (with a floor of 2,000 tokens). This ensures that even if an agent invokes 3 or 4 tools concurrently in one turn, 50% of the context remains available for persona instructions, memory, and chat history.

2. Writing Full Output to Disk

If the character count of a tool result exceeds the budget:

The harness preserves the raw text and writes it atomically to disk at $DATA_DIR/tool-output/<session_id>/<call_id>.txt.
Permissions are immediately locked to 0600 (to safeguard potential environment variables or credentials).
The spill directory is automatically added to the agent's authorized read-only sandbox roots, allowing the agent to read the file when needed.

3. Sending a Structured Excerpt (Head & Tail)

Instead of feeding 5 MB of raw text, the harness passes a structured excerpt:

Head (65%): The beginning of the command output, providing initialization context and configuration details.
Tail (35%): The end of the log, giving immediate visibility into compilation summaries or fatal exit codes.
Metadata: Exact line count, estimated size in KB, and the number of omitted lines in between.
Pointer Path: The physical file path on disk, accompanied by clear guidance: "Open it with read and an offset, or narrow it with grep."

The payload received by the agent looks like this:

[output too large for the context — 14,200 lines, ~2,840KB.
The first 65 and last 35 lines are below; 14,100 in the middle are not.]
Full output: /srv/nouride/.nouride/data/tool-output/s_019/call_exec_42.txt
Open it with `read` and an offset, or narrow it with `grep` — it is a normal file and it is readable.

... (initial command output) ...

... (final summary / error output) ...

Tangible Results

This spilling pattern directly addressed several operational issues:

Unbroken Error Visibility: Because the tail is always included, the agent recognizes the core failure reason without guessing or hallucinating.
Investigation-Ready: If the agent needs to trace a specific warning buried in the middle of a log, it does not re-run a slow command. It calls grep or read with an offset directly against the spilled file path.
Stable Context Size: Token consumption per turn remains lean and predictable from turn 1 through turn 100+. No massive token spikes from a single verbose command.
Automatic Retention: Spilled files follow an automatic lifecycle tied to chat session timestamps. Once a session has concluded for a few days, the daemon sweeps the files clean.

Takeaway

Reliable long-running agents depend on disciplined I/O control in the runtime harness. By offloading verbose command logs to disk and passing structured excerpts with file pointers, the model keeps its focus without blowing up token budgets.