Context

Introduction

Overview

LLM::Context is the runtime that powers every agent. When you call LLM::Agent#talk, the agent delegates to its internal context. The context manages the message history, sends requests to the provider, tracks pending tool calls, and feeds results back to the model. Everything an agent does, a context does too, but without the automatic tool loop.

Using a context directly gives you finer control over each step of the conversation. You decide when to send messages, when to execute tools, and when to stop. This is useful for custom confirmation flows, mixed concurrency strategies per tool, or any workflow where the agent’s automatic loop gets in the way.

How it works

A context wraps a provider and maintains the conversation state. Call LLM::Context#talk to send input to the model, check LLM::Context#pending_functions? to see if tools were requested, and use LLM::Context#wait to execute them. Each call to #talk appends to the conversation and returns the model’s response. The context serializes its state with to_h and to_json, and restores it with restore. This is how the ORM integrations and filesystem persistence work under the hood:

require "llm"

llm = LLM.deepseek(key: ENV["KEY"])
ctx = LLM::Context.new(llm)

res = ctx.talk "What's the weather in Tokyo?"
puts res.content

Why would I use it?

A bare context gives you control that the agent abstraction does not expose. Pre-flight checks on tool requests, per-tool confirmation prompts, mixed concurrency strategies across tools, or manual iteration until a condition is met are all easier with a bare context.

Notes

The agent uses LLM::Context internally. Anything you can do with a context, you can also do through an agent. The trade-off is convenience versus control. Contexts support the same concurrency strategies, compaction, cancellation, and serialization as agents.

Manual loop

Overview

LLM::Agent manages the tool loop automatically. It calls the model, checks for tool requests, runs the tools, feeds results back, and repeats until the model produces text. You can bypass this and drive LLM::Context directly instead. This gives you finer control over each step of the loop at the cost of more code.

How it works

When you want to control the tool loop yourself, drive LLM::Context directly instead of using an agent. Start a conversation, check for tool requests, execute them, and feed results back. The full loop is under your control. Each call to LLM::Context#talk appends to the conversation and returns the model’s response, and LLM::Context#pending_functions? tells you whether tools were requested. From that foundation you can inspect, iterate, or confirm per-tool in a single flow:

require "llm"

llm = LLM.deepseek(key: ENV["KEY"])
ctx = LLM::Context.new(llm)

loop do
  res = ctx.talk("What's the weather in Tokyo?")
  break unless ctx.pending_functions?

  puts "Model requested #{ctx.pending_functions.size} tool(s)"

  results = ctx.pending_functions.map do |fn|
    print "Run #{fn.name} with #{fn.arguments}? [y/N] "
    if $stdin.gets&.match?(/\Ay\z/i)
      fn.task(:thread).wait
    else
      fn.cancel(reason: "user declined")
    end
  end

  ctx.talk(results)
end

puts res.content

Why would I use it?

Manual control gives you pre-execution checks, custom confirmation flows, different strategies per tool, and fine-grained error recovery that the default tool loop does not expose.

Notes

LLM::Context#wait picks up pending functions, spawns them using the chosen strategy, waits for results, and records them back in the context. Each strategy is supported: :sequential, :thread, :fiber, :async, :fork, and :ractor. Functions are reset after each LLM::Context#wait or LLM::Context#talk call. Store the array if you need to preserve them.

Pending functions

Overview

After LLM::Context#talk returns, the context may have pending function calls if the model requested tools. These are available through LLM::Context#pending_functions, which returns an array of LLM::Function objects. Each function has a name, arguments, and methods for execution or cancellation.

How it works

When you want to check whether the model requested tools, call LLM::Context#pending_functions? after each LLM::Context#talk call. Each pending function has a name, arguments, and methods for execution or cancellation. Call LLM::Function#task to execute it or LLM::Function#cancel to skip it. Iterate over all pending functions to inspect or handle them individually:

res = ctx.talk "What's the weather in Tokyo?"

if ctx.pending_functions?
  puts "Model requested #{ctx.pending_functions.size} tool(s)"
  results = ctx.pending_functions.map do |fn|
    print "Run #{fn.name} with #{fn.arguments}? [y/N] "
    if $stdin.gets&.match?(/\Ay\z/i)
      fn.task(:thread).wait
    else
      fn.cancel(reason: "user declined")
    end
  end
  ctx.talk(results)
end

Why would I use it?

Inspecting pending functions lets you decide which tools to run, in what order, and with what strategy. This is essential for confirmation flows, selective execution, or logging which tools the model requested.

Notes

Pending functions are reset after each LLM::Context#wait or LLM::Context#talk call. If you need to preserve them, store the array before executing. Functions that are cancelled still count as completed from the model’s perspective; the model sees a cancellation result, not a tool error.

Tool responses

Overview

When a tool receives LLM::Interrupt, it can either cancel the turn or return a result. The choice depends on the situation. A hard cancel aborts the request outright and is the default. Returning a value lets the model adapt and continue the conversation, which can be useful when the interrupt is temporary, like a timeout or a user pause.

How it works

When a tool receives LLM::Interrupt, re-raise to abort the turn or return a value to continue the loop. The model receives the result and decides what to do next.

def call
  # do work
rescue LLM::Interrupt
  cleanup
  raise
end

def call
  # do work
rescue LLM::Interrupt
  cleanup
  {ok: false, reason: "interrupted"}
end

Why would I use it?

A hard cancel aborts the request outright. Useful when continuing would produce garbage. Returning a value lets the model adapt, which can be helpful when the interrupt is temporary.

Notes

The mechanism is the same across all six concurrency strategies. The :ractor strategy delivers the interrupt through ractor message passing. The :fork strategy delivers it via xchan.