Class: LLM::Context

Inherits:
Object
  • Object
show all
Includes:
Deserializer, Serializer
Defined in:
lib/llm/context.rb,
lib/llm/context/serializer.rb,
lib/llm/context/deserializer.rb

Overview

LLM::Context is the low-level stateful execution boundary in llm.rb. Most users should start with Agent, which wraps Context and manages tool loops automatically. Use Context directly when you need manual control over tool execution.

It holds the evolving runtime state for an LLM workflow: conversation history, tool calls and returns, schema and streaming configuration, accumulated usage, and request ownership for interruption.

This is broader than prompt context alone. A context is the object that lets one-off prompts, streaming turns, tool execution, persistence, retries, and serialized long-lived workflows all run through the same model.

A context can drive the chat completions API that all providers support or the Responses API on providers that expose it.

Examples:

#!/usr/bin/env ruby
require "llm"

llm = LLM.deepseek(key: ENV["KEY"])
ctx = LLM::Context.new(llm, stream: $stdout)
ctx.talk "If a train goes 60 mph for 1.5 hours, how far does it travel?"
ctx.messages.each { |m| puts "[#{m.role}] #{m.content}" }

See Also:

Defined Under Namespace

Modules: Deserializer, Serializer

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Methods included from Deserializer

#deserialize

Constructor Details

#initialize(llm, params = {}) ⇒ Context

Returns a new instance of Context.

Parameters:

  • llm (LLM::Provider) —

    A provider

  • params (Hash) (defaults to: {}) —

    The parameters to maintain throughout the conversation. Any parameter the provider supports can be included and not only those listed here.

Options Hash (params):

  • :mode (Symbol) —

    Defaults to :responses for OpenAI, otherwise it defaults to :completions.

  • :model (String) —

    Defaults to the provider's default model

  • :id (String) —

    A stable id for the context. It must be a UUIDv7 string, and anything else raises Error. Defaults to the record's id when the context is bound to a record whose id is a UUIDv7, and to a generated UUIDv7 otherwise.

  • :compactor (Class<LLM::Compactor>, nil) —

    A compactor class to use for context compaction. Defaults to LLM::Compactor::Null.

  • :compactor_options (Hash) —

    Options passed to the compactor's call method. Defaults to {}.

  • :filter (Class<LLM::MessageFilter>, nil) —

    A filter class to use for message filtering. Defaults to MessageFilter::Null.

  • :filter_options (Hash) —

    Options passed to the filter's call method. Defaults to {}.

  • :intercept (Class<LLM::Intercept>, nil) —

    Defaults to LLM::Intercept::Null.

  • :intercept_options (Hash) —

    Options passed to the intercept's call method. Defaults to {}.

  • :tools (Array<LLM::Function>, nil) —

    Defaults to nil

Raises:



144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
 
# File 'lib/llm/context.rb', line 144

def initialize(llm, params = {})
  params = {}.merge!(params)
  @llm = llm
  @record = params.delete(:record)
  @id = params.delete(:id) || makeid
  raise LLM::Error, "an id must be a UUIDv7 string" unless LLM::Utils.uuidv7?(@id)
  @mode = params.delete(:mode) || (llm.name == :openai ? :responses : :completions)
  tools = [*params.delete(:tools)]
  @params = {model: llm.default_model, schema: nil}.compact.merge!(params)
  @params[:tools] = tools unless tools.empty?
  @params[:store] ||= false if @mode == :responses
  @messages = LLM::Buffer.new(llm)
  extra = @params.slice(:model, :tools).merge!(ctx: self, tracer:)
  @params[:stream] = LLM::Stream.try(@params[:stream], extra:)
  @compactor = {
    klass: params.delete(:compactor) || LLM::Compactor::Null,
    options: params.delete(:compactor_options) || {}
  }
  @filter = {
    klass: params.delete(:filter) || LLM::MessageFilter::Null,
    options: params.delete(:filter_options) || {}
  }
  @intercept = {
    klass: params.delete(:intercept) || LLM::Intercept::Null,
    options: params.delete(:intercept_options) || {}
  }
  @retry_budget = params.delete(:retry_budget) || 0
end

Instance Attribute Details

#messages ⇒ LLM::Buffer<LLM::Message> (readonly)

Returns the accumulated message history for this context

Returns:



88
89
90
 
# File 'lib/llm/context.rb', line 88

def messages
  @messages
end

#llm ⇒ LLM::Provider (readonly)

Returns a provider

Returns:



93
94
95
 
# File 'lib/llm/context.rb', line 93

def llm
  @llm
end

#mode ⇒ Symbol (readonly)

Returns the context mode

Returns:

  • (Symbol) 


98
99
100
 
# File 'lib/llm/context.rb', line 98

def mode
  @mode
end

#record ⇒ Object? (readonly)

Returns the ORM record this context is bound to, or nil.

Returns:



103
104
105
 
# File 'lib/llm/context.rb', line 103

def record
  @record
end

#id ⇒ String (readonly)

Returns a stable id for this context. It is generated once on creation and restored with the runtime state on load - a context bound to a record takes the record's id instead, when that id is a UUIDv7 string - and it is always a UUIDv7 string.

Returns:

  • (String) 


111
112
113
 
# File 'lib/llm/context.rb', line 111

def id
  @id
end

#compacted ⇒ Boolean Also known as: compacted?

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns whether the context has been compacted and no later model response has cleared that state.

Returns:

  • (Boolean) 


206
207
208
 
# File 'lib/llm/context.rb', line 206

def compacted
  @compacted
end

Class Method Details

.validate_schema!(version:)

This method returns an undefined value.

Asserts that a payload can be restored by this runtime.

A payload is readable when it carries a version no newer than this runtime writes, so a payload from version 1 is still readable. A payload written by a newer runtime is refused, and one that carries no version at all is refused too.

Parameters:

  • version (Integer, nil) —

    The version a payload carries

Raises:

  • (LLM::Error) —

    When the payload cannot be restored



68
69
70
71
72
73
 
# File 'lib/llm/context.rb', line 68

def self.validate_schema!(version:)
  return if version and version <= schema_version
  raise LLM::Error, "the payload was written by another version of " \
                    "llm.rb (schema_version #{version.inspect}, this " \
                    "runtime reads up to #{schema_version})"
end

.params ⇒ Array<Symbol>

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns the set of runtime parameters that configure this context and must never be forwarded to a provider request body.

Returns:

  • (Array<Symbol>) 


81
82
83
 
# File 'lib/llm/context.rb', line 81

def self.params
  %w[intercept retry_budget concurrency filter compactor record id]
end

.schema_version ⇒ Integer

Returns the newest payload version this runtime can restore, and the version it writes.

A payload written by a newer runtime is refused, because a field this runtime does not know about would be dropped silently.

Returns:

  • (Integer) 


51
52
53
 
# File 'lib/llm/context.rb', line 51

def self.schema_version
  2
end

Instance Method Details

#registry ⇒ LLM::Registry

Returns:

See Also:



622
623
624
 
# File 'lib/llm/context.rb', line 622

def registry
  llm.registry
end

#retry_budget ⇒ Integer

Returns the retry budget for rate-limited requests.

Returns:

  • (Integer) 


176
177
178
 
# File 'lib/llm/context.rb', line 176

def retry_budget
  @retry_budget
end

#params ⇒ Hash

Returns the default params for this context

Returns:

  • (Hash) 


183
184
185
 
# File 'lib/llm/context.rb', line 183

def params
  @params.dup
end

#compactor ⇒ LLM::Compactor

Returns a context compactor

Returns:



190
191
192
 
# File 'lib/llm/context.rb', line 190

def compactor
  @compactor[:klass]
end

#compactor_options ⇒ Hash

Returns the options given to the context compactor

Returns:

  • (Hash) 


197
198
199
 
# File 'lib/llm/context.rb', line 197

def compactor_options
  @compactor[:options].dup
end

#intercept ⇒ LLM::Intercept

Returns:

See Also:



212
213
214
215
216
217
218
219
 
# File 'lib/llm/context.rb', line 212

def intercept
  klass = @intercept[:klass]
  intercept = Class === klass ? klass.new(llm) : klass
  intercept.tap do
    it.instance_variable_set(:@context, self)
    it.instance_variable_set(:@options, @intercept[:options])
  end
end

#filter ⇒ Class<LLM::MessageFilter>

Returns the configured filter class.

Filters rewrite the most recent message before it is sent to the provider.

Returns:



228
229
230
 
# File 'lib/llm/context.rb', line 228

def filter
  @filter[:klass]
end

#talk(prompt, params = {}) ⇒ LLM::Response

Interact with the context via the chat completions API. This method immediately sends a request to the LLM and returns the response.

Examples:

llm = LLM.openai(key: ENV["KEY"])
ctx = LLM::Context.new(llm)
res = ctx.talk("Hello, what is your name?")
puts res.messages[0].content

Parameters:

  • params (defaults to: {}) —

    The params, including optional :role (defaults to :user), :stream, :tools, :schema etc.

  • prompt (String) —

    The input prompt to be completed

Returns:



243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
 
# File 'lib/llm/context.rb', line 243

def talk(prompt, params = {})
  @owner = @llm.request_owner
  @compactor[:klass].new(self).call(**@compactor[:options])
  repair!(@messages, prompt)
  prompt, params, res = try { mode == :responses ? respond(prompt, params) : complete(prompt, params) }
  self.compacted = false
  if prompt.all?(&:tool_return?)
    @messages.concat prompt.map { LLM::Message.new(@llm.tool_role, _1.content, _1.extra) }
  else
    @messages.concat(prompt)
  end
  @messages.concat([res.choices[-1]].compact)
  stream.on_step(self, res)
  res
ensure
  @owner = nil
end

#ask(prompt, options = {}) {|String| ... } ⇒ LLM::Response

Ask a question and return the content string directly. Accepts with: for file attachments and a block for streaming, the way #talk does.

Parameters:

  • prompt (String)
  • options (Hash) (defaults to: {})

Options Hash (options):

  • :with (String, Array<String>, nil) —

    File path(s) to attach

  • :stream (#<<, LLM::Stream, nil) —

    A stream target

Yields:

  • (String) —

    content chunks when streaming

Returns:



274
275
276
277
278
279
280
281
282
283
284
285
286
 
# File 'lib/llm/context.rb', line 274

def ask(prompt, options = {}, &block)
  options = {with: nil, stream: nil}.merge!(options || {})
  with, stream = options.values_at(:with, :stream)
  prompt = with ? [prompt, [*with].map { local_file(_1) }] : prompt
  target = if block
    blk = block.dup
    blk.singleton_class.alias_method(:<<, :call)
    blk
  else
    stream
  end
  target ? talk(prompt, stream: target) : talk(prompt)
end

#inspect ⇒ String

Returns:

  • (String) 


290
291
292
293
294
 
# File 'lib/llm/context.rb', line 290

def inspect
  "#<#{LLM::Utils.object_id(self)} " \
  "@llm=#{@llm.class}, @mode=#{@mode.inspect}, @params=#{@params.inspect}, " \
  "@messages=#{@messages.inspect}>"
end

#pending_functions ⇒ Array<LLM::Function>

Returns an array of functions that can be called

Returns:



299
300
301
302
303
304
305
306
307
308
309
310
311
 
# File 'lib/llm/context.rb', line 299

def pending_functions
  return_ids = returns.map(&:id)
  @messages
    .select(&:assistant?)
    .flat_map do |msg|
      fns = msg.functions.select { _1.pending? and !return_ids.include?(_1.id) }
      fns.each do |fn|
        fn.tracer = tracer
        fn.model  = msg.model
        fn.intercept  = intercept
      end
    end.extend(LLM::Function::Array)
end

#pending_functions? ⇒ Boolean

Returns whether there is pending tool work in this context. This prefers queued streamed tool work when present, and otherwise falls back to unresolved functions derived from the message history.

Returns:

  • (Boolean) 


318
319
320
321
 
# File 'lib/llm/context.rb', line 318

def pending_functions?
  pending = queue
  (pending && !pending.empty?) || pending_functions.any?
end

#spawn(function, strategy) ⇒ LLM::Function::Task

Spawns a function through the context.

Parameters:

Returns:



329
330
331
 
# File 'lib/llm/context.rb', line 329

def spawn(function, strategy)
  function.task(strategy)
end

#returns ⇒ Array<LLM::Function::Return>

Returns tool returns accumulated in this context

Returns:



336
337
338
339
340
341
342
343
344
 
# File 'lib/llm/context.rb', line 336

def returns
  @messages
    .select(&:tool_return?)
    .flat_map do |msg|
      LLM::Function::Return === msg.content ?
        [msg.content] :
        [*msg.content].grep(LLM::Function::Return)
    end
end

#wait(strategy, except: []) ⇒ Array<LLM::Function::Return>

Waits for queued tool work to finish.

This prefers queued streamed tool work when the configured stream exposes a non-empty queue. Otherwise it falls back to waiting on the context's pending functions directly.

Parameters:

  • strategy (Symbol, Array<Symbol>) —

    If the stream queue already has tool work, wait will drain it without using this argument. Otherwise, this controls how pending functions are resolved directly. Use :sequential for sequential execution without spawning.

  • except (Array<LLM::Function>) (defaults to: []) —

    A list of functions to exclude from the wait

Returns:



361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
 
# File 'lib/llm/context.rb', line 361

def wait(strategy, except: [])
  if stream.queue.empty?
    ##
    # Every pending function is spawned as a task that checks its own
    # intercept (stamped on the function) before running. Blocked tasks
    # yield their intercept's return, so all pending calls still close.
    tools = except.empty? ? pending_functions : pending_functions - except
    @queue = tools.task(strategy)
    returns = @queue.wait
    emit_tool_returns(tools, returns)
    returns
  else
    @queue = stream.queue
    @queue.wait
  end
rescue LLM::Interrupt
  ##
  # Once, not once per tool. A cancel reaches every tool that is
  # running, and the caller hears one exception, so the phase is
  # announced here - the frame every strategy's interrupt unwinds
  # through - and before the caller is given it.
  tracer.on_interrupt(scope: :tool)
  raise
ensure
  @queue = nil
  @stream = nil
end

#interrupt! ⇒ nil Also known as: cancel!

Interrupt a turn: the request in flight, the tools that are running, and - when it is neither of those - the caller the turn is running under.

This is inspired by Go's context cancellation model, and the three are the whole of what a turn is doing. A request is closed at the socket, which is llm.interrupt!(owner): the transport keys its live requests by the owner the request was made on, and answers that it has nothing to close when there is no request registered under it. A tool is a task, and every task that is running is raised into. Between the two there is nothing: the loop is between two requests, or waiting out a retry, or building the next one - and until Agent#run_loop recorded the caller it is running under, an interrupt there was a cancel that did nothing at all.

The caller is the last resort rather than the first: the two precise interrupts are what close a socket and stop a tool, and what is left over is a turn that is running with nothing to point at. Ending that is what the caller is asked to do - it answers interrupt!, announces the phase to the tracer, and which of the thread, the fiber and the scheduler it holds receives the raise is its own business. See Agent::Interrupt.

Returns:

  • (nil) 


413
414
415
416
417
418
419
420
421
 
# File 'lib/llm/context.rb', line 413

def interrupt!
  llm.interrupt!(@owner)
  queue&.interrupt!
  pending_functions.each(&:interrupt!)
  @queue = nil
  @owner = nil
  @caller&.interrupt!
  nil
end

#token_usage ⇒ LLM::Usage Also known as: usage

Returns token usage accumulated in this context

Returns:



427
428
429
430
431
432
433
 
# File 'lib/llm/context.rb', line 427

def token_usage
  @messages
    .select(&:assistant?)
    .map(&:token_usage)
    .compact
    .reduce(LLM::Usage.zero, :+)
end

#context_used ⇒ Integer?

Returns the live context size (in tokens) of the most recent assistant message.

Returns:

  • (Integer, nil) —

    Returns the live context size (in tokens) of the most recent assistant message.



440
441
442
443
444
445
 
# File 'lib/llm/context.rb', line 440

def context_used
  @messages
    .find(&:assistant?)
    &.token_usage
    &.total_tokens
end

#context_usage ⇒ Rational?

Returns the fraction of the context window currently used. For example: Rational(100, 10_000), or nil when unknown

Returns:

  • (Rational, nil) —

    Returns the fraction of the context window currently used. For example: Rational(100, 10_000), or nil when unknown



451
452
453
454
455
456
457
458
459
 
# File 'lib/llm/context.rb', line 451

def context_usage
  return nil if @messages.size < 2
  used = context_used
  return nil if used.nil?
  total = context_window
  total.nil? || total <= 0 ? nil : Rational(used, total)
rescue LLM::NoSuchModelError, LLM::NoSuchRegistryError
  nil
end

#context_window ⇒ Integer?

Note:

This method returns nil when the context window size is not known to the runtime

Returns the model's context window. The context window is the maximum amount of input and output tokens a model can consider in a single request.

Returns:

  • (Integer, nil) 


469
470
471
472
473
474
475
 
# File 'lib/llm/context.rb', line 469

def context_window
  registry
    .limit(model:)
    .context
rescue LLM::NoSuchModelError, LLM::NoSuchRegistryError
  nil
end

#prompt(&b) ⇒ LLM::Prompt Also known as: build_prompt

Build a role-aware prompt for a single request.

Prefer this method over #build_prompt. The older method name is kept for backward compatibility.

Examples:

prompt = ctx.prompt do
  system "Your task is to assist the user"
  user "Hello, can you assist me?"
end
ctx.talk(prompt)

Parameters:

  • b (Proc) —

    A block that composes messages. If it takes one argument, it receives the prompt object. Otherwise it runs in prompt context.

Returns:



492
493
494
 
# File 'lib/llm/context.rb', line 492

def prompt(&b)
  LLM::Prompt.new(@llm, &b)
end

#image_url(url) ⇒ LLM::Object

Recongize an object as a URL to an image

Parameters:

  • url (String) —

    The URL

Returns:



503
504
505
 
# File 'lib/llm/context.rb', line 503

def image_url(url)
  LLM::Object.from(value: url, kind: :image_url)
end

#local_file(path) ⇒ LLM::Object

Recongize an object as a local file

Parameters:

  • path (String) —

    The path

Returns:



513
514
515
 
# File 'lib/llm/context.rb', line 513

def local_file(path)
  LLM::Object.from(value: LLM.File(path), kind: :local_file)
end

#remote_file(res) ⇒ LLM::Object

Reconginize an object as a remote file

Parameters:

Returns:



523
524
525
 
# File 'lib/llm/context.rb', line 523

def remote_file(res)
  LLM::Object.from(value: res, kind: :remote_file)
end

#tracer ⇒ LLM::Tracer

Returns an LLM tracer

Returns:



530
531
532
 
# File 'lib/llm/context.rb', line 530

def tracer
  @llm.tracer
end

#tracer=(other)

This method returns an undefined value.

Parameters:



538
539
540
 
# File 'lib/llm/context.rb', line 538

def tracer=(other)
  @llm.tracer = other || LLM::Tracer::Null.new(@llm)
end

#stream ⇒ LLM::Stream, ...

Returns a stream object, or nil

Returns:

  • (LLM::Stream, #<<, nil) —

    Returns a stream object, or nil



545
546
547
 
# File 'lib/llm/context.rb', line 545

def stream
  @stream || @params[:stream]
end

#model ⇒ String

Returns the model a Context is actively using

Returns:

  • (String) 


552
553
554
 
# File 'lib/llm/context.rb', line 552

def model
  messages.find(&:assistant?)&.model || @params[:model] || @llm.default_model
end

#created_at ⇒ Time?

Returns the time this context was created, derived from the timestamp embedded in its UUIDv7 id, or nil when it is not.

Returns:

  • (Time, nil) 


560
561
562
 
# File 'lib/llm/context.rb', line 560

def created_at
  @created_at ||= LLM::Utils.timestamp(@id)
end

#to_h ⇒ Hash

Returns the runtime state as a Hash.

context_used and context_window are projections written for queryability, so that saved state can be inspected at rest without loading it. They are never read back; the runtime always derives them from the messages and the registry.

Version 2 saves tool calls in the abstract form only. A version 1 payload carried the provider's wire form beside it, and that field is dropped on the way in - an adapter rebuilds its own blocks from the abstract form anyway.

Returns:

  • (Hash) 


577
578
579
580
581
582
583
584
585
586
587
588
 
# File 'lib/llm/context.rb', line 577

def to_h
  {
    schema_version: LLM::Context.schema_version,
    id: @id,
    model:,
    compacted:,
    context_used:,
    context_window:,
    token_usage: token_usage.to_h,
    messages: @messages.map { serialize_message(_1) }
  }
end

#to_json ⇒ String

Returns:

  • (String) 


592
593
594
 
# File 'lib/llm/context.rb', line 592

def to_json(...)
  LLM.json.dump(to_h, ...)
end

#serialize(path:) Also known as: save

This method returns an undefined value.

Save the current context state

Examples:

llm = LLM.openai(key: ENV["KEY"])
ctx = LLM::Context.new(llm)
ctx.talk "Hello"
ctx.save(path: "context.json")

Raises:

  • (SystemCallError) —

    Might raise a number of SystemCallError subclasses



606
607
608
 
# File 'lib/llm/context.rb', line 606

def serialize(path:)
  ::File.binwrite path, LLM.json.dump(to_h)
end

#cost ⇒ LLM::Cost

Returns an approximate cost for a given context based on both the provider, and model

Returns:

  • (LLM::Cost) —

    Returns an approximate cost for a given context based on both the provider, and model



615
616
617
 
# File 'lib/llm/context.rb', line 615

def cost
  LLM::Cost.from(self)
end