Module: LLM

Extended by:
LLM
Included in:
LLM
Defined in:
lib/llm.rb,
lib/llm/eval.rb,
lib/llm/lock.rb,
lib/llm/pipe.rb,
lib/llm/step.rb,
lib/llm/agent.rb,
lib/llm/error.rb,
lib/llm/utils.rb,
lib/llm/buffer.rb,
lib/llm/rescue.rb,
lib/llm/stream.rb,
lib/llm/tracer.rb,
lib/llm/console.rb,
lib/llm/context.rb,
lib/llm/message.rb,
lib/llm/session.rb,
lib/llm/uridata.rb,
lib/llm/version.rb,
lib/llm/contract.rb,
lib/llm/response.rb,
lib/llm/compactor.rb,
lib/llm/intercept.rb,
lib/llm/transport.rb,
lib/llm/tracer/null.rb,
lib/llm/eventhandler.rb,
lib/llm/json_adapter.rb,
lib/llm/providers/xai.rb,
lib/llm/providers/zai.rb,
lib/llm/tracer/logger.rb,
lib/llm/message_filter.rb,
lib/llm/providers/hanzo.rb,
lib/llm/request_adapter.rb,
lib/llm/providers/google.rb,
lib/llm/providers/ollama.rb,
lib/llm/providers/openai.rb,
lib/llm/tracer/telemetry.rb,
lib/llm/providers/alibaba.rb,
lib/llm/providers/bedrock.rb,
lib/llm/providers/mistral.rb,
lib/llm/providers/deepseek.rb,
lib/llm/providers/llamacpp.rb,
lib/llm/providers/moonshot.rb,
lib/llm/providers/anthropic.rb,
lib/llm/providers/deepinfra.rb,
lib/llm/providers/openrouter.rb,
lib/llm/tracer/enhanced_logger.rb

Overview

An agentic runtime.

Examples:

LLM::Agent

require "llm"
llm   = LLM.deepseek                             # 1. pick a provider
agent = LLM::Agent.new(llm, stream: $stdout)     # 2. create an agent
agent.talk "Hello world"                         # 3. talk to it

See Also:

Defined Under Namespace

Modules: ActiveRecord, Contract, Gem, Lock, RbConfig, RequestAdapter, Rescue, Sequel, Step, Utils Classes: A2A, Agent, Alibaba, Anthropic, Bedrock, Buffer, Command, Compactor, Console, Context, Cost, DeepInfra, DeepSeek, Error, Eval, File, Function, Google, Hanzo, Intercept, JSONAdapter, LlamaCpp, MCP, Message, MessageFilter, Mistral, Model, Moonshot, Object, Ollama, OpenAI, OpenRouter, Paint, Pipe, Prompt, Provider, Registry, Response, Schema, ServerTool, Stream, Tool, Tracer, Transport, URIData, Usage, XAI, ZAI

Constant Summary collapse

UNDEFINED =

This constant is part of a private API. You should avoid using this constant if possible, as it may be removed or be changed in the future. 

::Object.new.freeze
UnauthorizedError =

HTTPUnauthorized

Class.new(Error)
RateLimitError =

HTTPTooManyRequests

Class.new(Error)
InsufficientQuotaError =

A tokens-per-minute (TPM) rate limit. Alibaba reports this as Throttling.AllocationQuota / insufficient_quota. It is a rate limit (retriable), distinct from a request-rate limit only for classification.

Class.new(RateLimitError)
ServerError =

HTTPServerError

Class.new(Error)
NotFoundError =

HTTPNotFound

Class.new(Error)
FormatError =

When an given an input object that is not understood

Class.new(Error)
PromptError =

When given a prompt object that is not understood

Class.new(FormatError)
InvalidRequestError =

When given an invalid request

Class.new(Error)
ContextWindowError =

When the context window is exceeded

Class.new(InvalidRequestError)
Interrupt =

When a request is interrupted.

It sits outside StandardError, so that neither a bare rescue nor a rescue => e can swallow it by mistake: an interrupt is a request to stop, and a turn whose cancel was eaten looks like a turn that ignored one.

It is not a SignalException either. A signal is a framework's own condition rather than a task's - RSpec re-raises one that escapes an example, and an async reactor ends its own thread rather than failing the task that raised it - and an interrupt has to be something a caller can catch.

Class.new(Exception)
RactorError =

When a concurrency strategy cannot execute a given tool

Class.new(Error)
FiberError =

When a fiber tool's scheduler cannot hold a cancel until the tool starts.

A cancel that arrives before a call opens is issued at the call's first instruction, and only a scheduler that implements fiber_interrupt can be asked for a raise from there - it schedules the raise rather than issuing it. A scheduler that cannot is told so, rather than delivering the cancel before the call and never running the tool.

Name it in full. Ruby has a FiberError of its own, so a bare FiberError in a rescue written outside this namespace is core's rather than this one.

Class.new(Error)
NoSuchToolError =

When a tool call cannot be mapped to a local tool

Class.new(Error)
NoSuchModelError =

When Registry can't map a model

Class.new(Error)
NoSuchRegistryError =

When Registry can't map a registry

Class.new(Error)
LoadError =

When an optional runtime dependency cannot be required

Class.new(Error)
Repl =

Alias: LLM::Repl

Console
Session =
Deprecated.

Use Context instead. Scheduled for removal in v6.0.

Backward-compatible alias for LLM::Context

Context
VERSION = 
"19.0.1"

Class Method Summary collapse

Instance Method Summary collapse

Class Method Details

.File(obj) ⇒ LLM::File

Parameters:

  • obj (String, File, LLM::Response) —

    The path to the file, or an existing file reference

Returns:



89
90
91
92
93
94
95
96
97
98
 
# File 'lib/llm/file.rb', line 89

def LLM.File(obj)
  case obj
  when File
    obj.close unless obj.closed?
    LLM.File(obj.path)
  when LLM::File, LLM::Response then obj
  when String then LLM::File.new(obj)
  else raise TypeError, "don't know how to handle #{obj.class} objects"
  end
end

.debug(msg, io = debug_io)

Note:

This method is a noop when the debug_mode: method returns false and when the $LLMRB_DEBUG variable is not set.

This method returns an undefined value.

Logs a debug message to $stderr

Parameters:

  • msg (String)
  • io (IO) (defaults to: debug_io) —

    An optional IO



192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
 
# File 'lib/llm.rb', line 192

def self.debug(msg, io = debug_io)
  return unless @debug_mode || ENV["LLMRB_DEBUG_MODE"] == "1"
  log = -> do
    io.puts sprintf("runtime             llm.rb "    \
                    "\n"                             \
                    "runtime.version     %s"         \
                    "\n"                             \
                    "runtime.message     #{msg}"     \
                    "\n"                             \
                    "runtime.info"                   \
                    "\n"                             \
                    "  time.now            %s"       \
                    "\n"                             \
                    "  ractor.main         %s"       \
                    "\n"                             \
                    "  ractor.count        %s"       \
                    "\n"                             \
                    "\n",
                    LLM::VERSION.to_s,
                    Time.now.utc.to_s,
                    Ractor.main? ? "yes" : "no",
                    Ractor.count.to_s)
  end
  lock(:debug, &log)
end

.debug_io=(io)

Note:

Defaults to $stderr.

This method returns an undefined value.

Parameters:

  • io (IO) —

    An IO or file that debug writes to.



224
225
226
 
# File 'lib/llm.rb', line 224

def self.debug_io=(io)
  @io = io
end

.debug_io ⇒ IO

The IO used by the debug method. Defaults to $stderr.

Returns:

  • (IO) 


232
233
234
 
# File 'lib/llm.rb', line 232

def self.debug_io
  @io || $stderr
end

.debug_mode!

This method returns an undefined value.

Toggles debug mode



239
240
241
 
# File 'lib/llm.rb', line 239

def self.debug_mode!
  @debug_mode = !@debug_mode
end

.debug_mode? ⇒ Boolean

Are we in debug mode?

Returns:

  • (Boolean) 


246
247
248
 
# File 'lib/llm.rb', line 246

def self.debug_mode?
  !!@debug_mode
end

.require(name, version = nil) ⇒ Object

Requires an optional runtime dependency

Parameters:

  • name (String) —

    The name of a gem

  • version (String, nil) (defaults to: nil) —

    Optional gem version

Raises:



258
259
260
261
262
263
264
265
266
267
268
269
270
271
 
# File 'lib/llm.rb', line 258

def self.require(name, version = nil)
  names = {"xchan" => "xchan.rb",
            "net/http/persistent" => "net-http-persistent",
            "unicode/display_width" => "unicode-display_width"}
  gem(names[name] || name, version) if version
  super(name)
rescue ::LoadError
  name = names[name] || name
  raise LLM::LoadError,
    "#{name}#{version ? " #{version}" : ""} is an optional " \
    "runtime dependency but it does not appear to be installed. " \
    "Consider 'gem install #{name}', adding '#{name}' to your Gemfile or " \
    "opting out of the functionality provided by '#{name}'"
end

.registry_for(llm) ⇒ LLM::Object

Parameters:

  • llm (Symbol, LLM::Provider) —

    The name of a provider, or an instance of LLM::Provider

Returns:



277
278
279
280
281
282
 
# File 'lib/llm.rb', line 277

def self.registry_for(llm)
  lock(:registry) do
    name = Symbol === llm ? llm : llm.name
    @registry[name] ||= Registry.for(name)
  end
end

.lock(name, &b) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future. 



503
504
505
 
# File 'lib/llm.rb', line 503

def self.lock(name, &b)
  LLM::Lock.synchronize(name, &b)
end

.share!(obj) ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Note:

This method is used on class-level agent attributes that can be accessed across a Ractor boundary via a global reference (eg Agent.name).

Tries to deep-freeze an object so it can be shared across ractors. Might fail when the object cannot be deeply frozen. Failure is a noop that returns the parameter it is given.

Parameters:

Returns:



41
42
43
44
45
46
 
# File 'lib/llm.rb', line 41

def self.share!(obj)
  ::Ractor.make_shareable(obj)
rescue ::Ractor::Error
  debug "'#{obj.inspect}' cannot be made Ractor-shareable"
  obj
end

Instance Method Details

#mcp(**opts) ⇒ LLM::MCP

Parameters:

  • opts (Hash) —

    MCP client options

Options Hash (**opts):

  • :stdio (Hash, nil) —

    Standard I/O transport options

  • :stdio/:argv (Array<String>) —

    The command to run for the MCP process

  • :stdio/:env (Hash) —

    The environment variables to set for the MCP process

  • :stdio/:cwd (String, nil) —

    The working directory for the MCP process

Returns:



116
117
118
 
# File 'lib/llm.rb', line 116

def mcp(**opts)
  LLM::MCP.new(**opts)
end

#a2a(http:, binding: :rest) ⇒ LLM::A2A

Creates a new A2A client connected to a remote agent.

Parameters:

  • http (Hash, nil)
  • binding (Symbol) (defaults to: :rest) —

    The protocol binding to use. One of :rest or :jsonrpc

Options Hash (http:):

  • :url (String) —

    The base URL of the A2A agent (e.g., "https://agent.example.com")

  • :headers (Hash<String, String>) —

    Extra HTTP headers (e.g., Authorization)

  • :timeout (Integer, nil) —

    Request timeout in seconds

  • :transport (LLM::Transport, Class, nil) —

    Optional transport override

Returns:



135
136
137
 
# File 'lib/llm.rb', line 135

def a2a(http:, binding: :rest)
  LLM::A2A.http(**http, binding:)
end

#function(key, &b) ⇒ LLM::Function

Define a function

Examples:

LLM.function(:system) do |fn|
  fn.description "Run system command"
  fn.params do |schema|
    schema.object(command: schema.string.required)
  end
  fn.define do |command:|
    system(command)
  end
end

Parameters:

  • key (Symbol) —

    The function name / key

  • b (Proc) —

    The block to define the function

Returns:



154
155
156
 
# File 'lib/llm.rb', line 154

def function(key, &b)
  LLM::Function.new(key, &b)
end

#interrupt(agent: nil, id: nil, attempts: 1, interval: 0.1) ⇒ LLM::Agent, false

Interrupt an agent.

Parameters:

  • agent (LLM::Agent, nil) (defaults to: nil)
  • id (String, Integer, nil) (defaults to: nil)
  • attempts (Integer) (defaults to: 1)
  • interval (Numeric) (defaults to: 0.1)

Returns:

  • (LLM::Agent, false) —

    The agent that was interrupted, or false

See Also:



167
168
169
170
171
172
173
174
175
176
177
178
179
 
# File 'lib/llm.rb', line 167

def interrupt(agent: nil, id: nil, attempts: 1, interval: 0.1)
  attempts.times do |count|
    record = LLM::Agent.registry.find(agent:, id:)
    if record
      record.interrupt!
      return record
    else
      break if (count+1) == attempts
      sleep(interval)
    end
  end
  false
end

#json ⇒ Class

Returns the JSON adapter used by the library

Returns:

  • (Class) —

    Returns a class that responds to dump and load



288
289
290
 
# File 'lib/llm.rb', line 288

def json
  @json
end

#json=(adapter)

Note:

This should be set once from the main thread when your program starts. Defaults to LLM::JSONAdapter::JSON.

This method returns an undefined value.

Sets the JSON adapter used by the library

Parameters:

  • adapter (Class, String, Symbol) —

    A JSON adapter class or its name



300
301
302
303
304
305
306
307
308
309
310
311
312
313
 
# File 'lib/llm.rb', line 300

def json=(adapter)
  @json = case adapter.to_s
  when "JSON", "json" then JSONAdapter::JSON
  when "Oj", "oj" then JSONAdapter::Oj
  else
    is_class = Class === adapter
    is_subclass = is_class && adapter.ancestors.include?(LLM::JSONAdapter)
    if is_subclass
      adapter
    else
      raise TypeError, "Adapter must be a subclass of LLM::JSONAdapter"
    end
  end
end

#anthropic(key: UNDEFINED) ⇒ Object 



319
320
321
322
323
324
325
326
 
# File 'lib/llm.rb', line 319

def anthropic(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/anthropic" unless defined?(LLM::Anthropic) }
  if key == UNDEFINED
    LLM::Anthropic.new(key: key(name: __method__), **)
  else
    LLM::Anthropic.new(key:, **)
  end
end

#google(key: UNDEFINED) ⇒ Object 



331
332
333
334
335
336
337
338
 
# File 'lib/llm.rb', line 331

def google(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/google" unless defined?(LLM::Google) }
  if key == UNDEFINED
    LLM::Google.new(key: key(name: __method__), **)
  else
    LLM::Google.new(key:, **)
  end
end

#ollama(key: nil) ⇒ Object 



343
344
345
346
 
# File 'lib/llm.rb', line 343

def ollama(key: nil, **)
  lock(:require) { require_relative "llm/providers/ollama" unless defined?(LLM::Ollama) }
  LLM::Ollama.new(key:, **)
end

#llamacpp(key: nil) ⇒ Object 



351
352
353
354
 
# File 'lib/llm.rb', line 351

def llamacpp(key: nil, **)
  lock(:require) { require_relative "llm/providers/llamacpp" unless defined?(LLM::LlamaCpp) }
  LLM::LlamaCpp.new(key:, **)
end

#deepseek(key: UNDEFINED) ⇒ Object 



359
360
361
362
363
364
365
366
 
# File 'lib/llm.rb', line 359

def deepseek(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/deepseek" unless defined?(LLM::DeepSeek) }
  if key == UNDEFINED
    LLM::DeepSeek.new(key: key(name: __method__), **)
  else
    LLM::DeepSeek.new(key:, **)
  end
end

#hanzo(key: UNDEFINED) ⇒ Object 



371
372
373
374
375
376
377
378
 
# File 'lib/llm.rb', line 371

def hanzo(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/hanzo" unless defined?(LLM::Hanzo) }
  if key == UNDEFINED
    LLM::Hanzo.new(key: key(name: __method__), **)
  else
    LLM::Hanzo.new(key:, **)
  end
end

#openai(key: UNDEFINED) ⇒ Object 



383
384
385
386
387
388
389
390
 
# File 'lib/llm.rb', line 383

def openai(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/openai" unless defined?(LLM::OpenAI) }
  if key == UNDEFINED
    LLM::OpenAI.new(key: key(name: __method__), **)
  else
    LLM::OpenAI.new(key:, **)
  end
end

#deepinfra(key: UNDEFINED) ⇒ Object 



395
396
397
398
399
400
401
402
 
# File 'lib/llm.rb', line 395

def deepinfra(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/deepinfra" unless defined?(LLM::DeepInfra) }
  if key == UNDEFINED
    LLM::DeepInfra.new(key: key(name: __method__), **)
  else
    LLM::DeepInfra.new(key:, **)
  end
end

#xai(key: UNDEFINED) ⇒ Object 



408
409
410
411
412
413
414
415
 
# File 'lib/llm.rb', line 408

def xai(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/xai" unless defined?(LLM::XAI) }
  if key == UNDEFINED
    LLM::XAI.new(key: key(name: __method__), **)
  else
    LLM::XAI.new(key:, **)
  end
end

#mistral(key: UNDEFINED) ⇒ Object 



421
422
423
424
425
426
427
428
 
# File 'lib/llm.rb', line 421

def mistral(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/mistral" unless defined?(LLM::Mistral) }
  if key == UNDEFINED
    LLM::Mistral.new(key: key(name: __method__), **)
  else
    LLM::Mistral.new(key:, **)
  end
end

#zai(key: UNDEFINED) ⇒ Object 



434
435
436
437
438
439
440
441
 
# File 'lib/llm.rb', line 434

def zai(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/zai" unless defined?(LLM::ZAI) }
  if key == UNDEFINED
    LLM::ZAI.new(key: key(name: __method__), **)
  else
    LLM::ZAI.new(key:, **)
  end
end

#moonshot(key: UNDEFINED) ⇒ Object 



447
448
449
450
451
452
453
454
 
# File 'lib/llm.rb', line 447

def moonshot(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/moonshot" unless defined?(LLM::Moonshot) }
  if key == UNDEFINED
    LLM::Moonshot.new(key: key(name: __method__), **)
  else
    LLM::Moonshot.new(key:, **)
  end
end

#openrouter(key: UNDEFINED) ⇒ Object 



460
461
462
463
464
465
466
467
 
# File 'lib/llm.rb', line 460

def openrouter(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/openrouter" unless defined?(LLM::OpenRouter) }
  if key == UNDEFINED
    LLM::OpenRouter.new(key: key(name: __method__), **)
  else
    LLM::OpenRouter.new(key:, **)
  end
end

#alibaba(key: UNDEFINED) ⇒ Object Also known as: aliyun 



473
474
475
476
477
478
479
480
 
# File 'lib/llm.rb', line 473

def alibaba(key: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/alibaba" unless defined?(LLM::Alibaba) }
  if key == UNDEFINED
    LLM::Alibaba.new(key: key(name: :alibaba), **)
  else
    LLM::Alibaba.new(key:, **)
  end
end

#bedrock(access_key_id: UNDEFINED, secret_access_key: UNDEFINED, region: UNDEFINED) ⇒ Object 



486
487
488
489
490
491
492
493
494
495
496
497
498
499
 
# File 'lib/llm.rb', line 486

def bedrock(access_key_id: UNDEFINED, secret_access_key: UNDEFINED, region: UNDEFINED, **)
  lock(:require) { require_relative "llm/providers/bedrock" unless defined?(LLM::Bedrock) }
  if [access_key_id, secret_access_key, region].any? { _1 == UNDEFINED }
    access_key_id = ENV["AWS_ACCESS_KEY_ID"] if access_key_id == UNDEFINED
    secret_access_key = ENV["AWS_SECRET_ACCESS_KEY"] if secret_access_key == UNDEFINED
    region = ENV["AWS_REGION"] if region == UNDEFINED
    if access_key_id.to_s.strip.empty? || secret_access_key.to_s.strip.empty?
      raise ArgumentError, "you must provide an API key"
    end
    LLM::Bedrock.new(access_key_id:, secret_access_key:, region:, **)
  else
    LLM::Bedrock.new(access_key_id:, secret_access_key:, region:, **)
  end
end