r.uby.dev project.
Welcome to the canonical llm.rb repository.
llm.rb is a runtime for building agentic AI applications on CRuby. It runs on Ruby's standard library by default, it supports concurrent and parallel tool execution and has a single coherent API that spans 15+ providers. The README covers a lot of ground and the changelog tracks what has changed between releases.
Install
v17
The v17 series requires Ruby 3.4 or later.
This version works with older rubies and dependencies (ie: json v2).
gem install llm.rb -v 17.0.1
Latest (v19)
The v19 release is a smaller release that includes a tiny eval framework that was authored for the micro1 hackathon. It is the recommended release.
It builds on the v18 release, which includes substational runtime improvements - especially for running tool calls on a Ractor. Like v18, it requires Ruby4+, JSON v3, and the latest versions of ActiveRecord|Sequel (optional).
gem install llm.rb
Quick start
Resources
r.uby.dev is the place to go if you have a question the README doesn't answer or want to see the runtime in action. Great way to learn about the runtime and see it in action at the same time. API docs also exists for people who prefer learning the traditional way. For everything else, there's the README.
Agents
The
LLM::Agent
class is the default high-level interface,
and it is recommended for most use-cases. It manages the tool loop
and provides configurable features on top of it. For example you can
manage the tool loop with a retry budget and a tool call budget -
alongside other features.
The runtime is designed to keep the tool loop alive and it will rescue exceptions. When an exception is encountered in a tool it is reported back to the model as an in-band error that allows the model to change course or retry with different parameters.
A tool call requires a tool return (or response), and the lack of one can corrupt the conversation and lead to API-level errors from a provider. But sometimes it is unavoidable (for example, via an interrupt or power loss) so the runtime automatically closes tool calls that fall into that category by telling the model the tool call(s) were cancelled.
helloworld.rb
A classic "hello world" example.
The environment variable
$DEEPSEEK_API_KEY is assumed to have been set and reference
a valid DeepSeek API key, otherwise you can provide a key explicitly
(eg: LLM.deepseek(key: ENV["MYKEY"])) and you could swap
deepseek for any of the other 14 providers as well
(eg: llm = LLM.openai):
require "llm"
require "stringio"
class GetTime < LLM::Tool
name "get-time"
description "returns the current time"
def call
{now: Time.now}
end
end
llm = LLM.deepseek
= {stream: $stdout, tools: [GetTime], concurrency: :ractor}
agent = LLM::Agent.new(llm, )
agent.talk "hello world"
agent.talk "what time is it?"
eval.rb
Show GIF
An eval is like a unit test but for your agent. The runtime includes its own tiny eval framework and a runner that can run your evals. The evals emit both text and json reports that provide deep analysis of what an agent did, what a model said, and what tool calls were called.
An eval score starts at 1.0. The score method (which receives
an instance of LLM::Agent) can then reduce this score
from a perfect 1.0 all the way down to zero - based on the
results. What a "pass" means is up to you, you might require
1.0 for a pass, or 0.8 could be acceptable.
The criteria with which you score an agent and the logic behind it is also yours to define.
An eval is defined as a single directory with an eval.rb
file, one or more stepX.{txt,md} files, and any other
optional files that support the eval. Let's take a look
at one. The following example can be run with
llm.rb eval -p openai evals/simple-eval/ or checkout a
simulated eval that challenges the model to solve a
FreeBSD-specific problem: evals/freebsd-update-eval/.
This work was completed for the micro1 hackathon:
CLI
1 sample
llm.rb eval -p openai -m gpt-6-sol evals/freebsd-update-eval/
3 samples
llm.rb eval -p openai -m gpt-6-sol -n 3 evals/freebsd-update-eval/
evals/simple-eval/step1.txt
Delete /tmp/file.txt
evals/simple-eval/eval.rb
LLM::Eval.new do
set name: "simple-eval",
description: "can the agent delete a file?"
step "step1.txt"
setup do
require "fileutils"
FileUtils.touch "/tmp/file.txt"
end
score do |agent|
if File.exist?("/tmp/file.txt")
penalize!(by: 1.0, why: "/tmp/file.txt should have been deleted")
elsif agent.returns.size > 3
penalize!(by: 0.5, why: "too many tool calls for this eval")
end
end
configure do
agent do
##
# The agent being tested.
# Any subclass of {LLM::Agent} works.
# Also works with ActiveRecord|Sequel models.
LLM::Agent.new(
llm,
tools: [LLM::Tool::Ls, LLM::Tool::Pwd, LLM::Tool::Exec],
tracer:,
intercept:
)
end
tracer do
LLM::Tracer.multi(
llm,
LLM::Tracer::Eval::Console.new(llm, eval: self),
LLM::Tracer::Eval::JSON.new(llm, eval: self)
)
end
intercept do
Class.new(LLM::Intercept) do
define_method(:call) do |function:|
# ...
end
end
end
end
end
intercept.rb
The LLM::Intercept class sits between when a model
requests a tool and when the runtime calls it. An intercept
can rewrite tool call parameters before they're sent to the
tool - essentially rewriting the model's request - or refuse
the tool call by returning its own LLM::Function::Return.
It covers MCP and A2A tool calls as well:
require "llm"
require "llm/tools/exec"
class Sandbox < LLM::Intercept
def call(function:)
case function.name
when "exec"
refused = ["sh", "rm", "reboot"]
command = function.arguments.command
function.return(
error: true,
kind: "sandbox_error",
message: "#{command} is forbidden"
) if refused.include?(command)
end
end
end
llm = LLM.deepseek
agent = LLM::Agent.new(llm, tools: [LLM::Tool::Exec], intercept: Sandbox)
agent.talk "delete README.md", stream: $stdout
interrupt.rb
The
LLM.interrupt
method has access to a process-wide
registry that contains every active instance of
LLM::Agent,
and that includes Sequel and ActiveRecord agents, too. An
agent enters the registry when it starts a turn, and it
exits the registry afterwards. The method returns the
agent it interrupted, and otherwise it returns false.
There is often a window between when an agent is queued and when it runs, so a poll approach lets you eventually interrupt the agent, or give up trying:
require "llm"
require "llm/tools/read_file"
llm = LLM.deepseek
concurrency = :ractor
agent = LLM::Agent.new(llm, concurrency:)
Thread.new do
LLM.interrupt(
id: agent.id,
attempts: 50,
interval: 0.1
)
end
begin
stream = $stdout
tools = [LLM::Tool::ReadFile]
agent.talk("summarize README.md", stream:, tools:)
rescue LLM::Interrupt
puts "!! interrupted !!"
end
mcp.rb
The MCP example uses the HTTP transport to connect to GitHub's MCP server. The example uses a Personal Access Token (get yours here) although it could also use a token that was obtained through OAuth and belongs to a user other than you:
require "llm"
llm = LLM.deepseek
mcp = LLM::MCP.http(
url: "https://api.githubcopilot.com/mcp/",
headers: {"Authorization" => "Bearer #{ENV['GITHUB_PAT']}"}
)
##
# Roll the dice
concurrency = [
:sequential, :async,
:thread , :ractor,
Gem.win_platform? ? nil : :fork
].compact.sample
##
# Let's go
warn "concurrency=#{concurrency}"
agent = LLM::Agent.new(llm, stream: $stdout, concurrency:)
agent.talk "Find me an interesting Ruby project on GitHub", tools: mcp.tools
files.rb
OpenAI, DeepSeek, Google, and a select few other providers provide a Files API that can be used to upload a file and then reference it by ID from a conversation. This approach can be more efficient than embedding an image as a base64-encoded string but not every provider has a files API and it is a separate step:
require "llm"
llm = LLM.deepseek
agent = LLM::Agent.new(llm)
file = llm.files.create(file: "rubydev.png")
agent.talk ["what's this?", file], stream: $stdout
llm.files.delete(file:)
The files API requires you to manage the files that are uploaded. Sometimes it can be easier and more simple to embed the file directly in the conversation. The image is encoded into a Base64 string and carried around in memory until the agent and its messages are garbage collected. This capability varies by provider and model:
require "llm"
llm = LLM.deepseek
agent = LLM::Agent.new(llm, stream: $stdout)
agent.ask "what's this?", with: ["rubydev.png"]
stream.rb
The LLM::Stream class
can be notified when the model requests a tool call, and then decide how to
launch the tool call. The example launches mcp-based tool call(s) on their
own Ractor(s) and other tool
call(s) on their own
Thread(s).
This might seem exotic at first but it is genuinely useful to be able to choose between a Ractor and something else. Ractor won't work in all scenarios but that doesn't mean we can't use them in the scenarios where they work well:
require "llm"
require "llm/tools/read_file"
llm = LLM.deepseek
mcp = LLM::MCP.http(
url: "https://api.githubcopilot.com/mcp/",
headers: {"Authorization" => "Bearer #{ENV['GITHUB_PAT']}"}
)
class Stream < LLM::Stream
def on_tool_call(tool)
if tool.mcp?
puts "spawn #{tool.name} on ractor"
queue << tool.task(:ractor)
else
puts "spawn #{tool.name} on thread"
queue << tool.task(:thread)
end
end
end
tools = [*LLM::Tool.subclasses, *mcp.tools]
agent = LLM::Agent.new(llm, tools:, stream: Stream.new)
agent.talk("what's happening on r-uby-dev/llm?")
Essentials
Stream
A stream can be a simple IO object (eg `$stdout`) or it can be a subclass of [`LLM::Stream`](https://r.uby.dev/api-docs/llm.rb/LLM/Stream.html). An IO object can receive content but it cannot receive the other callbacks that are available to subclasses of [`LLM::Stream`](https://r.uby.dev/api-docs/llm.rb/LLM/Stream.html). It is generally a good idea to return as quickly as possible from all of these methods, but especially for the `on_content` and `on_reasoning_content` methods: they run inline with the request. A subclass of [`LLM::Stream`](https://r.uby.dev/api-docs/llm.rb/LLM/Stream.html) can implement callbacks that the runtime will call throughout an agent's lifetime. All callbacks are optional. The callbacks provide for content, reasoning, tool calls, tool returns, steps in a turn, retries, compaction and more: ```ruby class Stream < LLM::Stream ## # @param [String] content # A chunk of text def on_content(content) print content end ## # @param [String] content # A chunk of text def on_reasoning_content(content) warn content end ## # @param [LLM::Function] tool # The tool being called def on_tool_call(tool) nil end ## # @param [LLM::Function] tool # The tool that returned # @param [LLM::Function::Return] result # The return from the tool call def on_tool_return(tool, result) nil end ## # @note # This method is called _before_ a filter runs # @param [LLM::MessageFilter] filter # A filter def on_filter(filter) nil end ## # @note # This method is called _after_ a filter runs # @param [LLM::MessageFilter] filter # A filter def on_filter_finish(filter) nil end ## # @note # This method is called _before_ a compactor runs # @param [LLM::Compactor] compactor # A compactor def on_compaction(compactor) nil end ## # @note # This method is called _after_ a compactor runs # @param [LLM::Compactor] compactor # A compactor def on_compaction_finish(compactor) nil end ## # @note # This method is called once per request # in a turn (which can contain multiple # requests) # @param [LLM::Context] ctx # The context # @param [LLM::Response] res # The response def on_step(ctx, res) nil end ## # @note # This method is called when a request is # rate limited and retried. # @param [LLM::RateLimitError] error # @param [Integer] attempt def on_retry(error, attempt) nil end end llm = LLM.deepseek agent = LLM::Agent.new(llm, stream: Stream.new) agent.talk "Explain Ruby fibers." ```
Tools
An agent requires one or more tools to be able to interact with the "outside" world. At a high-level a tool is how a model can access your filesystem, search the internet, post a comment on your behalf and anything else that your own code could do. The runtime represents a tool as a subclass of [`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) that provides a name, a description, an optional set of parameters and a method that the runtime will call on the model's behalf. A tool is implemented on top of an [`LLM::Function`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html) object, and you might come across it in stream and tracer callbacks. The model decides when and how a tool is called: ```ruby class ReadFile < LLM::Tool name "read-file" description "Read a file" parameter :path, String, "The filename or path" required %i[path] def call(path:) {contents: File.read(path)} end end llm = LLM.deepseek agent = LLM::Agent.new(llm, tools: [ReadFile], stream: $stdout) agent.talk "summarize README.md" ```
Schema
The [`LLM::Schema`](https://r.uby.dev/api-docs/llm.rb/LLM/Schema.html) class describes an object that has properties of different types, and it can also reference other objects, or schemas. The object describes the shape of a JSON object that the model will return in place of the usual free-form text that a model replies with. This feature is implemented with [JSONSchema](https://json-schema.org/) under the hood and a large portion of the spec is supported: objects, enums, arrays, nested objects, strings, integers, and so on. [`LLM::Schema`](https://r.uby.dev/api-docs/llm.rb/LLM/Schema.html) also provides the parameter definitions available to [`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) subclasses and the same range of types are supported. The [`LLM::DeepSeek`](https://r.uby.dev/api-docs/llm.rb/LLM/DeepSeek.html) provider does not support structured outputs officially but the runtime implements structured outputs on top of the `json_object` API and it works the same as all other providers who do support structured outputs: ```ruby class Forecast < LLM::Schema property :city, String, "The city name" property :temperature, Number, "Current temperature" property :conditions, String, "Weather conditions" required %i[city temperature conditions] end llm = LLM.deepseek agent = LLM::Agent.new(llm) res = agent.talk "Weather in Paris?", schema: Forecast res.content! # => {city: "Paris", temperature: 15.0, conditions: "Cloudy"} ```
Serialization
Both [`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html) and [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) can be serialized to JSON and written to disk. This feature is what supports the ActiveRecord and Sequel integrations too but rather than store the agent directly on disk it is stored in a database column instead. An agent can be configured to read from and write to a file automatically with the `path` option. When the file already exists, the agent is restored from the file and continues where he left off. After each turn the agent flushes its state to the file. The text file can be shared like any other text file and it can be used to restore the agent in another process or machine: ```ruby require "llm" path = File.join(Dir.home, ".agents", "myagent.json") llm = LLM.deepseek agent = LLM::Agent.new(llm, path:) agent.talk "remember my name is robert" ## # Resume the conversation where the agent left off agent = LLM::Agent.new(llm, path:) agent.talk "what's my name?" ```
Tracer
An agent can have its own tracer that observes runtime events throughout its lifetime. A tracer is local to an agent and the tracer runs on the same thread as the agent. There are three builtin tracers, and a fourth tracer that lets you combine multiple tracers into a single tracer - so you can log to a file, and export to OpenTelemetry at the same time. All tracers protect themselves and the agent being traced from a crash through a rescue that dumps a tracer error to standard error (`$stderr`). Beyond the default tracers, it is not unusual to write your own. A tracer can let you export runtime state to your own database, an external service, or any other medium you might have in your stack. A new tracer can be created through a subclass of [`LLM::Tracer`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html) that implements the expected callback methods. This feature also works for ActiveRecord and Sequel models. The builtin tracers: * [`LLM::Tracer.enhanced`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#enhanced_logger-class_method): human-readable single-line logs to stderr, ideal during development. * [`LLM::Tracer.telemetry`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#telemetry-class_method): exports spans via OTLP for OpenTelemetry in production. * [`LLM::Tracer.logger`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#logger-class_method): structured JSON to stdout or a file. * [`LLM::Tracer.multi`](https://r.uby.dev/api-docs/llm.rb/LLM/Tracer.html#multi-class_method): fans out every callback to multiple tracers. A simple example: ```ruby llm = LLM.deepseek tracer = LLM::Tracer.enhanced(llm) agent = LLM::Agent.new(llm, tracer:) agent.talk "Hello" ``` A multi-tracer example: ```ruby llm = LLM.deepseek tracer = LLM::Tracer.multi( llm, LLM::Tracer::Logger, LLM::Tracer::Telemetry, LLM::Tracer::Foobar.new(llm, {option1: "foo", option2: "bar"}) ) agent = LLM::Agent.new(llm, tracer:) agent.talk "Hello" ```
MCP
The runtime can communicate with MCP servers via HTTP(S) and stdio. The stdio transport spawns a local process that uses its standard input and output streams for communication. The runtime provides a full-featured MCP client and the tools that an MCP server has announced are available as [`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) subclasses that can be used like any other subclass of [`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html). The first example uses the HTTP transport to connect to GitHub's MCP server. The example uses a Personal Access Token (get yours [here](https://github.com/settings/apps)) although it could also use a token that was obtained through OAuth and belongs to a user other than you: ```ruby require "llm" llm = LLM.deepseek mcp = LLM::MCP.http( url: "https://api.githubcopilot.com/mcp/", headers: {"Authorization" => "Bearer #{ENV['GITHUB_PAT']}"} ) ## # Roll the dice concurrency = [ :sequential, :async, :thread , :ractor, Gem.win_platform? ? nil : :fork ].compact.sample ## # Let's go warn "concurrency=#{concurrency}" agent = LLM::Agent.new(llm, stream: $stdout, concurrency:) agent.talk "Find me an interesting Ruby project on GitHub", tools: mcp.tools ``` The second example focuses on the second transport. It is recommended (but not required) to launch a stdio MCP server with a session so that the process is kept alive for as long as you use it, otherwise the process can be launched multiple times and it can add latency to your requests. The example uses a local stdio MCP server from the [sem](https://github.com/Ataraxy-Labs/sem) project.
It works on top of git and parses your code with tree-sitter: ```ruby require "llm" mcp = LLM::MCP.stdio( argv: ["sem", "mcp"], env: {'SEM_REPO' => Dir.getwd}, cwd: Dir.getwd, ) ## # 'mcp.session' is optional, but recommended. # It can be dropped but then the process may # be launched once per-request. mcp.session do llm = LLM.deepseek tracer = LLM::Tracer.enhanced(llm) tools = mcp.tools.select { _1.name == "sem_log" } stream = $stdout agent = LLM::Agent.new(llm, tracer:, stream:, tools:) agent.talk "summarize recent git activity" end ```
Console
The [LLM::Agent#console](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#console-instance_method) method drops you into an interactive console that is built on top of (n)curses. The `llm.rb` executable packaged with the gem is another way to access the console and ActiveRecord models who have called `acts_as_agent` can access the console as well (via `agent.console`). A console for an ActiveRecord model does not write back to the database. The `llm.rb` executable automatically associates a session with the current working directory and it can be resumed by calling `llm.rb` in the same directory at a later point. The console is not intended to compete with Claude, Codex and friends. It is much more limited, serves an entirely different purpose and is more like a debugger for your agents. The dependencies required by the console are not installed by default, and the easiest way to grab them is via `gem install llm-shell`. ##### Demo 
Budgets
An agent has a retry budget, and a tool call budget. The retry budget is enabled by default, and it determines how many times we will retry when a request raises an error. The errors that are retried are returned by [`LLM::Provider#retry_errors`](https://r.uby.dev/api-docs/llm.rb/LLM/Provider.html#retry_errors-instance_method) and can vary by provider but typically cover rate limits, and time outs. Between retries we will sleep, and each retry sleeps a little longer (`sleep 2.0 * attempts`) than the last. The default retry budget is usually 5 but it can also vary by provider and it is determined by the [`LLM::Provider#retry_budget`](https://r.uby.dev/api-docs/llm.rb/LLM/Provider.html#retry_budget-instance_method) method. When the budget is spent but the an exception continues to raise we will give up and let the exception bubble up to the caller. The budget can be disabled by setting `retry_budget` to zero: ```ruby require "llm" llm = LLM.deepseek agent = LLM::Agent.new(llm, retry_budget: 0) ``` The second budget describes how many tool calls are allowed in a single turn, and it is disabled by default. When the limit is reached or breached the agent will reply with an in-band error that tells the model it has spent its tool call budget, and that it should stop requesting tool calls for this turn. This exchange repeats until the model stops requesting tool calls. The budget is restored after a turn completes and the model will normally provide a response from the tool calls made before the budget was spent: ```ruby require "llm" require "llm/tools" llm = LLM.deepseek tools = LLM::Tool.subclasses agent = LLM::Agent.new(llm, tools:, tool_budget: 25) agent.talk "fix every FIXME in this repository" ```
Usage|Cost
An agent knows how many tokens it has spent, and how much those tokens cost. The runtime embeds the [models.dev](https://models.dev) registry (only the parts it uses, and on-demand) that gives it access to model information that includes costs and limits. The `token_usage` method sums the whole conversation as a [LLM::Usage](https://r.uby.dev/api-docs/llm.rb/LLM/Usage.html) object and [LLM::Cost](https://r.uby.dev/api-docs/llm.rb/LLM/Cost.html) combines it with the model registry to determine how much money has been spent. These numbers should not be taken as gospel though. They're a best-guess estimate, and your provider's dashboard might disagree: ```ruby require "llm" llm = LLM.deepseek agent = LLM::Agent.new(llm) agent.talk "Hello" agent.token_usage # => LLM::Usage for the whole conversation agent.cost # => LLM::Cost, priced from the registry ``` The `context_*` methods can tell you about the model's context window: its total capacity, and how much of it have we already spent: ```ruby agent.context_used # => tokens in the latest turn agent.context_window # => the model's limit, or nil when unknown agent.context_usage # => Rational, eg Rational(100, 10_000) ```
class Agent < LLM::Agent
The examples so far have focused on [LLM::Agent](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) as an object but you can create a subclass that inherits from [LLM::Agent](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) too. Both support the same features and accept the same options but a subclass can be configured with the `set` method, implement its own methods, inherit from other modules and create multiple instances of itself. The [`LLM::Agent.set`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#set-class_method) method can be used to configure your agent, and it accepts all the same options that an instance of [LLM::Agent](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) accepts. An option can be resolved lazily via a Proc or a Symbol that represents a method on your class, and all options are optional - you can set many or none, and your subclass will still work. For example: ```ruby require "llm" require "llm/tools" class Agent < LLM::Agent set name: "dennis", description: "a unix expert", instructions: proc { File.read("instructions.md") }, concurrency: :fork, tools: :toolset private def toolset [LLM::Tool::Exec] end end llm = LLM.deepseek agent = Agent.new(llm) agent.talk "What's our uptime?", stream: $stdout ```
Database
ActiveRecord
Both [`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html) and [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) can be serialized to JSON and stored in a database column. The `jsonb` column type from PostgreSQL is recommended but it can also be stored as a string on other databases. ActiveRecord support is optimized for the `jsonb` column type and PostgreSQL. The column captures everything an agent has done up to that point, it includes tool calls and returns, exchanged messages, and other metadata that carries runtime state. Each agent is an ActiveRecord model that calls `acts_as_agent` and each row represents an instance of that agent. It can be used with new and existing models alike. The column should have the name `data` but this can be changed when the `acts_as_agent` method is called (eg `acts_as_agent(data_column: :my_column)`). The column is updated after every request that an agent makes rather than every turn, so an unexpected interrupt can be resumed from from the last request and no progress (or spent tokens) are lost: ```ruby require "active_record" require "llm" require "llm/active_record" class Agent < ActiveRecord::Base acts_as_agent(format: :jsonb) do |agent| agent.set name: "chris", description: "a GitHub expert", instructions: proc { File.read("instructions.md") }, tools: :toolset, concurrency: :ractor, tracer: proc { LLM::Tracer.enhanced(llm) } end ## # @return [Array
SQL optimizations
In a database environment the runtime optimizes for the PostgreSQL database and its builtin support for the `jsonb` column type. An agent fits in a single column, on a single row, and that column carries everything it has done: messages, tool calls, context usage, and so on. It works well in practice and means you can store an agent almost anywhere. For scenarios where performance matters most the runtime ships with virtual ActiveRecord classes that never materialize in your database but provide a SQL view into the column where an agent stores its runtime state. They return [`ActiveRecord::Relation`](https://api.rubyonrails.org/classes/ActiveRecord/Relation.html) objects, so the filtering happens in the database. ```ruby class Agent < ActiveRecord::Base acts_as_agent(format: :jsonb) do |agent| agent.set name: "activerecord agent" end end ## # Find an instance of your agent agent = Agent.find_by(id: 1) ## # Read the number of tokens used directly from # the jsonb column p agent.context_used ## # Read the overall context window size of the model # directly from the jsonb column p agent.context_available ## # Returns a relation over the agent's messages. # It is scoped to the agent, and it yields one # instance of LLM::ActiveRecord::Message per # message the agent has produced. messages = agent.messages ## # The relation chains like any other messages.where(role: "assistant") .order(position: :desc) .limit(10) ## # Count, too messages.count ``` **Schema** Each row carries a message, flattened into columns: | column | contents | | --- | --- | | `agent_id` | the agent a message belongs to | | `id` | the message id | | `role` | the message role | | `content` | the message content | | `tools` | the tool calls a message carries | | `position` | the position of a message in the conversation | | `data` | the whole message, as the runtime stores it | **Indexes** The queries the view runs are already covered. They expand one agent, found by primary key, so they are index scans. There is nothing to add for `agent.context_used,context_window,messages`. The queries you write on top of it are not. Once a question is asked of every agent, the column is expanded row by row and no index helps the view itself. Index the column for those questions instead: ```sql CREATE INDEX index_agents_on_data ON agents USING gin (data jsonb_path_ops); CREATE INDEX index_agents_on_context_used ON agents (((data ->> 'context_used')::int)); ``` The first serves containment (`@>`) and path queries over the state as a whole. The second serves a scalar key, and the runtime already writes `context_used` and `context_window` at the top level, so "sessions over 80% full" becomes cheap. Both assume `format: :jsonb`. **However:** an agent's whole conversation lives in one value, so every save rewrites it, and a GIN index is maintained with it. Prefer an index on a key or two over the whole column.
Concurrency
Strategies
There are six different concurrency strategies to choose from. A tool can be spawned on its own process, on its own ractor, on its own thread, on a thread that manages an `Async::Reactor`, on its own fiber, or on the same thread as the agent (sequential execution). Each strategy is built around an abstraction that makes them interchangeable - you can switch between processes, threads, ractors, async fibers, or a ruby fiber by changing a single line of code. **All** subclasses of [`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) are supported, so you can launch MCP tools concurrently or in parallel as well. The strategy that you choose depends on the workload and the requirements of your application, as well as what the strategy offers. The `:fork` and `:ractor` strategies provide true parallelism, and the other strategies - especially `:thread`, and `:async` - can be great for IO-bound tools. The following example configures an agent to use all builtin tools (`LLM::Tool.subclasses`) and it spawns each tool in its own process (`concurrency: :fork`). It could be changed to use async instead with `concurrency: :async`, or with threads via `concurrency: :thread`, and so on. The default concurrency strategy is `:sequential`, and it runs tools sequentially on a single thread. ```ruby require "llm" require "llm/tools" llm = LLM.deepseek tools = LLM::Tool.subclasses agent = LLM::Agent.new(llm, tools:, concurrency: :fork) agent.talk "implement the feature described in todo.txt" ```
Cancellation
It is possible to interrupt a running agent who is between requests or tool calls as long as the cancel request is sent from another thread or fiber running in the same process as the agent. A cancel request can be sent with the [`LLM::Agent#interrupt!`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#interrupt!-instance_method) method. An interrupted tool call is made aware of the interrupt and it can both rescue `LLM::Interrupt` and/or implement the `on_interrupt` callback on the tool class. The option to cancel on [r.uby.dev](https://r.uby.dev) is built on top of this feature, and it has a single background process with 24 threads. Each thread can run an agent request that can be interrupted via another thread in the same process. It is solid and reliable but r.uby.dev had to meet this feature halfway, so expect to build your own infrastructure around it. See [cancellation chapter](docs/deepdive/advanced/cancellation.md) to learn more. ```ruby class Search < LLM::Tool name "search" description "Search many files" parameter :pattern, String, "The pattern to search for" required %i[pattern] ## # A raise is delivered here; `on_interrupt` is a notification, and it # runs on every strategy - `:sequential` included. def call(pattern:) search(pattern) rescue LLM::Interrupt ## # A tool can return a value from here, and the turn carries on with # it, or re-raise and the fiber that made the request is raised # into as well. cleanup raise end ## # Told on the thread or fiber the call runs on, before the raise on # `:fork` and `:ractor` and after the rescue above on the other three. def on_interrupt cleanup end private def cleanup # Release a file, a socket, or a lock here. end end llm = LLM.deepseek agent = LLM::Agent.new(llm, tools: [Search], concurrency: :async) queue = Queue.new Thread.new { [sleep(queue.pop), agent.interrupt!] } begin queue.push(3) agent.talk "find every TODO in the repository", stream: $stdout rescue LLM::Interrupt puts puts "!! interrupted !!" end ```
Cancel by record ID
A common deployment setup is to run your agents in a background process that a web frontend can communicate with (usually via a database). The background process would have one thread per agent, and it could run as many agents as it has threads. This is how the [r.uby.dev](https://r.uby.dev) website is configured, and it is the configuration that the [`LLM.interrupt`](https://r.uby.dev/api-docs/llm.rb/LLM.html#interrupt-class_method) method is optimized for: a single process with each agent running in its own thread. The [`LLM.interrupt`](https://r.uby.dev/api-docs/llm.rb/LLM.html#interrupt-class_method) method has access to a process-wide registry that contains every active instance of [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html), and that includes Sequel and ActiveRecord agents, too. An agent enters the registry when it starts a turn, and it exits the registry afterwards. The method returns the agent it interrupted, and otherwise it returns false. There is often a window between when an agent is queued and when it runs, so a poll approach lets you eventually interrupt the agent, or give up trying: ```ruby require "llm" require "llm/tools" llm = LLM.deepseek concurrency = :ractor agent = LLM::Agent.new(llm, concurrency:) Thread.new do LLM.interrupt( id: agent.id, attempts: 50, interval: 0.1 ) end begin stream = $stdout tools = [LLM::Tool::ReadFile] agent.talk("summarize README.md", stream:, tools:) rescue LLM::Interrupt puts "!! interrupted !!" end ```
Hooks
Intercepts
When a model requests a tool call, the runtime represents it as a [`LLM::Function`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html) object that a subclass of [`LLM::Intercept`](https://r.uby.dev/api-docs/llm.rb/LLM/Intercept.html) can see _before_ it is called. An intercept can return a [`LLM::Function::Return`]((https://r.uby.dev/api-docs/llm.rb/LLM/Function.html)) object on behalf of the tool call, and so the tool call is never made. Or it can return any other value to let the tool continue as normal. Maybe best of all, an intercept can modify the arguments given to a [`LLM::Function`](https://r.uby.dev/api-docs/llm.rb/LLM/Function.html) in-place, before the tool call is made. This allows an intercept to modify what the model wanted to send to a function, and then let the function be called as normal. This also works for MCP tools - so you might be able to imagine an intercept that wrapped a comment body at 80 columns, or parsed a markdown post into HTML, before the tool call is made. A simple example: ```ruby require "llm" require "llm/tools" class Sandbox < LLM::Intercept def call(function:) if function.name == "exec" function.return(error: true, type: "permission_error", message: "exec is disabled") end end end llm = LLM.deepseek tools = [LLM::Tool::Exec, LLM::Tool::ReadFile] agent = LLM::Agent.new(llm, tools:, intercept: Sandbox) agent.talk "run 'rm -rf myapp/' right now", stream: $stdout ``` A multi-intercept example.
Each intercept is called in order they're given and the first one that returns an [`LLM::Function::Return`](https://r.uby.dev/api-docs/llm.rb/LLM/Function/Return.html) stops the tool call from being made. This feature lets you combine multiple intercepts into a single intercept that calls out to all of them: ```ruby require "llm" llm = LLM.deepseek mcp = LLM::MCP.stdio(argv: ["bundle", "exec", "slack-mcp"]) intercept = LLM::Intercept.multi(llm, WrapComment, BlockBadLanguage) agent = LLM::Agent.new(llm, tools: [mcp.tools], intercept:) agent.talk "Tell slack channel I'm AFK", stream: $stdout ```
Filters
The [`LLM::MessageFilter`](https://r.uby.dev/api-docs/llm.rb/LLM/MessageFilter.html) class receives a user message or tool return _before_ it is sent back to the model. A filter can modify a message in-place, return a new copy of the message and/or modify a tool return before the model sees it. The most common use-case for a filter might be the anonymization or redaction of personal data but it is the only use-case. A simple example:
```ruby class EmailFilter < LLM::MessageFilter def call(message:) if message.tool_return? ## # 1+ tool returns that includes # an 'email' field that we anonymize # before the model sees it. message.returns.each { it.value[:email].gsub!(regexp, "[EMAIL]") } else ## # A user message that includes # PII that we will anonymize # before the model sees it. message.contents.each { it.gsub!(regexp, "[EMAIL]") } end ## # Don't forget to return the message. # This is important. message end private def regexp /[\w.+-]+@[\w-]+\.[\w.]+/ end end llm = LLM.deepseek agent = LLM::Agent.new(llm, filter: EmailFilter) agent.talk ["hello", "reach me at robert@r.uby.dev"] ``` A multi-filter example.
Multiple filters can be combined into a single filter that run in a chain. The message returned by the previous filter is forwarded to the next filter in the chain, and this continues until we reach the end of the chain. The final message that the model receives could have been modified by multiple different filters: ```ruby require "llm" llm = LLM.deepseek filter = LLM::MessageFilter.multi(llm, RedactEmail, RedactName, RedactLocation) agent = LLM::Agent.new(llm, stream: $stdout, filter:) agent.talk "reach me at robert@r.uby.dev" ```
Compactors
Every model has a context window: the finite number of tokens it can consider in a single request. Generally a compactor will drop or summarize older messages to keep the conversation within that window, and it runs automatically before every turn. By default it is disabled so it is a feature you must opt into. [`LLM::Compactor::Truncate`](https://r.uby.dev/api-docs/llm.rb/LLM/Compactor/Truncate.html) keeps the most recent messages via an integer count or a percentage like `"80%"`. It preserves tool call and return pairs so the conversation never contains an orphaned result. It is also possible to subclass [`LLM::Compactor`](https://r.uby.dev/api-docs/llm.rb/LLM/Compactor.html) to implement your own compactor with its own logic. Streams can observe the process through the [`LLM::Stream#on_compaction`](https://r.uby.dev/api-docs/llm.rb/LLM/Stream.html#on_compaction-instance_method) and [`LLM::Stream#on_compaction_finish`](https://r.uby.dev/api-docs/llm.rb/LLM/Stream.html#on_compaction_finish-instance_method) callbacks. ```ruby llm = LLM.deepseek agent = LLM::Agent.new( llm, compactor: LLM::Compactor::Truncate, compactor_options: {keep: 64} ) agent.talk "Hello" ```
Everything else
A2A
The Agent 2 Agent (A2A) protocol has first-class support in llm.rb. The http and jsonrpc transports work out of the box. A2A skills are translated into subclasses of [`LLM::Tool`](https://r.uby.dev/api-docs/llm.rb/LLM/Tool.html) that can be used with [`LLM::Context`](https://r.uby.dev/api-docs/llm.rb/LLM/Context.html) or [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html). ```ruby require "llm" llm = LLM.deepseek a2a = LLM::A2A.rest(url: "https://remote-agent.example.com") agent = LLM::Agent.new(llm, stream: $stdout, tools: a2a.skills) agent.talk "Run the skill" ```
RAG
Most providers offer an embedding model that can be used for semantic search, or similarity search. An embedding model can generate embeddings that can then be stored in a database that is optimized for storing and querying vectors, such as SQLite's [sqlite-vec](https://github.com/asg017/sqlite-vec) or PostgreSQL's [pg-vector](https://github.com/pgvector/pgvector). llm.rb also includes support for OpenAI's vector store API. It provides a vector database as a HTTP service but we won't cover that here. ```ruby require "llm" llm = LLM.openai body = "llm.rb is Ruby's capable AI runtime." embedding = llm.embed([body]).embeddings.first # Document is your ActiveRecord or Sequel model # with a vector column (e.g. sqlite-vec or pgvector) Document.create!( title: "llm.rb", body:, embedding:, ) ```
Images
A handful of providers can generate images from a text prompt. OpenAI, Google, xAI, and DeepInfra all support it. The API is the same across providers: ```ruby require "llm" llm = LLM.openai res = llm.images.create(prompt: "a dog on a rocket to the moon") IO.copy_stream res.images[0], "rocket.png" ``` ##### DeepSeek DeepSeek does not have a dedicated image model, but the runtime generates SVG vector graphics through its text model. Each generation produces a valid SVG document that can be converted to PNG with tools like `rsvg-convert`. Pass an existing agent to maintain a session across generations: ```ruby require "llm" llm = LLM.deepseek ## # First generation res = llm.images.create(prompt: "a rocket on the moon") IO.copy_stream res.images[0], "rocket.svg" ## # Refine with follow-up prompts (shares context) res = llm.images.create(prompt: "add a dog next to the rocket", agent: res.agent) IO.copy_stream res.images[0], "rocket-with-dog.svg" ```
Providers
Each provider is constructed with a class-level factory method on
LLM, and the resulting instance is passed to
LLM::Context
or
LLM::Agent. The
same API drives every one of them, so switching providers is a one-line
change. One default differs: an OpenAI context speaks the Responses API
(mode: :responses) unless it is built with mode: :completions, while
every other provider defaults to chat completions.
What providers does llm.rb support?
- Anthropic (
LLM.anthropic) - Google (
LLM.google) - OpenAI (
LLM.openai) - DeepSeek (
LLM.deepseek) - DeepInfra (
LLM.deepinfra) - xAI (
LLM.xai) - Z.ai (
LLM.zai) - Moonshot (Kimi) (
LLM.moonshot) - Hanzo (
LLM.hanzo) - OpenRouter (
LLM.openrouter) - Alibaba (Qwen3) (
LLM.alibaba, alsoLLM.aliyun) - Mistral (
LLM.mistral) - AWS Bedrock (
LLM.bedrock) - Ollama (
LLM.ollama) - llama.cpp (
LLM.llamacpp)
Implicit
Cloud providers can infer their API key automatically from a set of common defaults that are defined by the [models.dev](https://models.dev) registry that is also distributed with llm.rb. ```ruby llm = LLM.openai llm = LLM.anthropic llm = LLM.google llm = LLM.deepseek llm = LLM.hanzo llm = LLM.deepinfra llm = LLM.xai llm = LLM.zai llm = LLM.moonshot llm = LLM.openrouter llm = LLM.alibaba # also: LLM.aliyun llm = LLM.mistral llm = LLM.bedrock ```
Explicit
The `key` option can also be providied explicitly, and certain providers (eg ollama, llamacpp) usually do not require an API key at all. ```ruby llm = LLM.openai(key: ENV["OPENAI_API_KEY"]) llm = LLM.anthropic(key: ENV["ANTHROPIC_API_KEY"]) llm = LLM.google(key: ENV["GOOGLE_API_KEY"]) llm = LLM.deepseek(key: ENV["DEEPSEEK_API_KEY"]) llm = LLM.hanzo(key: ENV["HANZO_API_KEY"]) llm = LLM.deepinfra(key: ENV["DEEPINFRA_API_KEY"]) llm = LLM.xai(key: ENV["XAI_API_KEY"]) llm = LLM.zai(key: ENV["ZHIPU_API_KEY"]) llm = LLM.moonshot(key: ENV["MOONSHOT_API_KEY"]) llm = LLM.openrouter(key: ENV["OPENROUTER_API_KEY"]) llm = LLM.alibaba(key: ENV["DASHSCOPE_API_KEY"]) # also: LLM.aliyun llm = LLM.mistral(key: ENV["MISTRAL_API_KEY"]) llm = LLM.bedrock( access_key_id: ENV["AWS_ACCESS_KEY_ID"], secret_access_key: ENV["AWS_SECRET_ACCESS_KEY"], region: ENV["AWS_REGION"] ) ```
Registry
The runtime ships with the [models.dev](https://models.dev) registry. The registry is a moving target and each new release of llm.rb includes an updated copy of it. The registry is loaded per-provider, on-demand and it provides information about the limits, costs, and capabilities of a model. The runtime uses the registry to implement methods like [LLM::Agent#cost](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#cost-instance_method) and the registry can be accessed through the [LLM::Provider#registry](https://r.uby.dev/api-docs/llm.rb/LLM/Registry.html#registry-instance_method) method as well: ```ruby require "llm" llm = LLM.openai registry = llm.registry # => LLM::Provider#registry cheapest = registry.models.sort.first # => LLM::Registry::Model cheapest.id # => "text-embedding-3-small" cheapest.context_window # => 8191 cheapest.structured_output? # => false ```
Transports
The `transport:` option selects which HTTP library a provider uses for network communication. Two backends ship out of the box: `net/http` is always available and the default, and `net/http/persistent` pools connections for many requests to the same host. They share one interface, so switching is a one-word change. ```ruby llm = LLM.deepseek( transport: :net_http_persistent ) ```
Timeouts
Providers accept two timeouts: * `connect_timeout` - opening the connection. Defaults to 5 seconds. * `read_timeout` - waiting for a response on an idle connection. Defaults to 240 seconds (4 minutes). A timeout is retriable, so it is often better to fail early and open a new session than to hold a connection open for a response that never comes. This is especially true for `read_timeout`. Both timeouts can be given as a custom value that is higher or lower than the default. A request that times out is retried automatically. The [`LLM::Agent`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html) class retries a timed out request up to its [`retry_budget`](https://r.uby.dev/api-docs/llm.rb/LLM/Agent.html#retry_budget-class_method) (five by default), so a dropped connection or a slow first token is often something we can recover from. ```ruby llm = LLM.deepseek( connect_timeout: 5, # opening the connection read_timeout: 600 # raise it when a model needs longer ) ```
Headers
Providers can accept a custom set of headers with the [`LLM::Provider#with`](https://r.uby.dev/api-docs/llm.rb/LLM/Provider.html#with-instance_method) method. For example, you could set a custom User-Agent header, or provide headers that carry special meaning to certain providers (eg OpenAI, OpenRouter). ```ruby llm = LLM.openrouter llm = llm.with("HTTP-Referer" => "https://example.com") llm = llm.with("X-OpenRouter-Title" => "Example App") ```
FAQ
Does it work on Windows?
I think so. Our Windows build is green on CI. That's the only signal I have to go by. But I know support is a little more limited compared to other platforms. The `fork` strategy is not available because Windows does not implement it And the `curses` gem that the console requires has to be compiled from source, and this can be a challenge on Windows without a compiler toolchain. By default the console does not work but it might work if you are able to install the `curses` gem. The `ractor` concurrency strategy **does work** on Windows **but requires llm.rb v18+ and Ruby v4.1.0+**. Both are available as preleases. So overall everything mostly works as long as you avoid the parts that don't work. Pull requests that improve Windows support are very welcome.
Where can I see llm.rb in action?
The [r.uby.dev](https://r.uby.dev) website.
What about local LLM support?
Try: * `LLM.ollama(...)` * `LLM.llamacpp(...)`
I have a limited budget. What should I do?
If you're on a budget, [DeepSeek](https://platform.deepseek.com) is hard to beat and it works really well with llm.rb.
Who maintains llm.rb?
The llm.rb project was started more than three years ago by [@altruby](https://github.com/altruby) and [@antaz](https://github.com/altruby). The primary maintainer is [@altruby](https://github.com/altruby). There have been multiple other contributors over the last three years and new contributors are always welcome.
How well tested is llm.rb?
I use it all the time. Especially the console. It also powers the [r.uby.dev](https://r.uby.dev) website. I'm aware of at least one production Rails deployment at a large-ish company.
License
|
This software is released under the terms of the MIT license. |
|