Welcome

Welcome to the canonical llm.rb homepage.

llm.rb is an advanced runtime for building capable AI applications on CRuby. It has zero runtime dependencies by default, and a single coherent API that spans 11+ providers. Streaming, tools, guards, compaction, the REPL, and the database integrations all build on the same three concepts: providers, contexts, and agents.

Once you learn the fundamentals, everything else falls into place naturally. Some features, such as ActiveRecord support, require optional dependencies that are opt-in.

When you want to learn more than what this page covers, checkout the deepdive.

Install

Source code and releases are available from github.com/r-uby-dev/llm.

$ gem install llm.rb

Quick start

require "llm"

llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, stream: $stdout)
agent.talk "Hello world"

class SystemAdmin < LLM::Agent
  set name: "sysadmin",
      description: "system administration agent",
      model: "deepseek-v4-pro",
      tools: [Shell]
end

llm = LLM.deepseek(key: ENV["KEY"])
agent = SystemAdmin.new(llm)
agent.talk "Run 'date'"

require "llm"

llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, path: "session.json")
agent.talk "remember my name is robert"

# Next time, the conversation is restored automatically:
agent = LLM::Agent.new(llm, path: "session.json")
agent.talk "what's my name?"

class PolicyGuard < LLM::Guard
  def call(function:)
    if function.name == "shell"
      function.return(error: true, type: "policy_error",
                      message: "shell is disabled")
    end
  end
end

agent = LLM::Agent.new(llm, guard: PolicyGuard)

require "llm"

llm = LLM.deepseek(key: ENV["KEY"])
ctx = LLM::Context.new(llm, stream: $stdout)
ctx.talk "Hello world"

class ReadFile < LLM::Tool
  name "read-file"
  description "Read a file"
  parameter :path, String, "The filename or path"
  required %i[path]

  def call(path:)
    {contents: File.read(path)}
  end
end

class MathTool < LLM::Tool
  set name: "math",
      description: "Performs arithmetic",
      parameters: [
        [:x, Integer, "first number" , {required: true}],
        [:y, Integer, "second number", {default: 0}]
      ]

  def call(x:, y: 0)
    {result: x + y}
  end
end

LLM::Stream

Streams can be simple IO objects or subclasses of LLM::Stream with structured callbacks for content, reasoning, tool calls, tool returns, and compaction. Streams can also observe message transformers, which rewrite outgoing messages before they reach the provider (see the deepdive).

class MyStream < LLM::Stream
  def on_content(content)
    print content
  end

  def on_reasoning_content(content)
    warn content
  end
end

llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, stream: MyStream.new)
agent.talk "Explain Ruby fibers."

LLM::Schema

LLM::Schema subclasses produce typed, structured output from any model call. Pass a schema to LLM::Context#talk, LLM::Agent#talk, or LLM::Provider#complete to receive validated JSON instead of free text. Schemas work alongside tools and streams.

LLM::Schema can define objects, arrays, enums, nested schemas, and more. It is also used internally by LLM::Tool for parameter definitions, so you already benefit from it when you declare tool parameters.

The LLM::DeepSeek provider includes runtime-level optimisations such as structured output support (despite no official structured outputs API) and SVG image generation. This example uses LLM::Schema with DeepSeek:

class Weather < LLM::Schema
  property :city, String, "The city name"
  property :temperature, Float, "Current temperature"
  property :conditions, String, "Weather conditions"
  required %i[city temperature conditions]
end

llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, schema: Weather)
res = agent.talk "Weather in Paris?"
res.content!  # => {city: "Paris", temperature: 15.0, conditions: "Cloudy"}

LLM::REPL

The LLM::Agent#repl method drops you into a curses-based TUI for talking to an agent interactively. It renders markdown directly in the terminal and shows a live status line with context usage, running cost, and the current tool call. A second thread keeps the UI responsive while the model works. Think of it as binding.pry but for agents.

Set path: on the agent for automatic persistence across REPL sessions. The tools: option attaches extra tools for the duration of the session. Recall previous turns with Ctrl+P and Ctrl+N. For the full reference, see the REPL section in the deepdive.

require "llm"
require "llm/tools"

llm = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, name: "my-agent", path: "agent.json")
agent.repl(tools: LLM::Tool.subclasses)

CLI

The llm.rb executable is available on your PATH after installation. It starts a REPL session from any directory:

llm.rb                     # auto-detect from $DEEPSEEK_API_KEY
llm.rb -p openai           # use OpenAI explicitly
llm.rb -t                  # temporary session, no persistence

The CLI auto-detects your provider from standard environment variables (DEEPSEEK_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.). Persistent sessions are stored under ~/.llm.rb/ and restored automatically on your next visit.

LLM::MCP

The Model Context Protocol (MCP) has first-class support in llm.rb. The stdio and http transports work out of the box. MCP tools are translated into subclasses of LLM::Tool that can be used with LLM::Context or LLM::Agent.

require "llm"

llm   = LLM.deepseek(key: ENV["KEY"])
mcp   = LLM::MCP.stdio(argv: ["ruby", "server.rb"])
agent = LLM::Agent.new(llm, stream: $stdout, tools: mcp.tools)
agent.talk "Run the tool"

Persistent connections

Set persistent: true on HTTP transports to reuse connections across requests. This uses Net::HTTP::Persistent under the hood and avoids opening a new TCP connection for every request:

mcp = LLM::MCP.http(
  url: "https://api.githubcopilot.com/mcp/",
  headers: {"Authorization" => "Bearer #{ENV.fetch('GITHUB_PAT')}"},
  persistent: true
)

LLM::A2A

The Agent 2 Agent (A2A) protocol has first-class support in llm.rb. The http and jsonrpc transports work out of the box. A2A skills are translated into subclasses of LLM::Tool that can be used with LLM::Context or LLM::Agent.

require "llm"

llm   = LLM.deepseek(key: ENV["KEY"])
a2a   = LLM::A2A.rest(url: "https://remote-agent.example.com")
agent = LLM::Agent.new(llm, stream: $stdout, tools: a2a.skills)
agent.talk "Run the skill"

Persistent connections

Set persistent: true on HTTP transports to reuse connections across requests. This uses Net::HTTP::Persistent under the hood and avoids opening a new TCP connection for every request:

a2a = LLM::A2A.rest(url: "https://agent.example.com", persistent: true)
a2a = LLM::A2A.jsonrpc(url: "https://agent.example.com", persistent: true)

LLM::Skill

A skill turns a markdown file into a callable tool. When the model calls it, the runtime spawns a subagent with the skill's instructions as its system prompt and the skill's own tool set. The subagent runs one turn and returns the result, then is discarded. Each call is fresh and stateless. For a deeper explanation see the deepdive.

SKILL.md

---
name: summary
description: Reads recent git history and writes a summary
tools: all
---

Collect the recent git log, analyze each commit,
and write a summary to summary.txt.

agent.rb

require "llm"

llm   = LLM.deepseek(key: ENV["KEY"])
agent = LLM::Agent.new(llm, skills: ["./skills/summary"])
agent.talk "Summarize the last week of work"

RAG

Most providers offer an embedding model that can be used for semantic search, or similarity search. An embedding model can generate embeddings that can then be stored in a database that is optimized for storing and querying vectors, such as SQLite's sqlite-vec or PostgreSQL's pg-vector.

llm.rb also includes support for OpenAI's vector store API. It provides a vector database as a HTTP service but we won't cover that here. For a deeper explanation see the deepdive.

require "llm"

llm  = LLM.openai(key: ENV["KEY"])
body = "llm.rb is Ruby's capable AI runtime."
embedding = llm.embed([body]).embeddings.first

# Document is your ActiveRecord or Sequel model
# with a vector column (e.g. sqlite-vec or pgvector)
Document.create!(
  title: "llm.rb",
  body:,
  embedding:,
)

Concurrency

The runtime supports six different concurrency strategies that have different attributes. The choice between all of them often depends on the requirements of your application.

IO-bound tools are a good fit for the :async, :thread, and :fiber strategies while true parallelism can be achieved with the :fork and :ractor strategies. The :sequential strategy runs tools one at a time and is the default. The :fork strategy also provides a separate process that offers isolation from its parent.

You can learn more about the llm.rb concurrency model in the deepdive.

require "llm"

llm   = LLM.deepseek(key: ENV["KEY"])
tools = [FetchNews, FetchStocks, FetchFeeds]
agent = LLM::Agent.new(llm, tools:, concurrency: :fork)
agent.talk "Run the tools in parallel"

ORM

Because both LLM::Context and LLM::Agent can be serialized to JSON and stored in a simple string, both ActiveRecord and Sequel support can be implemented within a single column on a single row.

The runtime includes first-class support for both ActiveRecord and Sequel, and for both Rack-based applications and Rails-based applications. On databases where it is supported, such as PostgreSQL, the column can be optimized by using the jsonb type.

require "active_record"
require "llm"
require "llm/active_record"

class Agent < ApplicationRecord
  acts_as_agent
  set name: "my-agent",
      instructions: "solve the user's query",
      model: "deepseek-v4-pro",
      tools: [Research, FinalizeResearch, ActOnResearch]

  private

  # By convention, this method defines the provider for a model.
  # If necessary, it can be renamed with: provider: :your_method.
  def set_provider
    LLM.deepseek(key: ENV["KEY"])
  end

  # By convention, this method returns the context options given
  # to LLM::Context or LLM::Agent.
  def set_context
    {}
  end
end

agent = Agent.create!
agent.talk "perform research"

Images

A handful of providers can generate images from a text prompt. OpenAI, Google, xAI, and DeepInfra all support it. The API is the same across providers:

require "llm"

llm = LLM.openai(key: ENV["KEY"])
res = llm.images.create(prompt: "a dog on a rocket to the moon")
IO.copy_stream res.images[0], "rocket.png"

DeepSeek

DeepSeek does not have a dedicated image model, but the runtime generates SVG vector graphics through its text model. Each generation produces a valid SVG document that can be converted to PNG with tools like rsvg-convert. Pass an existing agent to maintain a session across generations:

require "llm"
llm = LLM.deepseek(key: ENV["KEY"])

##
# First generation
res = llm.images.create(prompt: "a rocket on the moon")
IO.copy_stream res.images[0], "rocket.svg"

##
# Refine with follow-up prompts (shares context)
res = llm.images.create(prompt: "add a dog next to the rocket",
                        agent: res.agent)
IO.copy_stream res.images[0], "rocket-with-dog.svg"

Learn more

If you like what you've read so far, check out the deepdive to learn more. Unfortunately it wasn't possible to cover every feature without the README becoming a small book. The r.uby.dev homepage also includes more learning material and resources.

Developers

The llm.rb project is quite large and maintained primarily by one person. It would be near impossible for me to maintain both the codebase and its documentation, especially the deepdive, so I have written agents that maintain the documentation assets and that allows me to put more focus on the code.

The following agents are available for those tasks, and all of them use the most cost effective option: DeepSeek. Feel free to use them in your own fork.

##
# Maintains the deepdive and API docs
rake agents:scribe:yardoc
rake agents:scribe:coverage
rake agents:scribe:regressions
rake agents:scribe:style

##
# Maintains the release
rake agents:dexter:changelog
rake agents:dexter:release

##
# Maintains mruby-llm backports
rake agents:mruby:research
rake agents:mruby:implement

##
# Refresh the data/ registry
rake models.dev:download

License

This software is released under the terms of the MIT license.
See LICENSE for details.