Changelog
r.uby.dev project

This file covers the v19 series.
Older releases:

What's next

No unreleased changes yet. Check back after the next release.

v19.0.1

Changes since v19.0.0.

This is a patch release. It fixes the :ractor strategy, which called #new on the LLM::Tool instance given to LLM::Function#task, and it makes llm.rb eval resolve a relative path from the working directory. It also defaults the OpenAI images model: to gpt-image-2.6-flare, ahead of the gpt-image-1-mini retirement.

LLM::OpenAI

  • openai: images default to gpt-image-2.6-flare
    LLM::OpenAI::Images#create and LLM::OpenAI::Images#edit default the model: parameter to gpt-image-2.6-flare now, where they defaulted it to gpt-image-1-mini. OpenAI removes gpt-image-1-mini from its API on December 1st, 2026. A model: that is given is sent unchanged.

LLM::Function

  • function: the :ractor strategy runs a LLM::Tool instance
    LLM::Function#task runs the LLM::Tool instance it is given under the :ractor strategy, where it called #new on that instance and raised NoMethodError. An instance that takes an argument in #initialize runs with the state it was given.

Console

  • console: llm.rb eval resolves a relative path from the working directory
    llm.rb eval joins a relative path with Dir.getwd now, where it joined it with the root of the llm.rb checkout, so the path is read relative to the directory the command runs in.

v19.0.0

Changes since v18.0.0.

This is a smaller release than v18. It adds the LLM::Eval framework and the llm.rb eval command that runs it, LLM::Command, LLM::Paint, and a few tools, and it carries breaking changes to LLM::Tracer::PrettyLogger, LLM::Tool::Exec#call and LLM::Tool::Utils::Process#spawn.

Breaking

Migration

Old New
LLM::Tool::Exec#call(arguments: ["echo", "hi"]) LLM::Tool::Exec#call(command: "echo", arguments: ["hi"])
LLM::Tool::Utils::Process#spawn(name:) LLM::Tool::Utils::Process#spawn(command:)
LLM::Tracer::PrettyLogger LLM::Tracer::EnhancedLogger
LLM::Tracer.pretty_logger, and LLM::Tracer.pretty LLM::Tracer.enhanced_logger, and LLM::Tracer.enhanced
LLM::Command, an alias for LLM::Console::Command LLM::Command spawns a command, and LLM::Console::Command is spelled out
LLM::Tool::Utils::Process::Command, and the test-cmd.rb gem LLM::Command, which ships with llm.rb

LLM

  • core: LLM.json.dump takes a pretty: option
    LLM.json.dump dumps an object with JSON.pretty_generate when it is given pretty: true, so the string it returns is indented and read by a person. Without the option it dumps with JSON.dump, the way it did before. The LLM::JSONAdapter::Oj adapter takes no such option.

LLM::Eval

  • eval: add LLM::Eval
    LLM::Eval scores a model against a task the way a unit test scores code. An eval is a directory that holds an eval.rb file and one or more step files, and the file declares a name and a description with set, the prompts the agent is given with step, the state it runs against with setup and teardown, the agent, tracer and intercept it runs with through configure, and the score with score and penalize!. LLM::Eval#call sends each step to the agent, runs the score block, subtracts each penalty, and returns {name:, score:}. A score is a float between 0.0 and 1.0.

  • eval: add LLM::Eval.from
    LLM::Eval.from reads an eval.rb file, evaluates it, and binds the eval it returns to a provider, a model, and a sample number, so the same eval can be run against each of them.

  • eval: aggregate samples into a run report
    When an eval is given a tracer, every sample it runs is added to runs/<eval>/<provider>/<model>/run.json, which holds each sample beside the average score of them all, and run.txt beside it holds the same as a table. A sample that runs a second time replaces the entry it wrote before, so a sample number can be repeated without counting twice.

  • eval: add LLM::Eval::Tracer::Console
    LLM::Eval::Tracer::Console draws an eval as it runs: a line for each request, each tool call, each error, and each interrupt, and a summary on exit with the number of turns, requests, tool calls and tokens, the slowest call, the score and each penalty. The summary is written to runs/<eval>/<provider>/<model>/<sample>/sample.txt without the escape sequences it draws with.

  • eval: add LLM::Eval::Tracer::JSON
    LLM::Eval::Tracer::JSON records an eval as JSON: each chat request with its model, duration, tokens, reply and reasoning, each tool call with its arguments and duration, every error, and the score and its penalties. The report is written to runs/<eval>/<provider>/<model>/<sample>/sample.json on exit, as indented JSON.

LLM::Command

  • command: add LLM::Command
    LLM::Command spawns a command without a shell and collects its streams. #env, #stdin and #limit configure it, #spawn starts it, #stdout and #stderr return what it wrote, and #status, #exit_status and #success? report how it ended. A command that is not on the system is reported by #not_found?, and #kill! sends a running command SIGKILL. The class was imported from the test-cmd.rb gem, which llm.rb depends on no longer.

LLM::Tool

  • tools: add a git-ls-files tool
    LLM::Tool::Git::LsFiles (git-ls-files) lists the files git tracks, the index beside the changes that git-status reports. Like the other git tools it takes an optional path:, which narrows the list, and an optional timeout:.

  • tools: add a git-pull tool
    LLM::Tool::Git::Pull (git-pull) pulls the changes of a remote branch into the current branch. It takes an optional remote:, an optional branch:, which requires a remote, and an optional timeout:.

  • tools: add LLM::Tool::DirTree
    LLM::Tool::DirTree (dir-tree) returns the contents of a directory as a nested hash: a directory is a hash of its entries and a file is nil. It takes the path: to walk and an optional depth:, which is five levels by default, and it raises when the path does not exist.

  • tools: the git tools take a cwd:
    Each tool that LLM::Tool::Git provides takes an optional cwd:, which names the repository to run in. The command is built as git -C <cwd> rather than run from the process's working directory, so a tool call can reach a repository outside the one the process was started in, and two tool calls that run at the same time can reach different repositories. It is Dir.getwd by default, and the superclass declares the parameter, so every git tool has it.

  • tools: LLM::Tool::Exec#call takes a command and its arguments apart
    LLM::Tool::Exec#call takes the command to run as command: and its arguments as arguments:, where arguments: alone took the command and its arguments as one array. The parameter the model is given is split the same way, so a model passes a command and, separately, an arguments array, and command is the required one.

  • tools: LLM::Tool::Utils::Process#spawn takes command:
    LLM::Tool::Utils::Process#spawn takes the command to run as command:, where it took it as name:, so the keyword matches the one LLM::Tool::Exec#call takes. A tool of your own that spawns a command passes command: now.

LLM::Function

  • function: LLM::Function#task(:ractor) accepts a tool instance
    LLM::Function#task accepts an instance of LLM::Tool when it is given the :ractor strategy, where it took a class alone and raised LLM::RactorError for an instance. A runner that is neither a class nor an instance of LLM::Tool still raises LLM::RactorError.

  • function: an unknown tool returns its error in band under :ractor
    A tool call for a tool the runtime does not know is turned into an anonymous subclass of LLM::Tool whose #call raises LLM::NoSuchToolError, and the runtime returns that error to the model in band, the way the other strategies do. The tool is a class rather than a proc, so the :ractor strategy runs it, where it raised LLM::RactorError before.

  • function: the LLM::RactorError names the function and its runner
    The LLM::RactorError that LLM::Function#task raises for a runner the :ractor strategy cannot use names the function and inspects the runner, where its message was one generic line.

Console

  • console: add llm.rb eval
    llm.rb eval -p <provider> [-m <model>] [-n <samples>] <path...> runs the evals it is given, and it hands off to libexec/llm-eval. The provider is built from -p, which is required, the model is taken from -m or from the provider's default_model, and each eval is run -n times, where -n is 1 by default. A relative path is resolved against the root of the llm.rb checkout.

  • console: models.dev:download is update-data
    The command that refreshes the data/ directory from models.dev is llm.rb update-data now, and llm.rb models.dev:download is still accepted.

LLM::Paint

  • paint: add LLM::Paint
    LLM::Paint wraps a string in the escape sequences a terminal understands, eg LLM::Paint["hello", :blue, :bold], and returns the string unchanged when color is disabled. It honors $NO_COLOR, it is a small stand-in for the paint gem, and it is what the eval console tracer uses.

  • paint: add LLM::Paint::IO
    LLM::Paint::IO wraps an IO so that the text written to it is wrapped in the escape sequences a terminal understands, and written unchanged when color is disabled. LLM::Paint::IO#write writes text with the colors and effects it is given, LLM::Paint::IO#puts writes it with a newline, LLM::Paint::IO#color? returns true when the wrapper writes color, LLM::Paint::IO#color= sets it, and LLM::Paint::IO#io returns the IO that is wrapped.

  • paint: LLM::Paint.detect takes an IO
    LLM::Paint.detect takes the IO to inspect as an argument now, where it inspected $stdout alone. It still returns false when $NO_COLOR is set, and it returns false for an IO that does not respond to tty?.

LLM::Tracer

  • tracer: rename LLM::Tracer::PrettyLogger as LLM::Tracer::EnhancedLogger
    LLM::Tracer::EnhancedLogger is the tracer that LLM::Tracer::PrettyLogger was. LLM::Tracer.enhanced_logger builds it, LLM::Tracer.enhanced is its alias, and both replace LLM::Tracer.pretty_logger and LLM::Tracer.pretty.

  • tracer: the enhanced logger draws a readable log
    A chat request, a tool call and an interrupt are each a line of their own, with a marker that names the event, a duration for a request and a tool call, and the tokens a request used. The timestamp, and the comma separated fields it prefixed, are gone.

  • tracer: the enhanced logger writes a summary on exit
    LLM::Tracer::EnhancedLogger#on_exit writes the provider, the model, the number of requests, the tool calls with a breakdown by name, the tokens returned in and out, the cost the registry prices them at, the slowest tool call, the errors, and the elapsed time. It is written only when the tracer saw a request or a tool call, and it closes the file the tracer opened.

  • tracer: the enhanced logger writes color
    LLM::Tracer::EnhancedLogger writes through an LLM::Paint::IO, so its log is colored when the IO it writes to is a terminal, and plain when it is not, when $NO_COLOR is set, or when LLM::Paint.enabled is false.