Class: LLM::Provider Abstract

Inherits:
Object
  • Object
show all
Includes:
Transport::Execution
Defined in:
lib/llm/provider.rb

Overview

This class is abstract.

The Provider class is the abstract base for LLM service integrations. Most users interact with providers through Agent or Context rather than calling #complete directly.

Direct Known Subclasses

Anthropic, Bedrock, Google, Ollama, OpenAI

Instance Method Summary collapse

Constructor Details

#initialize(key:, host:, port: 443, timeout: 240, read_timeout: nil, connect_timeout: 5, ssl: true, base_path: "", persistent: false, transport: nil) ⇒ Provider

Returns a new instance of Provider.

Parameters:

  • key (String, nil) —

    The secret key for authentication

  • host (String) —

    The host address of the LLM provider

  • port (Integer) (defaults to: 443) —

    The port number

  • timeout (Integer) (defaults to: 240) —

    The number of seconds to wait for a response. Also serves as the default read timeout.

  • read_timeout (Integer, nil) (defaults to: nil) —

    The number of seconds to wait for a response. Defaults to timeout.

  • connect_timeout (Integer) (defaults to: 5) —

    The number of seconds to wait for a TCP connection to open.

  • ssl (Boolean) (defaults to: true) —

    Whether to use SSL for the connection

  • base_path (String) (defaults to: "") —

    Optional base path prefix for HTTP API routes.

  • persistent (Boolean) (defaults to: false) —

    Whether to use a persistent connection. Requires the net-http-persistent gem.

  • transport (LLM::Transport, Class, nil) (defaults to: nil) —

    Optional override with any Transport instance or subclass.



35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
 
# File 'lib/llm/provider.rb', line 35

def initialize(key:, host:, port: 443, timeout: 240, read_timeout: nil, connect_timeout: 5, ssl: true, base_path: "", persistent: false, transport: nil)
  @key = key
  @host = host
  @port = port
  @read_timeout = read_timeout || timeout
  @timeout = @read_timeout
  @connect_timeout = connect_timeout
  @ssl = ssl
  @base_path = LLM::Utils.normalize_base_path(base_path)
  @base_uri = URI("#{ssl ? "https" : "http"}://#{host}:#{port}/")
  @headers = {"User-Agent" => "llm.rb v#{LLM::VERSION}"}
  @monitor = Monitor.new
  @transport = LLM::Transport::Utils.resolve_transport(
    host:,
    port:,
    timeout: @read_timeout,
    connect_timeout: @connect_timeout,
    ssl:,
    transport:,
    persistent:
  )
end

Instance Method Details

#retry_errors ⇒ Array<Exception>

Returns an array of exceptions that a request could raise and that will be retried up to #retry_budget times.

Returns:

  • (Array<Exception>) 


501
502
503
504
505
506
507
508
509
510
 
# File 'lib/llm/provider.rb', line 501

def retry_errors
  [
    LLM::InsufficientQuotaError,
    LLM::RateLimitError,
    SystemCallError,
    Net::ReadTimeout,
    Net::WriteTimeout,
    Net::OpenTimeout
  ]
end

#build_messages(prompt, params, role, key: :messages) ⇒ Array<LLM::Message>

Builds the outgoing message array for a turn. Normalizes the prompt into one or more Message objects and prepends the existing history.

The method is idempotent. If the prompt is already an Message or an array of Messages (ie it was built by a previous call and possibly transformed), it is returned as-is without rebuilding.

Parameters:

  • prompt (String, Array, LLM::Message, LLM::Prompt)
  • params (Hash) —

    Turn params. The history is taken from params[:messages].

  • role (Symbol) —

    The role to assign to a raw prompt

  • key (Symbol) (defaults to: :messages) —

    The params key that holds the history (:messages for chat completions, :input for the responses API).

Returns:



77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
 
# File 'lib/llm/provider.rb', line 77

def build_messages(prompt, params, role, key: :messages)
  case prompt
  when LLM::Message
    [prompt]
  when Array
    if prompt.all? { LLM::Message === _1 }
      prompt
    else
      [*(params.delete(key) || []), LLM::Message.new(role, prompt)]
    end
  when LLM::Prompt
    [*(params.delete(key) || []), *prompt.to_a]
  else
    [*(params.delete(key) || []), LLM::Message.new(role, prompt)]
  end
end

#inspect ⇒ String

Note:

The secret key is redacted in inspect for security reasons

Returns an inspection of the provider object

Returns:

  • (String) 


98
99
100
 
# File 'lib/llm/provider.rb', line 98

def inspect
  "#<#{LLM::Utils.object_id(self)} @key=[REDACTED] @transport=#{transport.inspect} @tracer=#{tracer.inspect}>"
end

#name ⇒ Symbol

Returns the provider's name

Returns:

  • (Symbol) —

    Returns the provider's name

Raises:

  • (NotImplementedError) —

    When the method is not implemented by a subclass



107
108
109
 
# File 'lib/llm/provider.rb', line 107

def name
  raise NotImplementedError
end

#host ⇒ String

Returns:

  • (String) 


113
114
115
 
# File 'lib/llm/provider.rb', line 113

def host
  @host
end

#port ⇒ Integer

Returns:

  • (Integer) 


119
120
121
 
# File 'lib/llm/provider.rb', line 119

def port
  @port
end

#registry ⇒ LLM::Registry

Returns the provider's model registry.

Returns:



126
127
128
 
# File 'lib/llm/provider.rb', line 126

def registry
  LLM.registry_for(self)
end

#embed(input, model: nil, **params) ⇒ LLM::Response

Provides an embedding

Parameters:

  • input (String, Array<String>) —

    The input to embed

  • model (String) (defaults to: nil) —

    The embedding model to use

  • params (Hash) —

    Other embedding parameters

Returns:

Raises:

  • (NotImplementedError) —

    When the method is not implemented by a subclass



141
142
143
 
# File 'lib/llm/provider.rb', line 141

def embed(input, model: nil, **params)
  raise NotImplementedError
end

#ocr ⇒ LLM::Response

Note:

This feature is not implemented by all providers, and it will raise NotImplementedError for providers that do not support it.

Returns:

Raises:

  • (NotImplementedError) 


151
152
153
 
# File 'lib/llm/provider.rb', line 151

def ocr(...)
  raise NotImplementedError
end

#complete(prompt, params = {}) ⇒ LLM::Response

Provides an interface to the chat completions API. Most users should use Context#talk or Agent#talk instead.

Examples:

llm = LLM.openai(key: ENV["KEY"])
messages = [{role: "system", content: "Your task is to answer all of my questions"}]
res = llm.complete("5 + 2 ?", messages:)
print "[#{res.messages[0].role}]", res.messages[0].content, "\n"

Parameters:

  • prompt (String) —

    The input prompt to be completed

  • params (Hash) (defaults to: {}) —

    The parameters to maintain throughout the conversation. Any parameter the provider supports can be included and not only those listed here.

Options Hash (params):

  • :role (Symbol) —

    Defaults to the provider's default role

  • :model (String) —

    Defaults to the provider's default model

  • :schema (#to_json, nil) —

    Defaults to nil

  • :tools (Array<LLM::Function>, nil) —

    Defaults to nil

Returns:

Raises:

  • (NotImplementedError) —

    When the method is not implemented by a subclass



177
178
179
 
# File 'lib/llm/provider.rb', line 177

def complete(prompt, params = {})
  raise NotImplementedError
end

#chat(prompt, params = {}) ⇒ LLM::Context

Starts a new chat powered by the chat completions API

Parameters:

  • prompt (String) —

    The input prompt to be completed

  • params (Hash) (defaults to: {}) —

    The parameters to maintain throughout the conversation. Any parameter the provider supports can be included and not only those listed here.

Returns:



186
187
188
189
 
# File 'lib/llm/provider.rb', line 186

def chat(prompt, params = {})
  role = params.delete(:role)
  LLM::Context.new(self, params).talk(prompt, role:)
end

#respond(prompt, params = {}) ⇒ LLM::Context

Starts a new chat powered by the responses API

Parameters:

  • prompt (String) —

    The input prompt to be completed

  • params (Hash) (defaults to: {}) —

    The parameters to maintain throughout the conversation. Any parameter the provider supports can be included and not only those listed here.

Returns:

Raises:

  • (NotImplementedError) —

    When the method is not implemented by a subclass



197
198
199
200
 
# File 'lib/llm/provider.rb', line 197

def respond(prompt, params = {})
  role = params.delete(:role)
  LLM::Context.new(self, params).respond(prompt, role:)
end

#responses ⇒ LLM::OpenAI::Responses

Note:

Compared to the chat completions API, the responses API can require less bandwidth on each turn, maintain state server-side, and produce faster responses.

Returns:

Raises:

  • (NotImplementedError) 


209
210
211
 
# File 'lib/llm/provider.rb', line 209

def responses
  raise NotImplementedError
end

#images ⇒ LLM::OpenAI::Images, LLM::Google::Images

Returns an interface to the images API

Returns:

Raises:

  • (NotImplementedError) 


216
217
218
 
# File 'lib/llm/provider.rb', line 216

def images
  raise NotImplementedError
end

#audio ⇒ LLM::OpenAI::Audio

Returns an interface to the audio API

Returns:

Raises:

  • (NotImplementedError) 


223
224
225
 
# File 'lib/llm/provider.rb', line 223

def audio
  raise NotImplementedError
end

#files ⇒ LLM::OpenAI::Files

Returns an interface to the files API

Returns:

Raises:

  • (NotImplementedError) 


230
231
232
 
# File 'lib/llm/provider.rb', line 230

def files
  raise NotImplementedError
end

#models ⇒ LLM::OpenAI::Models

Returns an interface to the models API

Returns:

Raises:

  • (NotImplementedError) 


237
238
239
 
# File 'lib/llm/provider.rb', line 237

def models
  raise NotImplementedError
end

#moderations ⇒ LLM::OpenAI::Moderations

Returns an interface to the moderations API

Returns:

Raises:

  • (NotImplementedError) 


244
245
246
 
# File 'lib/llm/provider.rb', line 244

def moderations
  raise NotImplementedError
end

#vector_stores ⇒ LLM::OpenAI::VectorStore

Returns an interface to the vector stores API

Returns:

  • (LLM::OpenAI::VectorStore) —

    Returns an interface to the vector stores API

Raises:

  • (NotImplementedError) 


251
252
253
 
# File 'lib/llm/provider.rb', line 251

def vector_stores
  raise NotImplementedError
end

#default_model ⇒ String

Returns the default model for chat completions

Returns:

  • (String) —

    Returns the default model for chat completions

Raises:

  • (NotImplementedError) 


258
259
260
 
# File 'lib/llm/provider.rb', line 258

def default_model
  raise NotImplementedError
end

#retry_budget ⇒ Integer

Returns the number of times a rate-limited request is retried before the error is raised.

Each retry sleeps a growing interval, so a budget that is exhausted surfaces the rate-limit error instead of blocking indefinitely. A provider can return a different budget when its API recovers from rate limits more slowly.

Returns:

  • (Integer)

See Also:



274
275
276
 
# File 'lib/llm/provider.rb', line 274

def retry_budget
  5
end

#schema ⇒ LLM::Schema

Returns an object that can generate a JSON schema

Returns:



281
282
283
 
# File 'lib/llm/provider.rb', line 281

def schema
  LLM::Schema.new
end

#with(**headers) { ... } ⇒ LLM::Provider

Note:

For backwards compatibility, headers can be provided via the headers: keyword argument, or provided directly as a Hash without the headers: key namespace.

Add one or more headers to all requests, or scope them to a block.

Without a block the headers are merged into the provider's defaults permanently. With a block the headers apply only to the current fiber, for the duration of the block, then the previous headers are restored. This is useful for a header that varies per context, such as OpenRouter's x-session-id.

Examples:

Permanent

llm = LLM.openai(key: ENV["KEY"])
llm.with("OpenAI-Organization" => ENV["ORG"])
llm.with("OpenAI-Project" => ENV["PROJECT"])

Scoped

llm.with("x-session-id" => ctx.id) do
  llm.complete("hello")
end

Parameters:

  • headers (Hash<String,String>) —

    One or more headers

Yields:

Returns:

  • (LLM::Provider) —

    Returns self without a block, the block's value with one



311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
 
# File 'lib/llm/provider.rb', line 311

def with(**headers, &block)
  headers = headers.merge(headers.delete(:headers) || {})
  if block
    wm = weakmaps.header
    previous = wm[self]
    wm[self] = (previous || {}).merge(headers)
    block.call
  else
    lock { tap { @headers.merge!(headers) } }
  end
ensure
  if block
    if previous.nil?
      wm.respond_to?(:delete) ? wm.delete(self) : wm[self] = nil
    else
      wm[self] = previous
    end
  end
end

#server_tools ⇒ String => LLM::ServerTool

Note:

This method might be outdated, and the LLM::Provider#server_tool method can be used if a tool is not found here.

Returns all known tools provided by a provider.

Returns:



337
338
339
 
# File 'lib/llm/provider.rb', line 337

def server_tools
  {}
end

#server_tool(name, options = {}) ⇒ LLM::ServerTool

Note:

OpenAI, Anthropic, and Gemini provide platform-tools for things like web search, and more.

Returns a tool provided by a provider.

Examples:

llm   = LLM.openai(key: ENV["KEY"])
tools = [llm.server_tool(:web_search)]
res   = llm.responses.create("Summarize today's news", tools:)
print res.output_text, "\n"

Parameters:

  • name (String, Symbol) —

    The name of the tool

  • options (Hash) (defaults to: {}) —

    Configuration options for the tool

Returns:



354
355
356
 
# File 'lib/llm/provider.rb', line 354

def server_tool(name, options = {})
  LLM::ServerTool.new(name, options, self)
end

#web_search(query:) ⇒ LLM::Response

Provides a web search capability

Parameters:

  • query (String) —

    The search query

Returns:

Raises:

  • (NotImplementedError) —

    When the method is not implemented by a subclass



364
365
366
 
# File 'lib/llm/provider.rb', line 364

def web_search(query:)
  raise NotImplementedError
end

#assistant_role ⇒ Symbol

Returns the role of the assistant in the conversation.

Returns:

  • (Symbol) —

    Returns the role of the assistant in the conversation.



371
372
373
 
# File 'lib/llm/provider.rb', line 371

def assistant_role
  :assistant
end

#user_role ⇒ Symbol

Returns:

  • (Symbol) 


377
378
379
 
# File 'lib/llm/provider.rb', line 377

def user_role
  :user
end

#system_role ⇒ Symbol

Returns:

  • (Symbol) 


383
384
385
 
# File 'lib/llm/provider.rb', line 383

def system_role
  :system
end

#developer_role ⇒ Symbol

Returns:

  • (Symbol) 


389
390
391
 
# File 'lib/llm/provider.rb', line 389

def developer_role
  :developer
end

#tool_role ⇒ Symbol

Returns:

  • (Symbol) 


395
396
397
 
# File 'lib/llm/provider.rb', line 395

def tool_role
  :tool
end

#tracer ⇒ LLM::Tracer

Returns the current scoped tracer override or provider default tracer

Returns:

  • (LLM::Tracer) —

    Returns the current scoped tracer override or provider default tracer



402
403
404
 
# File 'lib/llm/provider.rb', line 402

def tracer
  weakmaps.tracer[self] || @tracer || LLM::Tracer::Null.new(self)
end

#tracer=(tracer)

This method returns an undefined value.

Set the provider's default tracer This tracer is shared by the provider instance and becomes the fallback whenever no scoped override is active.

A tracer assigned this way is not scoped, so nothing declares when it is finished and Tracer#on_exit never fires for it. Release what it holds when the provider is done with, or scope it with #with_tracer instead.

Examples:

llm = LLM.openai(key: ENV["KEY"])
llm.tracer = LLM::Tracer.logger(llm, path: "/path/to/log.txt")
llm.with_tracer(llm.tracer) do
  # ...
end

Parameters:



425
426
427
 
# File 'lib/llm/provider.rb', line 425

def tracer=(tracer)
  @tracer = tracer || LLM::Tracer::Null.new(self)
end

#with_tracer(tracer) { ... } ⇒ Object

Override the tracer for the current fiber while the block runs. This is useful when you want per-request or per-turn tracing without replacing the provider's default tracer.

Examples:

llm.with_tracer(LLM::Tracer.logger(llm, io: $stdout)) do
  llm.complete("hello", model: "gpt-5.4-mini")
end

Parameters:

Yields:

Returns:



440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
 
# File 'lib/llm/provider.rb', line 440

def with_tracer(tracer)
  scoped = tracer || LLM::Tracer::Null.new(self)
  wm = weakmaps.tracer
  had_override = wm.key?(self)
  previous = wm[self]
  wm[self] = scoped
  entered = LLM::Tracer.enter(scoped)
  yield
ensure
  if had_override
    wm[self] = previous
  else
    wm.respond_to?(:delete) ? wm.delete(self) : wm[self] = nil
  end
  ##
  # The scope this tracer was opened for is over, and the tracer is
  # told so when the last one that is open for it ends - on whichever
  # thread that happens, because a tool opens a scope of its own for
  # the turn's tracer.
  LLM::Tracer.exit(scoped) if entered
end

#interrupt!(owner) ⇒ nil Also known as: cancel!

Interrupt the active request, if any.

Parameters:

  • owner (Fiber)

Returns:

  • (nil) 


466
467
468
 
# File 'lib/llm/provider.rb', line 466

def interrupt!(owner)
  transport.interrupt!(owner)
end

#request_owner ⇒ Object

This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.

Returns the current request owner used by the transport.

Returns:



475
476
477
 
# File 'lib/llm/provider.rb', line 475

def request_owner
  transport.request_owner
end

#key? ⇒ Boolean

Returns true when an API key is configured

Returns:

  • (Boolean) —

    Returns true when an API key is configured



482
483
484
 
# File 'lib/llm/provider.rb', line 482

def key?
  @key != nil && @key.to_s.strip.size > 0
end

#adapt_function(fn) ⇒ Hash

This method is abstract.

Adapt a Function to the provider-specific tool schema.

Parameters:

Returns:

  • (Hash)

Raises:

  • (NotImplementedError) 


492
493
494
 
# File 'lib/llm/provider.rb', line 492

def adapt_function(fn)
  raise NotImplementedError
end