Class: LLM::Tracer::Telemetry

Inherits:
Tracer
  • Object
show all
Defined in:
lib/llm/tracer/telemetry.rb

Overview

The LLM::Tracer::Telemetry tracer provides telemetry support through the opentelemetry-ruby RubyGem. The gem should be installed separately since this feature is opt-in and disabled by default.

Examples:

InMemory export

#!/usr/bin/env ruby
require "llm"
require "pp"

llm = LLM.openai(key: ENV["KEY"])
llm.tracer = LLM::Tracer.telemetry(llm)

ctx = LLM::Context.new(llm)
ctx.talk "hello"
ctx.talk "how are you?"
ctx.tracer.spans.each { |span| pp span }

OTLP export

#!/usr/bin/env ruby
require "llm"
require "opentelemetry-exporter-otlp"

exporter = OpenTelemetry::Exporter::OTLP::Exporter.new

llm = LLM.openai(key: ENV["KEY"])
llm.tracer = LLM::Tracer.telemetry(llm, exporter:)

ctx = LLM::Context.new(llm)
ctx.talk "hello"
ctx.talk "how are you?"

See Also:

Instance Method Summary collapse

Constructor Details

#initialize(llm, options = {}) ⇒ LLM::Tracer::Telemetry

param [LLM::Provider] provider An LLM provider



44
45
46
47
48
49
50
51
 
# File 'lib/llm/tracer/telemetry.rb', line 44

def initialize(llm, options = {})
  ##
  # Optional stdlib dependencies
  LLM.require "digest", "~> 3.3" unless defined?(::Digest)
  super
  @exporter = options.delete(:exporter)
  setup!
end

Instance Method Details

#on_exit ⇒ nil

This method is called when the outermost Provider#with_tracer method has returned control to the caller. It will flush the queue of any spans that are not yet exported, and it is called after a trace has finished.

Returns:

  • (nil) 


253
254
255
 
# File 'lib/llm/tracer/telemetry.rb', line 253

def on_exit
  flush!
end

#start_trace(trace_group_id: nil, name: "llm", attributes: {}, metadata: nil) ⇒ self

When +trace_group_id+ is provided, it is converted to an OpenTelemetry trace_id (via a deterministic 16-byte hash) so all spans until #stop_trace share that trace_id and appear as one trace in OTLP/Langfuse.

Parameters:

  • trace_group_id (String, nil) (defaults to: nil) —

    Optional. When present, converted to a 16-byte trace_id so all spans created until #stop_trace are grouped in one trace.

  • name (String) (defaults to: "llm") —

    Name for the root span (e.g. "chatbot.turn").

  • attributes (Hash) (defaults to: {}) —

    OpenTelemetry attributes to set on the root span.

  • metadata (Hash, nil) (defaults to: nil) —

    Optional. Trace-level metadata merged into the trace by tracers that support it.

Returns:

  • (self) 


60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'lib/llm/tracer/telemetry.rb', line 60

def start_trace(trace_group_id: nil, name: "llm", attributes: {}, metadata: nil)
  return self if trace_group_id.to_s.empty?
  span_context = span_context_from_trace_group_id(trace_group_id.to_s)
  parent_ctx = ::OpenTelemetry::Trace.context_with_span(
    ::OpenTelemetry::Trace.non_recording_span(span_context)
  )
  attrs = attributes.compact
  attrs["llm.trace_group_id"] = trace_group_id.to_s
  root_span = @tracer.start_span(
    name,
    kind: :server,
    attributes: attrs,
    with_parent: parent_ctx
  )
  @root_span = root_span
  @root_context = ::OpenTelemetry::Trace.context_with_span(root_span)
  self
end

#stop_trace ⇒ self

Returns:

  • (self) 


81
82
83
84
85
86
 
# File 'lib/llm/tracer/telemetry.rb', line 81

def stop_trace
  @root_span&.finish
  @root_span = nil
  @root_context = nil
  self
end

#on_request_start(operation:, model: nil, inputs: nil, request_id:) ⇒ Object

Parameters:

  • operation (String)
  • model (String) (defaults to: nil)
  • inputs (Hash, nil) (defaults to: nil) —

    Optional span attributes (e.g. gen_ai.input.messages) from llm.rb or caller.

  • request_id (String) —

    The id of the request. It is a UUIDv7, and it is shared by #on_request_start, #on_request_finish, and #on_request_error for the same request.



90
91
92
93
94
95
96
 
# File 'lib/llm/tracer/telemetry.rb', line 90

def on_request_start(operation:, model: nil, inputs: nil, request_id:)
  case operation
  when "chat" then start_chat(operation:, model:, inputs:)
  when "retrieval" then start_retrieval(operation:)
  else nil
  end
end

#on_request_finish(operation:, res:, model: nil, span: nil, outputs: nil, metadata: nil, request_id:) ⇒ Object

Parameters:

  • operation (String)
  • res (LLM::Response)
  • span (Object, nil) (defaults to: nil)
  • model (String) (defaults to: nil)
  • outputs (Hash, nil) (defaults to: nil) —

    Optional span attributes (e.g. gen_ai.output.messages) from llm.rb or caller.

  • metadata (Hash, nil) (defaults to: nil) —

    Optional metadata from llm.rb or caller.

  • request_id (String) —

    The id of the request that can be matched back to #on_request_start



100
101
102
103
104
105
106
107
 
# File 'lib/llm/tracer/telemetry.rb', line 100

def on_request_finish(operation:, res:, model: nil, span: nil, outputs: nil, metadata: nil, request_id:)
  return nil unless span
  case operation
  when "chat" then finish_chat(operation:, model:, res:, span:, outputs:, metadata:)
  when "retrieval" then finish_retrieval(operation:, res:, span:)
  else nil
  end
end

#on_request_error(ex:, span:, request_id:) ⇒ Object

Parameters:



111
112
113
114
115
116
117
118
 
# File 'lib/llm/tracer/telemetry.rb', line 111

def on_request_error(ex:, span:, request_id:)
  return nil unless span
  attributes = {"error.type" => ex.class.to_s}.compact
  attributes.each { span.set_attribute(_1, _2) }
  span.add_event("gen_ai.request.finish")
  span.status = ::OpenTelemetry::Trace::Status.error(ex.message)
  span.tap(&:finish)
end

#on_tool_start(id:, name:, arguments:, model:)

This method returns an undefined value.

Parameters:

  • id (String) —

    The tool call ID assigned by the model/provider

  • name (String) —

    The tool (function) name.

  • arguments (Hash) —

    The parsed tool arguments.

  • model (String) —

    The model name



123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
 
# File 'lib/llm/tracer/telemetry.rb', line 123

def on_tool_start(id:, name:, arguments:, model:)
  attributes = {
    "gen_ai.operation.name" => "execute_tool",
    "gen_ai.request.model" => model,
    "gen_ai.tool.call.id" => id,
    "gen_ai.tool.name" => name&.to_s,
    "gen_ai.tool.call.arguments" => LLM.json.dump(arguments),
    "gen_ai.provider.name" => llm.name.to_s,
    "server.address" => llm.host,
    "server.port" => llm.port
  }.merge!(trace_attributes(span_kind: "tool")).compact
  span_name = ["execute_tool", name].compact.join(" ")
  span = create_span(span_name.empty? ? "gen_ai.tool" : span_name, attributes:)
  span.add_event("gen_ai.tool.start")
  span
end

#on_tool_finish(result:, span:, started_at:, finished_at:)

This method returns an undefined value.

Parameters:



143
144
145
146
147
148
149
150
151
152
153
 
# File 'lib/llm/tracer/telemetry.rb', line 143

def on_tool_finish(result:, span:, started_at:, finished_at:)
  return nil unless span
  attributes = {
    "gen_ai.tool.call.id" => result.id,
    "gen_ai.tool.name" => result.name&.to_s,
    "gen_ai.tool.call.result" => LLM.json.dump(result.value)
  }.compact
  attributes.each { span.set_attribute(_1, _2) }
  span.add_event("gen_ai.tool.finish")
  span.tap(&:finish)
end

#on_tool_error(ex:, span:, started_at:, finished_at:)

This method returns an undefined value.

Parameters:



158
159
160
161
162
163
164
165
 
# File 'lib/llm/tracer/telemetry.rb', line 158

def on_tool_error(ex:, span:, started_at:, finished_at:)
  return nil unless span
  attributes = {"error.type" => ex.value[:type]}.compact
  attributes.each { span.set_attribute(_1, _2) }
  span.add_event("gen_ai.tool.finish")
  span.status = ::OpenTelemetry::Trace::Status.error(ex.value[:message].to_s)
  span.tap(&:finish)
end

#on_tool_interrupt(ex:, span:, started_at:, finished_at:)

This method returns an undefined value.

Parameters:



170
171
172
173
174
175
176
177
 
# File 'lib/llm/tracer/telemetry.rb', line 170

def on_tool_interrupt(ex:, span:, started_at:, finished_at:)
  return nil unless span
  attributes = {"error.type" => "LLM::Interrupt"}
  attributes.each { span.set_attribute(_1, _2) }
  span.status = ::OpenTelemetry::Trace::Status.error("tool interrupted")
  span.add_event("gen_ai.tool.finish")
  span.tap(&:finish)
end

#on_interrupt(scope:, span: nil, request_id: nil)

This method returns an undefined value.

Parameters:

  • scope (Symbol)
  • span (Object) (defaults to: nil)
  • request_id (String) (defaults to: nil) 


184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
 
# File 'lib/llm/tracer/telemetry.rb', line 184

def on_interrupt(scope:, span: nil, request_id: nil)
  case scope
  when :request
    ##
    # This is where we match on_request_start and
    # close the span it had opened. It is the only
    # scope where we have a span to close.
    attributes = {"error.type" => "LLM::Interrupt"}
    attributes.each { span.set_attribute(_1, _2) }
    span.status = OpenTelemetry::Trace::Status.error("request interrupted")
    span.add_event("gen_ai.request.finish")
    span.tap(&:finish)
  when :tool
    ##
    # This means an interrupt was received when
    # an agent was running tools and no request
    # was active at the time.
    #
    # There is no span to close, and each tool
    # call is closed by #{on_tool_interrupt}.
    # There is seemingly nothing useful to do
    # here.
  when :agent
    ##
    # This means an interrupt was received when
    # an agent was between requests and no request
    # was active at the time.
    #
    # There is no span to close, and no tools are
    # running. There is seemingly nothing useful to
    # do here.
  else
    raise LLM::Error,
          "scope '#{scope}' is not a valid tracer scope"
  end
end

#spans ⇒ Array<OpenTelemetry::SDK::Trace::SpanData>

Note:

This method returns an empty array for exporters that do not implement 'finished_spans' such as the OTLP exporter

Returns:

  • (Array<OpenTelemetry::SDK::Trace::SpanData>) 


227
228
229
230
231
 
# File 'lib/llm/tracer/telemetry.rb', line 227

def spans
  return [] unless @exporter.respond_to?(:finished_spans)
  flush!
  @exporter.finished_spans
end

#flush! ⇒ nil

Note:

Exports are batched in the background by default. Long-lived processes usually do not need to call this method. Short-lived scripts should call #flush! before exit to reduce the risk of losing spans that are still buffered.

Flushes queued telemetry to the configured exporter.

Returns:

  • (nil) 


241
242
243
244
 
# File 'lib/llm/tracer/telemetry.rb', line 241

def flush!
  @tracer_provider.force_flush
  nil
end