Class: LLM::Eval::Tracer::Console

Inherits:
Tracer::Null show all
Defined in:
lib/llm/eval/tracer/console.rb

Overview

Console-based tracer.

Constant Summary collapse

RULE =

Returns:

  • (Integer) 
58

Instance Method Summary collapse

Methods inherited from Tracer::Null

#set_finish_metadata_proc

Constructor Details

#initialize(llm, options = {}) ⇒ Tracer

Returns a new instance of Tracer.

Parameters:

  • llm (LLM::Provider) —

    A provider

  • options (Hash) (defaults to: {}) —

    A hash of options



14
15
16
17
18
19
20
21
22
23
24
25
 
# File 'lib/llm/eval/tracer/console.rb', line 14

def initialize(llm, options = {})
  super
  @tracer = LLM::Object.from(
    io: options[:io] || $stdout, eval: options[:eval],
    provider: llm.name, model: nil, spans: {},
    started: nil, opened: clock, header: false
  )
  @counts = LLM::Object.from(
    turns: 0, requests: 0, input: 0, output: 0,
    tools: {}, failed: 0, errors: 0, slowest: nil
  )
end

Instance Method Details

#on_exit

This method returns an undefined value.



127
128
129
130
131
132
133
134
135
136
137
 
# File 'lib/llm/eval/tracer/console.rb', line 127

def on_exit
  return unless eval.result
  buffer = StringIO.new
  io = @tracer.io
  @tracer.io = buffer
  summary
  @tracer.io = io
  io.print(buffer.string)
  FileUtils.mkdir_p(rundir)
  File.write(File.join(rundir, "sample.txt"), buffer.string.gsub(/\e\[[0-9;]*m/, ""))
end

#eval ⇒ LLM::Eval

Returns:



29
30
31
 
# File 'lib/llm/eval/tracer/console.rb', line 29

def eval
  @tracer.eval
end

#rundir ⇒ String

Returns:

  • (String) 


35
36
37
38
39
 
# File 'lib/llm/eval/tracer/console.rb', line 35

def rundir
  File.join(Dir.getwd, "runs", eval.name,
            llm.name.to_s, eval.model.to_s,
            eval.sample.to_s)
end

#start_trace

This method returns an undefined value.

Parameters:

  • trace_group_id (String, nil) —

    Optional. When present, converted to a 16-byte trace_id so all spans created until #stop_trace are grouped in one trace.

  • name (String) —

    Name for the root span (e.g. "chatbot.turn").

  • attributes (Hash) —

    OpenTelemetry attributes to set on the root span.

  • metadata (Hash, nil) —

    Optional. Trace-level metadata merged into the trace by tracers that support it.



44
45
46
47
 
# File 'lib/llm/eval/tracer/console.rb', line 44

def start_trace(...)
  @counts.turns += 1
  super
end

#on_request_start(operation:, model: nil)

This method returns an undefined value.

Parameters:

  • operation (String)
  • model (String) (defaults to: nil)
  • inputs (Hash, nil) —

    Optional span attributes (e.g. gen_ai.input.messages) from llm.rb or caller.

  • request_id (String) —

    The id of the request. It is a UUIDv7, and it is shared by #on_request_start, #on_request_finish, and #on_request_error for the same request.



52
53
54
55
56
57
58
59
 
# File 'lib/llm/eval/tracer/console.rb', line 52

def on_request_start(operation:, model: nil, **)
  @tracer.started = clock
  @tracer.model ||= model
  @counts.requests += 1 if operation == "chat"
  return unless operation == "chat"
  header
  say "#{paint("->", :cyan)} #{@tracer.provider}#{model ? " #{model}" : ""}"
end

#on_request_finish(operation:, res:)

This method returns an undefined value.

Parameters:

  • operation (String)
  • res (LLM::Response)
  • span (Object, nil)
  • model (String)
  • outputs (Hash, nil) —

    Optional span attributes (e.g. gen_ai.output.messages) from llm.rb or caller.

  • metadata (Hash, nil) —

    Optional metadata from llm.rb or caller.

  • request_id (String) —

    The id of the request that can be matched back to #on_request_start



64
65
66
67
68
69
70
 
# File 'lib/llm/eval/tracer/console.rb', line 64

def on_request_finish(operation:, res:, **)
  return unless operation == "chat"
  usage = res.usage
  @counts.input += usage.input_tokens
  @counts.output += usage.output_tokens
  say "#{paint("<-", :green)} #{format("%.1fs", clock - @tracer.started)}"
end

#on_request_error(ex:)

This method returns an undefined value.

Parameters:



75
76
77
78
 
# File 'lib/llm/eval/tracer/console.rb', line 75

def on_request_error(ex:, **)
  @counts.errors += 1
  say "#{paint("!!", :red)} #{ex.class}: #{ex.message}"
end

#on_tool_start(id:, name:, arguments:) ⇒ LLM::Object

Returns The span an ending is handed, with the call's id and name.

Parameters:

  • id (String) —

    The tool call ID assigned by the model/provider

  • name (String) —

    The tool (function) name.

  • arguments (Hash) —

    The parsed tool arguments.

  • model (String) —

    The model name

Returns:

  • (LLM::Object) —

    The span an ending is handed, with the call's id and name



84
85
86
87
88
89
 
# File 'lib/llm/eval/tracer/console.rb', line 84

def on_tool_start(id:, name:, arguments:, **)
  @tracer.spans[id] = clock
  @counts.tools[name] = @counts.tools.fetch(name, 0) + 1
  say "   #{paint("..", :yellow)} #{name} #{trim(arguments)}"
  LLM::Object.from(id:, name:)
end

#on_tool_finish(result:)

This method returns an undefined value.

Parameters:



94
95
96
97
98
99
100
 
# File 'lib/llm/eval/tracer/console.rb', line 94

def on_tool_finish(result:, **)
  duration = took(result.id)
  if duration && (@counts.slowest.nil? || duration > @counts.slowest.last)
    @counts.slowest = [result.name, duration]
  end
  say "   #{paint("ok", :green)} #{result.name} #{duration ? format("%.2fs", duration) : "?"}"
end

#on_tool_error(ex:)

This method returns an undefined value.

Parameters:



105
106
107
108
 
# File 'lib/llm/eval/tracer/console.rb', line 105

def on_tool_error(ex:, **)
  @counts.failed += 1
  say "   #{paint("no", :red)} #{ex.value[:type]}: #{ex.value[:message]}"
end

#on_tool_interrupt(ex:, span:)

This method returns an undefined value.

Parameters:



113
114
115
116
 
# File 'lib/llm/eval/tracer/console.rb', line 113

def on_tool_interrupt(ex:, span:, **)
  @counts.failed += 1
  say "   #{paint("no", :red)} #{span.name} interrupted: #{ex.message}"
end

#on_interrupt(scope:)

This method returns an undefined value.

Parameters:



121
122
123
 
# File 'lib/llm/eval/tracer/console.rb', line 121

def on_interrupt(scope:, **)
  say "#{paint("!!", :red)} #{scope} interrupted"
end