Class: LLM::Eval::Tracer::JSON

Inherits:
Tracer::Null show all
Defined in:
lib/llm/eval/tracer/json.rb

Overview

JSON-based tracer.

Instance Attribute Summary collapse

Instance Method Summary collapse

Methods inherited from Tracer::Null

#set_finish_metadata_proc

Constructor Details

#initialize(llm, options = {}) ⇒ Tracer

Returns a new instance of Tracer.

Parameters:

  • llm (LLM::Provider) —

    A provider

  • options (Hash) (defaults to: {}) —

    A hash of options



14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
 
# File 'lib/llm/eval/tracer/json.rb', line 14

def initialize(llm, options = {})
  super
  @eval = options[:eval]
  @provider = llm.name
  @started_at = Time.now.utc
  @started = nil
  @spans = {}
  @requests = []
  @tools = []
  @errors = []
  @root = File.join(
    Dir.getwd,
    "runs",
    @eval.name,
    llm.name.to_s,
    eval.model.to_s,
    eval.sample.to_s
  )
end

Instance Attribute Details

#eval ⇒ LLM::Eval (readonly)

Returns:



9
10
11
 
# File 'lib/llm/eval/tracer/json.rb', line 9

def eval
  @eval
end

Instance Method Details

#on_exit ⇒ String

Writes the run to disk.

Returns:

  • (String) —

    Returns the path that was written



119
120
121
122
123
124
 
# File 'lib/llm/eval/tracer/json.rb', line 119

def on_exit
  FileUtils.mkdir_p(@root)
  dest = File.join(@root, "sample.json")
  File.write(dest, LLM.json.dump(report, pretty: true))
  dest
end

#on_request_start(operation:, model: nil)

This method returns an undefined value.

Parameters:

  • operation (String)
  • model (String) (defaults to: nil)
  • inputs (Hash, nil) —

    Optional span attributes (e.g. gen_ai.input.messages) from llm.rb or caller.

  • request_id (String) —

    The id of the request. It is a UUIDv7, and it is shared by #on_request_start, #on_request_finish, and #on_request_error for the same request.



37
38
39
40
41
42
 
# File 'lib/llm/eval/tracer/json.rb', line 37

def on_request_start(operation:, model: nil, **)
  @started = clock
  @model ||= model
  return unless operation == "chat"
  @requests << {"index" => @requests.size + 1, "model" => model}
end

#on_request_finish(operation:, res: nil)

This method returns an undefined value.

Parameters:

  • operation (String)
  • res (LLM::Response) (defaults to: nil)
  • span (Object, nil)
  • model (String)
  • outputs (Hash, nil) —

    Optional span attributes (e.g. gen_ai.output.messages) from llm.rb or caller.

  • metadata (Hash, nil) —

    Optional metadata from llm.rb or caller.

  • request_id (String) —

    The id of the request that can be matched back to #on_request_start



47
48
49
50
51
52
53
54
55
56
57
58
 
# File 'lib/llm/eval/tracer/json.rb', line 47

def on_request_finish(operation:, res: nil, **)
  return unless operation == "chat"
  request = @requests.last
  return if request.nil?
  input, output = tokens(res)
  request["duration_s"] = elapsed
  request["ok"] = res.respond_to?(:ok?) ? res.ok? : true
  request["input_tokens"] = input
  request["output_tokens"] = output
  request["reply"] = reply(res)
  request["reasoning"] = reasoning(res)
end

#on_request_error(ex:)

This method returns an undefined value.

Parameters:



63
64
65
66
67
 
# File 'lib/llm/eval/tracer/json.rb', line 63

def on_request_error(ex:, **)
  @errors << {"kind" => "request", "class" => ex.class.to_s, "message" => ex.message}
  request = @requests.last
  request["error"] = ex.message if request
end

#on_tool_start(id:, name:, arguments:) ⇒ LLM::Object

Returns The span an ending is handed, with the call's id and name.

Parameters:

  • id (String) —

    The tool call ID assigned by the model/provider

  • name (String) —

    The tool (function) name.

  • arguments (Hash) —

    The parsed tool arguments.

  • model (String) —

    The model name

Returns:

  • (LLM::Object) —

    The span an ending is handed, with the call's id and name



73
74
75
76
77
 
# File 'lib/llm/eval/tracer/json.rb', line 73

def on_tool_start(id:, name:, arguments:, **)
  @spans[id] = clock
  @tools << {"name" => name, "arguments" => plain(arguments)}
  LLM::Object.from(id:, name:)
end

#on_tool_finish(result:)

This method returns an undefined value.

Parameters:



82
83
84
85
86
87
 
# File 'lib/llm/eval/tracer/json.rb', line 82

def on_tool_finish(result:, **)
  tool = @tools.last
  return if tool.nil?
  tool["duration_s"] = span(result.id)
  tool["ok"] = true
end

#on_tool_error(ex:)

This method returns an undefined value.

Parameters:



92
93
94
95
96
97
 
# File 'lib/llm/eval/tracer/json.rb', line 92

def on_tool_error(ex:, **)
  message = ex.value[:message].to_s
  @errors << {"kind" => "tool", "class" => ex.value[:type].to_s, "message" => message}
  tool = @tools.last
  tool["error"] = message if tool
end

#on_tool_interrupt(ex:, span:)

This method returns an undefined value.

Parameters:



102
103
104
105
106
 
# File 'lib/llm/eval/tracer/json.rb', line 102

def on_tool_interrupt(ex:, span:, **)
  @errors << {"kind" => "tool", "class" => ex.class.to_s, "message" => ex.message}
  tool = @tools.last
  tool["error"] = ex.message if tool
end

#on_interrupt(scope:)

This method returns an undefined value.

Parameters:



111
112
113
 
# File 'lib/llm/eval/tracer/json.rb', line 111

def on_interrupt(scope:, **)
  @errors << {"kind" => scope.to_s, "class" => "LLM::Interrupt", "message" => "interrupted"}
end