e-volv

e-volv Berserker: agent observability and evalsOpt-in

Every agent run, and what it cost

Point any OpenTelemetry-instrumented agent at e-volv Berserker and every run appears with its tokens, cost, tool calls and session. Replay a production run on a new prompt with write tools stripped, catch a report that claims what no tool call did, and score a sample of live runs as they land. It works on its own, and beside e-volv Observer the run sits next to the logs and errors of the service that ran it.

  1. Prompt
  2. Model call
  3. Tool call
  4. Model call
  5. Answer

The strings part at the tool call. Everything after it is where the tokens went, which is the reading a bill cannot give you. Timings illustrative.

Eight surfaces

In the order the product's own navigation lists them. Behind a per-workspace flag while the first partners run it.

  1. RunsOpt-in

    One row per top-level agent span, rolled up from the GenAI conventions, OpenInference or the Vercel AI SDK: model, tokens with the cache split out, cost, tool calls, error. Open one and read it as a tree.

  2. SessionsOpt-in

    Runs threaded by conversation id, so a support thread or a ticket reads as one exchange rather than nine rows.

  3. CostOpt-in

    Spend by agent, by model and by day, priced per model call from a maintained list or from the price you set on the model.

  4. EvalsOpt-in

    Datasets from real runs, fixtures or a CSV; experiments that replay a dataset through up to four prompts or models, read only, and score every answer; online scoring of a sample of live runs.

  5. ReviewOpt-in

    Low-scoring and flagged runs in one queue: thumbs, a label, a note, and one click to add the run to a dataset.

  6. PromptsOpt-in

    Named prompts with numbered versions and labels an agent fetches by name, so a prompt change needs no deploy; each version with the cost and score of the runs that used it.

  7. AlertsOpt-in

    Spend per day, error rate and p95 duration per agent, as rules on the same channels as every other alert in the workspace.

  8. UsageOpt-in

    Agent spans this month against the allowance, so the ceiling is something you watch coming rather than something you hit.

What happens to a run

One run, from the span your agent emits to the score it gets. Each stage is named the way the code names it, so you can go and find it.

  1. Span

    Your agent emits invoke_agent, chat and execute_tool spans. Through OpenTelemetry today; @e-volv/agents and evolv-agents, which set the same attributes without a collector, are coming.

  2. Rollup

    After the span is stored, the trace is read back and every top-level agent span becomes an agent_run row: tokens summed from the model calls, tool calls counted, the cost priced per call.

  3. Score

    A share of runs you choose goes to the scorers as it lands: the claims guardrail, a JSON check, a judge model on your rubric. Low scores join the review queue.

  4. Replay

    Any run with its prompt captured can be replayed on a new prompt or model, read only, and diffed against what it said the first time.

How runs get in

Already on OpenTelemetry? Point your exporter at the Berserker endpoint and nothing else changes. Two thin packages that set the same attributes without a collector are built and not yet on a registry.

  • OpenTelemetryOTLP over HTTP to the traces endpoint, with the GenAI, OpenInference or Vercel AI SDK attributes read as they arrive. Docs
  • Node.jsNot in service@e-volv/agents, coming soon
  • PythonNot in serviceevolv-agents, coming soon

Point an agent at it

Works with or without e-volv Observer. Free for 50,000 agent spans a month with 14 days of retention; included with e-volv Observer paid or Studio at 1,000,000 spans and 30 days.

Start freeSee the fares