Skip to content

Canary analysis

An agent compares the canary with the baseline fleet and recommends promote or abort.

Source: scripts/dev/review-runbooks/canary-analysis.yaml. CI checks it with runbook spec validate; teams, people, skills and called runbooks it names resolve against your server when you publish it.

apiVersion: therunbook.app/v1
kind: Runbook
metadata:
id: canary-analysis
title: Canary analysis
description: An agent compares the canary with the baseline fleet and recommends promote or abort.
owners: [team:oncall]
tags: [deploy, canary, agentic]
spec:
execution:
title: "Canary ${{ inputs.service }} ${{ inputs.version }}"
inputs:
service: { type: enum, options: [api, web, worker, billing], required: true }
version: { type: string, required: true }
defaults: { actor: agent }
steps:
- id: compare_errors
title: Canary error rate within 10% of baseline
check: true
instructions: Compare the canary's 5xx rate with the baseline fleet over 20 minutes.
outputs:
canary_rate: { type: number, required: true }
baseline_rate: { type: number, required: true }
- id: compare_latency
title: Canary p95 within 15% of baseline
check: true
instructions: Compare p95 latency of canary and baseline over the same window.
outputs:
canary_p95_ms: { type: integer, required: true }
baseline_p95_ms: { type: integer, required: true }
- id: recommend
title: Recommend promote or abort
instructions: Recommend promoting the canary to the full fleet, or aborting, with one sentence of reasoning.
outputs:
recommendation: { type: enum, options: [promote, abort], required: true }

Publish it to a project:

Terminal window
runbook push canary-analysis.yaml --project <project> --draft
runbook publish canary-analysis --project <project>