Queue backlog check
An agent checks consumer lag on the job queues and scales workers when lag keeps growing.
Source: scripts/dev/review-runbooks/queue-backlog-check.yaml. CI checks it with runbook spec validate; teams,
people, skills and called runbooks it names resolve against your server
when you publish it.
apiVersion: therunbook.app/v1kind: Runbookmetadata: id: queue-backlog-check title: Queue backlog check description: An agent checks consumer lag on the job queues and scales workers when lag keeps growing. owners: [team:oncall] tags: [queues, worker, agentic]spec: defaults: { actor: agent } steps: - id: lag title: Consumer lag per queue check: true instructions: Read consumer lag for every queue over 15 minutes; warn when it grows for 10 minutes straight. outputs: worst_queue: { type: string, required: true } lag_seconds: { type: integer, required: true } - id: scale if: "${{ steps.lag.verdict != 'pass' }}" title: Scale the worker pool instructions: Raise the worker pool's desired count by 50% (at most 40) and watch lag for 10 minutes.Publish it to a project:
runbook push queue-backlog-check.yaml --project <project> --draftrunbook publish queue-backlog-check --project <project>