Thresholds and alerts

Thresholds take effect only with an API key on Starter, Team or the 14-day Team trial. On the Free plan, thresholds are kept but not evaluated: once the DiffCost GitHub App is installed, the diffcost/cost-check status always passes, and without the App no status is posted.

The cost policy is the set of thresholds that applies to a repository. In the dashboard, each row under Merge-blocking thresholds is one threshold; some of the form's notices also call it a policy.

How a threshold is evaluated#

Thresholds apply to the worst single query in the pull request (PR), per run:

• absolute: 5.00 blocks when any one query's cost goes up by more than $5.00 per run. A new query counts with its full cost.
• relative: 200 blocks when any one query's cost goes up by more than 200%. Only queries whose base cost (the Current column) is above $0 count.
• With both set, either one blocks.

Increases are not offset by decreases elsewhere in the PR: +$10/run on one query and −$10/run on another still exceeds a $5 threshold. The monthly projection never affects the verdict.

Unknown costs block. When a threshold applies and some queries have an unknown cost, DiffCost blocks the PR with the reason "N queries could not be costed". It never treats an unknown cost as $0. The exception is a dbt model marked Not costed (it depends on objects the same PR creates): it blocks only when structural analysis also flags a risk on it, or can't run on a model that already exists. See dbt projects.

Editions pricing: with the Editions pricing model, the comment shows bytes instead of dollars, but dollar thresholds still compare the on-demand equivalent at your Cost per TiB rate, and the PR comment's block reasons say "(on-demand equivalent)". Percentage thresholds work the same in both models.

Where to set thresholds#

In the dashboard: DiffCost admins use Settings → Merge-blocking thresholds → Add or update a threshold. See Using the dashboard.

In the repository: anyone who can merge into the base branch can change the file.

# .diffcost.yml (repository root)
thresholds:
  absolute: 5.00   # USD per run, worst single query
  relative: 200    # percent, worst single query

DiffCost reads .diffcost.yml from the PR's base commit, so a PR can't loosen its own policy by editing the file.

Change the policy for an open PR#

• Dashboard thresholds apply from the next DiffCost run. Re-run the DiffCost job on the PR to get a new status.
• .diffcost.yml (including ignored_paths): change it in a separate PR to the base branch; that PR changes no SQL, so it passes. After it merges, push a commit to the blocked PR, or click Update branch in GitHub. Re-running the old job doesn't pick up the change, because it keeps its old base commit.

Which threshold applies#

DiffCost uses the first threshold in this list that exists, and ignores the rest:

1. a dashboard threshold for the repository (Team plan and the trial);
2. the dashboard's org-wide threshold (scope *);
3. the thresholds: in the repository's .diffcost.yml;
4. none: every PR passes (when a query got more expensive, the status says "Worst query +$X/run (no threshold set)").

A dashboard threshold replaces the file's thresholds entirely: the limits aren't merged.

For a passing run, the dashboard's comparison details say which threshold applied (repository, org-wide or .diffcost.yml) and its limits. For a blocked run they list the reasons, which give the limit that was exceeded but not its source.

On Starter, per-repository dashboard thresholds (for example ones created during the trial) don't apply. The org-wide threshold applies instead. If there is none, each repository's .diffcost.yml applies. The dashboard marks those rows "Inactive: needs Team".

Choosing values#

• Start with an absolute limit sized to what one bad merge would cost per run. Multiply by runs per month to see the monthly exposure.
• Add a relative limit to catch big jumps on cheap queries, for example 200%. Small absolute changes on cheap queries can be large percentages, so keep it generous if many queries cost cents.
• Use a per-repository threshold (Team) for repositories with very different query sizes.
• Try a policy before enforcing it: set the threshold before you make diffcost/cost-check required, watch the PR comments and the dashboard's Result column for a week, then make the check required.

When a PR exceeds the limit#

• The diffcost/cost-check status fails, and the PR can't merge if the check is required (see Merge blocking).
• The PR comment lists the reasons, for example "fct_daily_revenue: +$7.57/run exceeds the $5.00/run per-query threshold". The status and the dashboard word them as "Worst query +$7.57/run exceeds $5.00/run limit".
• A Slack alert goes out, if one is configured (below).
• To merge anyway, fix the query, change the policy (see Change the policy for an open PR), or have an admin bypass branch protection.

Slack alerts#

Available on Starter, Team and the trial. DiffCost sends an alert for each run that exceeds the cost policy, including runs whose status couldn't be posted ("not enforced"). Every push that still exceeds the policy sends another alert, and so does a merge-queue run that exceeds it. Which webhook DiffCost uses depends on which threshold applied:

• Dashboard thresholds can carry their own webhook (the Slack webhook URL (optional) field). The DiffCost service posts to it, alongside the commit status. The alert is titled "DiffCost — Merge Blocked" ("… (merge queue)" for a merge-queue run), even when the status couldn't be posted.
• .diffcost.yml can set slack_webhook. The Action posts to it from your runner when the file's thresholds were the ones applied. It needs the dashboard settings to have been fetched: when DiffCost couldn't fetch them (key rejected, rate-limited or service unreachable at the start of the run), no alert is sent.

If your organization has an org-wide threshold, the .diffcost.yml webhook is never used, because the org-wide threshold applies instead of the file. Ask a DiffCost admin to add a webhook to the dashboard threshold.

Alerts link the PR and give the reasons and the worst query's increase per run. They contain no SQL. A dashboard-threshold alert names no queries; a .diffcost.yml alert, sent straight from your runner to Slack, also names queries: the one with the largest increase, and those in its block reasons.

If Slack refuses an alert (for example, the webhook was revoked), the PR comment, the job summary and the log say so, with Slack's answer. A dashboard threshold also shows its last failed alert on its row. To check a dashboard threshold's webhook, use Send test message on the threshold; for a .diffcost.yml webhook, send a test message yourself when you set it up. See Slack alerts don't arrive.

To use a webhook from .diffcost.yml, keep the URL in a secret and pass it to the DiffCost step under one of the allowed names (SLACK_WEBHOOK, SLACK_WEBHOOK_URL or DIFFCOST_SLACK_WEBHOOK):

# .diffcost.yml
slack_webhook: ${{ SLACK_WEBHOOK }}

# workflow
      - uses: diffcost/action@v1
        with:
          api_key: ${{ secrets.DIFFCOST_API_KEY }}
        env:
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}

The value must be an https://hooks.slack.com/ URL. Create one in Slack with an incoming webhook for the channel that should get the alerts.