dbt projects

How DiffCost reads a dbt project#

DiffCost compares two manifest.json files: one from your main branch (the baseline manifest) and one compiled in the pull request's (PR's) workflow run. A model counts as changed when its source file checksum differs or its compiled SQL differs. The compiled-SQL check catches changes that come from outside the model file: macros, var()s, dbt_project.yml and schema YAML config, and sources. Before comparing, DiffCost maps relation names between the two manifests, so a different target schema alone doesn't mark a model changed. DiffCost then dry-runs each manifest's compiled SQL. dbt Cloud isn't required.

Setup is two workflow changes:

1. a workflow on your main branch that saves the compiled manifest as an artifact;
2. the PR workflow compiles the PR, downloads that artifact and runs DiffCost.

Set up the main-branch workflow first: merge it, run it once, and check that the run has a dbt-manifest-main artifact. Then change the PR workflow.

Requirements#

• Compile both manifests against your production target, so the compiled SQL reads production tables. A CI target that writes to per-PR schemas would make DiffCost dry-run tables that don't exist yet, and those rows show Cost unavailable.
• Use dbt compile, not dbt parse: dbt parse leaves the compiled SQL empty.
• Compile the whole project. A model that has no compiled SQL in one manifest (for example after dbt compile --select …) is skipped with a warning, not reported as removed or new. If its file changed in the PR, it shows Cost unavailable instead, and blocks when a threshold applies.
• Keep target/ out of git, and run dbt compile in the same job before DiffCost. A model whose file changed in git while its manifest entry still matches the baseline shows "Cost unavailable: the model file changed but the PR manifest was not recompiled", and blocks when a threshold applies.
• actions/checkout needs fetch-depth: 0, as for plain SQL.
• dbt compile isn't a dry run. Macros that query the warehouse at compile time (such as run_query or dbt_utils.get_column_values) run real queries, billed to the project in your profile. Only DiffCost's own jobs are dry runs.
• For incremental models, DiffCost prices the SQL that dbt compile produced. When the table already exists in the production target, that is usually the incremental run, not a full refresh.

Install dbt with a keyless profile#

Both workflows below install dbt and write a profile. With Workload Identity Federation, use method: oauth: dbt then uses the credentials file that google-github-actions/auth wrote. Use the profile name from your dbt_project.yml (profile:), and your own project, dataset and location.

With a service account key instead, keep the same profile and give google-github-actions/auth@v2 credentials_json: ${{ secrets.GCP_SA_KEY }} in place of workload_identity_provider and service_account (keep its if:). DiffCost reads the same credentials file, so the DiffCost step doesn't need gcp_credentials.

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install dbt-bigquery
      - name: Write profiles.yml (keyless)
        run: |
          mkdir -p ~/.dbt
          cat > ~/.dbt/profiles.yml <<'EOF'
          my_dbt_project:          # the `profile:` name in dbt_project.yml
            target: prod
            outputs:
              prod:
                type: bigquery
                method: oauth      # the credentials google-github-actions/auth wrote
                project: my-project
                dataset: analytics
                location: US
                threads: 4
          EOF
      - run: dbt deps

The service account then needs what dbt compile needs in BigQuery as well as what DiffCost needs: usually read access to every dataset the project's models and sources read.

Save the main-branch manifest#

On every merge to your main branch, and on a schedule, save the manifest as an artifact. This is a separate workflow that runs on the main branch, not on pull requests. Replace main with your default branch, here and in the PR workflow's download step:

# .github/workflows/dbt-main.yml
name: dbt manifest (main)
on:
  push:
    branches: [main]
  schedule:
    - cron: "0 6 * * 1"   # weekly, so the artifact doesn't expire in a quiet repository (public repositories: GitHub disables schedules after 60 days without activity)
  workflow_dispatch:      # create or refresh the baseline on demand (Actions → Run workflow)

permissions:
  contents: read
  id-token: write   # only for Workload Identity Federation

jobs:
  manifest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: 'projects/123456789012/locations/global/workloadIdentityPools/github-pool/providers/github-provider'
          service_account: 'diffcost-ci@my-project.iam.gserviceaccount.com'
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install dbt-bigquery
      - name: Write profiles.yml (keyless)
        run: |
          mkdir -p ~/.dbt
          cat > ~/.dbt/profiles.yml <<'EOF'
          my_dbt_project:
            target: prod
            outputs:
              prod:
                type: bigquery
                method: oauth
                project: my-project
                dataset: analytics
                location: US
                threads: 4
          EOF
      - run: dbt deps
      - run: dbt compile --target prod
      - uses: actions/upload-artifact@v4
        with:
          name: dbt-manifest-main
          path: target/manifest.json
          retention-days: 90   # GitHub's maximum unless your organization raised it

Artifacts expire. Without the scheduled run, a repository with no merges for 90 days loses its baseline: PRs that modify existing models then show Cost unavailable and block under a threshold, including the PR whose merge would refresh the artifact. To create a baseline again, run the workflow from the Actions tab (Run workflow).

In the pull request workflow#

Compile the PR and download the latest main-branch manifest from the other workflow. These steps replace the steps of the diffcost job from the Quickstart; don't add a second workflow that runs DiffCost (one DiffCost run per PR). actions/download-artifact only sees the current run unless you pass a run ID, so this example uses dawidd6/action-download-artifact:

# .github/workflows/diffcost.yml
name: DiffCost
on: pull_request   # every PR, no `paths:` filter

permissions:
  contents: read
  pull-requests: write
  actions: read   # read artifacts from the main-branch workflow
  id-token: write # only for Workload Identity Federation

jobs:
  diffcost:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      # Fork PRs get no OpenID Connect (OIDC) token or secrets, so these steps can't run there (see below)
      - uses: google-github-actions/auth@v2
        if: github.event.pull_request.head.repo.full_name == github.repository
        with:
          workload_identity_provider: 'projects/123456789012/locations/global/workloadIdentityPools/github-pool/providers/github-provider'
          service_account: 'diffcost-ci@my-project.iam.gserviceaccount.com'
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install dbt-bigquery
      - name: Write profiles.yml (keyless)
        run: |
          mkdir -p ~/.dbt
          cat > ~/.dbt/profiles.yml <<'EOF'
          my_dbt_project:
            target: prod
            outputs:
              prod:
                type: bigquery
                method: oauth
                project: my-project
                dataset: analytics
                location: US
                threads: 4
          EOF
      - run: dbt deps
      - run: dbt compile --target prod
        if: github.event.pull_request.head.repo.full_name == github.repository
      - uses: dawidd6/action-download-artifact@v6
        if: github.event.pull_request.head.repo.full_name == github.repository
        with:
          workflow: dbt-main.yml
          branch: main
          name: dbt-manifest-main
          path: .diffcost-baseline/
          if_no_artifact_found: warn
      - uses: diffcost/action@v1
        with:
          manifest_path: target/manifest.json
          gcp_project: my-project
          # api_key: ${{ secrets.DIFFCOST_API_KEY }}

The download step always takes the artifact from the branch you name, so PRs into other branches are compared against that branch's manifest too.

Run the PR workflow on every PR (no paths: filter).

PRs from forks. On a fork PR, dbt compile can't reach BigQuery, so the auth, compile and download steps are skipped. DiffCost then reports in the job summary that no manifest was compiled ("could not analyze"); it never reports that as "no changes". With fail_on_error: true, this fails the DiffCost step on every fork PR. Without the if:, the auth step would fail the job before DiffCost runs.

Merge queues. Add merge_group to on: and let the auth, compile and download steps run on it, as in Merge queues.

Dependabot PRs run in your repository, so the if: doesn't skip them, but they only get Dependabot secrets. With a service account key, add it as a Dependabot secret too (in the repository's GitHub settings: Settings → Secrets and variables → Dependabot); otherwise dbt compile fails before DiffCost runs.

Projects in a subdirectory#

For a project in transform/ (the directory with its dbt_project.yml):

• run the dbt steps in that directory, in both workflows: add working-directory: transform to the dbt deps and dbt compile steps;
• upload transform/target/manifest.json in the main-branch workflow;
• keep the download path .diffcost-baseline/ as it is: the baseline paths are relative to the repository root, also for a project in a subdirectory;
• set manifest_path: transform/target/manifest.json;
• keep .diffcost.yml in the repository root.

      - run: dbt compile --target prod
        working-directory: transform
        if: github.event.pull_request.head.repo.full_name == github.repository   # this step is only in the PR workflow
      # main-branch workflow:
      - uses: actions/upload-artifact@v4
        with:
          name: dbt-manifest-main
          path: transform/target/manifest.json
      # PR workflow:
      - uses: diffcost/action@v1
        with:
          manifest_path: transform/target/manifest.json
          gcp_project: my-project

The project root is the nearest directory at or above the manifest that has a dbt_project.yml on the base branch. DiffCost matches model files and ignored_paths against repository-relative paths such as transform/models/orders.sql. If none of the manifest's model files exist in the checkout, DiffCost stops with an error rather than reporting no changes.

Where DiffCost looks for manifests#

The PR manifest is manifest_path. Without that input, DiffCost looks for target/manifest.json next to each dbt_project.yml on the base branch (the repository root or a subdirectory), so committing a manifest in a PR can't switch a plain-SQL repository to dbt.

DiffCost refuses the PR manifest, and reports an error instead of "no changes", when the manifest:

• can't be read or isn't a dbt manifest;
• has no models, or has models but no compiled SQL (written by dbt parse, not dbt compile);
• is the unmodified copy committed in the repository, not one compiled in this run (the PR author could have committed stale or edited compiled SQL);
• is reached through a symlink;
• is the same file as the baseline.

The baseline manifest is read from .diffcost-baseline/manifest.json or target/prod-manifest.json, relative to the repository root. If a file exists at one of these paths but DiffCost can't trust or use it, DiffCost stops with "could not analyze". It doesn't fall back to comparing without a baseline, because a PR could then switch off change detection by committing any file at that path. DiffCost can't use the file if it is:

• tracked in git (committed on the base branch or in the PR);
• reached through a symlink;
• unreadable;
• a manifest without compiled models.

Never commit files at these two paths: download the baseline as an artifact in each run.

If neither file exists and your repository commits a compiled manifest at manifest_path, DiffCost uses the base branch's committed copy as the baseline. That copy comes from the base branch, so the PR can't change it. The PR's own manifest must still be compiled in the run.

One dbt project per pull request#

DiffCost analyzes one dbt project per PR. If it finds manifests for several dbt projects, it stops with "could not analyze": set manifest_path to the one to analyze.

The project's paths are its model, macro, seed, snapshot, analysis and test paths from dbt_project.yml on the base branch, plus target/. DiffCost doesn't analyze .sql files that the PR changes outside them: a second dbt project, a project the PR adds, or loose SQL files such as scripts/*.sql, even inside the project directory. It reports them together as one Cost unavailable row ("not analyzed") instead of "no changes", and that row blocks when a threshold applies.

If the files don't affect BigQuery cost (scripts for another database, or a project you don't want to gate), add them to ignored_paths in .diffcost.yml, for example - "marketing/**". The comment shows the line to add. DiffCost reads .diffcost.yml from the base branch, so add the line in a separate PR to the base branch. Once that PR is merged, the next push to the blocked PR (or Update branch) picks it up; re-running the old job doesn't.

Without a baseline#

Before the first main-branch artifact exists (or after it expired), DiffCost analyzes only the files changed in the PR:

• models added in the PR show as New;
• models that already existed show Cost unavailable, because there is no base version to compare against;
• changes to other project files can't be costed without the baseline. They show as one Cost unavailable row, which blocks when a threshold applies. These files are:
  • files under macro-paths, seed-paths, snapshot-paths and analysis-paths in the base branch's dbt_project.yml (by default macros/, seeds/, snapshots/, analyses/ or analysis/);
  • YAML files under model-paths or next to model files;
  • dbt_project.yml, packages.yml, dependencies.yml and selectors.yml;
  • files in local packages (local: entries).

The same row appears with a baseline when the PR changes such files and some models aren't compiled in one of the two manifests (the PR's or the baseline), for example after dbt compile --select …, because those models' compiled SQL can't be compared.

Models that depend on objects the PR creates#

When a model's PR dry run fails only because it depends on something the same PR creates, DiffCost marks the row Not costed: depends on objects this PR creates. Typical cases: a model selects a column that an upstream model adds in the same PR ("Unrecognized name"), or reads a model the PR adds ("Not found: Table").

DiffCost shows this only when both are true:

• A model, seed or snapshot upstream of the model, directly or through other models, is new or changed in this PR. An ephemeral model counts only through its own parents. Without a baseline manifest, only upstream files changed in git count.
• For a "Not found" error that names the missing table, view or dataset: an upstream model, seed or snapshot creates it in this PR. It is new, newly materialized, or moved by a schema, database or alias change. If BigQuery's error doesn't name the object, the first condition is enough.

Otherwise the PR doesn't explain the error (for example a column typo, or a manifest compiled against another target), and the row shows Cost unavailable, like any other unknown cost.

A Not costed model counts as an unknown cost, and blocks under a threshold, only if structural analysis also flags a risk on it, such as a removed partition filter, SELECT * or a new JOIN, or if the model already exists and structural analysis couldn't run on it (SQL that DiffCost's parser can't read, such as BigQuery pipe syntax). A new model has no earlier version to compare, so it never gets structural analysis and doesn't block. In plain SQL files, the same errors produce a Cost unavailable row, which blocks under a threshold.