BigQuery access and keyless auth

What DiffCost does in BigQuery#

DiffCost submits dry-run jobs only (dryRun: true). A dry run returns the number of bytes the query would scan and no rows, and it is free: it uses no slots and isn't billed. DiffCost runs up to two dry runs per changed query: one for the base version, if there is one (for dbt, only with a baseline manifest), and one for the pull request (PR) version, unless the PR removes the query.

BigQuery checks the same permissions on a dry run as on a real query, so the service account needs:

• bigquery.jobs.create on the project that runs the jobs (the gcp_project input). This permission has no effect when granted on a dataset;
• bigquery.tables.get and bigquery.tables.getData on every table and view the queries read.

roles/bigquery.jobUser on the project and roles/bigquery.dataViewer on each dataset give exactly that. The getData permission allows reading table data, so grant it only on the datasets your queries use. The commands are in the Quickstart.

If your SQL reads tables in other projects (for example dbt sources), grant roles/bigquery.dataViewer on those datasets as well. Otherwise those queries show "Access Denied" instead of a cost.

Keyless authentication with Workload Identity Federation#

With Workload Identity Federation (WIF), you don't store a long-lived service account key in GitHub. google-github-actions/auth exchanges the job's short-lived GitHub OpenID Connect (OIDC) token for temporary Google Cloud credentials. It writes them to a file in the workspace and points GOOGLE_APPLICATION_CREDENTIALS at that file, and DiffCost reads it. Keep the auth step's default create_credentials_file: true.

Prerequisites#

You need these roles on the project, or a project Owner who runs the steps for you:

• roles/serviceusage.serviceUsageAdmin: enable the APIs below;
• roles/iam.serviceAccountAdmin: create the service account and its impersonation binding;
• roles/resourcemanager.projectIamAdmin: grant roles/bigquery.jobUser on the project;
• roles/bigquery.jobUser: run the GRANT statements (they are query jobs);
• roles/bigquery.dataOwner on each dataset (or roles/bigquery.admin): grant read access on the datasets;
• roles/iam.workloadIdentityPoolAdmin: create the pool and provider;
• roles/iam.serviceAccountKeyAdmin: only if you create a service account key instead.

Enable the required APIs:

PROJECT_ID="your-gcp-project-id"

gcloud services enable \
    iam.googleapis.com \
    iamcredentials.googleapis.com \
    sts.googleapis.com \
    cloudresourcemanager.googleapis.com \
    bigquery.googleapis.com \
    --project="${PROJECT_ID}"

Step 1: Create the service account#

Already followed the Quickstart? You created diffcost-ci and its permissions there. Set the variables below and skip to step 2: running service-accounts create again fails with "already exists".

PROJECT_ID="your-gcp-project-id"
SA_NAME="diffcost-ci"
SA_EMAIL="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"

gcloud iam service-accounts create "${SA_NAME}" \
    --project="${PROJECT_ID}" \
    --description="DiffCost CI dry runs" \
    --display-name="diffcost-ci"

# Run (dry-run) jobs: on the project
gcloud projects add-iam-policy-binding "${PROJECT_ID}" \
    --member="serviceAccount:${SA_EMAIL}" \
    --role="roles/bigquery.jobUser"

# Read the tables your SQL references: on each dataset (repeat per dataset)
bq query --nouse_legacy_sql --project_id="${PROJECT_ID}" \
    "GRANT \`roles/bigquery.dataViewer\` ON SCHEMA \`${PROJECT_ID}.analytics\` TO 'serviceAccount:${SA_EMAIL}'"

To grant read access on every dataset in the project instead, bind roles/bigquery.dataViewer with gcloud projects add-iam-policy-binding in place of the last command.

Step 2: Create a workload identity pool and provider#

Google Cloud requires an attribute condition on GitHub OIDC providers. Use it to restrict token exchange to your GitHub organization:

POOL_NAME="github-pool"
PROVIDER_NAME="github-provider"
GITHUB_ORG="your-github-org"

gcloud iam workload-identity-pools create "${POOL_NAME}" \
    --project="${PROJECT_ID}" \
    --location="global" \
    --display-name="GitHub Actions Pool"

gcloud iam workload-identity-pools providers create-oidc "${PROVIDER_NAME}" \
    --project="${PROJECT_ID}" \
    --location="global" \
    --workload-identity-pool="${POOL_NAME}" \
    --display-name="GitHub Actions Provider" \
    --issuer-uri="https://token.actions.githubusercontent.com" \
    --attribute-mapping="google.subject=assertion.sub,attribute.actor=assertion.actor,attribute.repository=assertion.repository,attribute.repository_owner=assertion.repository_owner" \
    --attribute-condition="assertion.repository_owner == '${GITHUB_ORG}'"

Get the provider's full resource name for the workflow:

gcloud iam workload-identity-pools providers describe "${PROVIDER_NAME}" \
    --project="${PROJECT_ID}" \
    --location="global" \
    --workload-identity-pool="${POOL_NAME}" \
    --format="value(name)"

The output looks like projects/123456789012/locations/global/workloadIdentityPools/github-pool/providers/github-provider. It contains your numeric project number, not the project ID.

Step 3: Let the repository impersonate the service account#

Restrict impersonation to one repository, so other workflows can't mint tokens for this service account:

GITHUB_REPO="your-github-org/your-repo"
PROJECT_NUMBER=$(gcloud projects describe "${PROJECT_ID}" --format="value(projectNumber)")

gcloud iam service-accounts add-iam-policy-binding "${SA_EMAIL}" \
    --project="${PROJECT_ID}" \
    --role="roles/iam.workloadIdentityUser" \
    --member="principalSet://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_NAME}/attribute.repository/${GITHUB_REPO}"

Repeat this binding for every repository that runs DiffCost.

Step 4: Use it in the workflow#

The job needs id-token: write for the OIDC exchange:

# .github/workflows/diffcost.yml
name: DiffCost
on: pull_request   # every pull request, no `paths:` filter

jobs:
  diffcost:
    runs-on: ubuntu-latest
    permissions:
      contents: read       # git diff against the base branch
      pull-requests: write # post/update the cost comment
      id-token: write      # Workload Identity Federation OIDC exchange
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # required: DiffCost diffs against the PR's base commit

      - uses: google-github-actions/auth@v2
        # Fork PRs get no OIDC token: without this `if:` the step fails the job before
        # DiffCost runs. On fork PRs DiffCost then runs with structural analysis only
        # (plain SQL; a dbt project's fork PR has no compiled manifest: "could not analyze").
        if: github.event.pull_request.head.repo.full_name == github.repository
        with:
          workload_identity_provider: 'projects/123456789012/locations/global/workloadIdentityPools/github-pool/providers/github-provider'
          service_account: 'diffcost-ci@your-gcp-project-id.iam.gserviceaccount.com'

      - uses: diffcost/action@v1
        with:
          gcp_project: 'your-gcp-project-id'   # set it with WIF: the project that runs the dry-run jobs
          # api_key: ${{ secrets.DIFFCOST_API_KEY }}   # dashboard history; merge blocking on paid plans and the trial

Set gcp_project with WIF: it's the project that runs the dry-run jobs. Without it, the Google client library picks a project from the environment.

DiffCost uses the gcp_credentials input or the file named in GOOGLE_APPLICATION_CREDENTIALS, nothing else. On a self-hosted runner, other Application Default Credentials (such as a VM's metadata server) aren't used. Two setup mistakes lead to a structural-only report (risk flags, no dollar amounts):

• The auth step didn't export GOOGLE_APPLICATION_CREDENTIALS, for example because it was skipped, ran in another job, or ran with create_credentials_file: false. The comment says "No BigQuery credentials configured".
• The variable names a file that doesn't exist in the DiffCost container, for example because actions/checkout ran after the auth step and removed the credentials file from the workspace. The log warns that GOOGLE_APPLICATION_CREDENTIALS points to a file that does not exist, and the comment says the credentials "could not be used".

In both cases, run the auth step in the same job, after actions/checkout and before DiffCost, with the default create_credentials_file: true.

If you use dbt, put the same if: on the dbt compile and artifact download steps (see dbt projects).

Using a service account key instead#

Pass the key's raw JSON as gcp_credentials, usually from a secret: gcp_credentials: ${{ secrets.GCP_SA_KEY }}. The value must be the raw JSON, which starts with {, or the path of a credentials file that exists in the workspace. DiffCost refuses a base64-encoded key with a message saying so, and never prints the value. If the key can't be used (for example it is malformed), the PR comment says so and gives the reason, instead of "no credentials". Without gcp_project, DiffCost uses the key's project_id.

Dependabot PRs only get Dependabot secrets. Add GCP_SA_KEY as a Dependabot secret too (in the repository's GitHub settings: Settings → Secrets and variables → Dependabot). Otherwise Dependabot PRs get a structural-only report (in a dbt repository, dbt compile fails before DiffCost runs). If DIFFCOST_API_KEY is also a Dependabot secret, that report counts every changed query as an unknown cost and blocks when a threshold applies. Without the key, no threshold is evaluated and no diffcost/cost-check status is posted; the comment says "No DiffCost API key".

Rotate the key regularly; see Maintenance and updates. Moving to WIF removes the stored key.