Sources and Freshness Checks

Analogy categoryAnalogies

Declaring Sources

A source is a raw table loaded into your warehouse by an external tool — Fivetran, Airbyte, a Kafka connector — that dbt itself did not create. You declare sources in a .yml file under a source: key, grouping tables by the loading tool or system, and reference them in staging models with the source() function instead of hardcoding the raw database and schema name.

🏏

Cricket analogy: This is like a stadium's ground staff declaring the pitch and outfield conditions before a match begins — data the batting side didn't create but must reference precisely to plan their innings.

yaml

# models/staging/sources.yml
sources:
  - name: raw_shopify
    database: raw
    schema: shopify
    tables:
      - name: orders
        description: "Raw orders synced via Fivetran every 15 minutes."
        loaded_at_field: _fivetran_synced
        freshness:
          warn_after: {count: 6, period: hour}
          error_after: {count: 24, period: hour}

Freshness: Detecting a Stalled Pipeline

The freshness config compares the current time against loaded_at_field — a timestamp column like _fivetran_synced or updated_at — to determine how stale the source data is, using separate warn_after and error_after thresholds. Running dbt source freshness executes this check independently of any model build, which is exactly what you want in an orchestration pipeline: fail fast before spending compute rebuilding downstream models on data you already know is stale.

🏏

Cricket analogy: This is like a groundskeeper checking the pitch moisture reading against a threshold before play starts, rather than letting the match begin and only discovering unsafe conditions mid-over.

bash

# run only the freshness check, independent of building any models
dbt source freshness

# in an orchestrated DAG: fail the pipeline early if freshness fails
dbt source freshness && dbt build --select source:raw_shopify+

Testing and Documenting Sources Like Models

Sources support the exact same tests: key as models — you can attach not_null, unique, or a custom generic test directly to a source table's column, catching a broken upstream sync (like a Fivetran schema drift dropping a column) before it propagates into staging models. Sources also support description fields the same way models do, so the docs site documents raw tables alongside your transformed ones, closing the gap between 'what data do we have' and 'what did we build from it'.

🏏

Cricket analogy: This is like applying the same fitness-screening protocol used on the senior squad to incoming academy trainees, catching a physical issue before it ever affects a first-team selection.

Referencing source('raw_shopify', 'orders') instead of a hardcoded database.schema.table string means changing the underlying raw database name (say, during a warehouse migration) only requires editing sources.yml in one place, not every staging model that touches that table.

loaded_at_field must be a column that reliably reflects when the row was loaded, not when the underlying business event happened — using an order's created_at instead of a sync timestamp like _fivetran_synced will produce a freshness check that's meaningless once historical backfills or late-arriving events are involved.

Sources declare raw, externally-loaded tables in YAML, referenced in models via source() instead of hardcoded table names.
freshness config compares the current time to loaded_at_field against warn_after/error_after thresholds.
dbt source freshness runs independently of model builds, ideal for fast-failing an orchestrated pipeline early.
Sources support the same tests: key as models, catching upstream schema drift before it hits staging models.
Sources support description fields, so raw tables appear documented in the dbt docs site alongside models.
loaded_at_field must reflect load/sync time, not business-event time, or freshness results will be misleading.
Changing a raw database/schema name only requires editing sources.yml, not every model that references it.