dbt integration

Overview (dbt 1.5)

With the integration of dbt, you can create metric definitions in code within your dbt core project and effortlessly synchronize them with your Steep workspace. By doing so, you can maintain a unified source of truth for your metric definitions within version-controlled code. Additionally, metrics generated in dbt can be combined with manually defined metrics in Steep, allowing for flexibility when organizing your workspace.

Prerequisites

Before you can import your dbt metrics to Steep you need to connect Steep to your database, the same database that is used by your dbt project. See Connect database.

Metric definitions in dbt

As a starting point you need to define your metrics in your dbt core project according to the dbt metrics specification. Definitions can be placed in any .yml file under the metrics key. For the full specification, read more in the dbt docs.

Example yml definition below.

metrics:
  - name: new_customers
    label: New Customers
    model: ref('dim_customers')
    description: "Count of new paying customers using the product"
    calculation_method: count_distinct
    expression: user_id

    timestamp: signup_date
    time_grains: [day, week, month, quarter, year]

    dimensions:
      - plan
      - country

    filters:
      - field: is_paying
        operator: "is"
        value: "true"

Manual import of metrics

You can manually upload your dbt metrics to Steep, which is a good choice as a starting point or for metric definitions that rarely change. To manually import metric definitions, you can run your dbt project locally and import the resulting manifest.json file into Steep.

1. Run your dbt project

Run your dbt project to produce a manifest.json eg. using dbt run or dbt build.

2. Import manifest.json file in Steep

Go to the dbt integration page in your Steep workspace and upload the manifest.json file. Steep will then provide feedback on any metrics that were created, updated, or skipped during the import process.

3. Explore your metrics

Once the import process is complete, explore your metrics in the sidebar and begin analyzing the metric data right away. Explore metrics.

Automatic sync of metrics via API

To ensure that your metrics are always up to date in Steep, you can sync your dbt metrics automatically via the Steep API as part of your continuous integration process or scheduled production run.

1. Create API key

To get started, you need to create an API key. You can create a key on the dbt integration page in your Steep workspace. Click the "Create API Key" button to create a new key.

2. Post manifest.json to API

Run your dbt project in the environment of your choice, and then upload the manifest.json by using POST to https://api.steep.app/dbt-metrics, attaching your API key as an authorization header. See example below using curl.

curl -H "Content-Type: application/json" \
     -H "Authorization: ApiKey API_KEY_HERE" \
     -X POST \
     -d @/path/to/manifest.json \
     https://api.steep.app/dbt-metrics

The API will return 200 with a response body indicating the outcome of the operation.

If validation of the manifest fails the response will include an error message.

{
  outcome: "failed"
  error: string
}

If successul the response will contain references to the dbt unique_id field for each metric created, updated or that failed with an error message.

{
  outcome: "completed";
  created: string[];
  updated: string[];
  errors: Record<string, string>;
}

3. Explore your metrics

Once the sync has successfully completed, your Steep metric definitions will have been updated accordingly and you can explore and analyze your metric data within Steep. Explore metrics.