dbt Cloud environments
An environment determines how dbt Cloud will execute your project in the dbt Cloud IDE or dbt Cloud CLI (for development) and scheduled jobs (for deployment).
Critically, in order to execute dbt, environments define three variables:
- The version of dbt Core that will be used to run your project
- The warehouse connection information (including the target database/schema settings)
- The version of your code to execute
Each dbt Cloud project can have only one development environment, but there is no limit to the number of deployment environments, providing you the flexibility and customization to tailor the execution of scheduled jobs.
Use environments to customize settings for different stages of your project and streamline the execution process by using software engineering principles. This page will detail the different types of environments and how to intuitively configure your development environment in dbt Cloud.
Types of environments
In dbt Cloud, there are two types of environments:
- Deployment environment — Determines the settings used when jobs created within that environment are executed.
Types of deployment environments:- General
- Staging
- Production
- Development environment — Determines the settings used in the dbt Cloud IDE or dbt Cloud CLI, for that particular project.
Each dbt Cloud project can only have a single development environment but can have any number of deployment environments.
Development | Staging | Deployment | |
---|---|---|---|
Determines settings for | dbt Cloud IDE or dbt Cloud CLI | dbt Cloud Job runs | dbt Cloud Job runs |
How many can I have in my project? | 1 | Any number | Any number |
For users familiar with development on dbt Core, each environment is roughly analogous to an entry in your profiles.yml
file, with some additional information about your repository to ensure the proper version of code is executed. More info on dbt core environments here.
Common environment settings
Both development and deployment environments have a section called General Settings, which has some basic settings that all environments will define:
Setting | Example Value | Definition | Accepted Values |
---|---|---|---|
Name | Production | The environment name | Any string! |
Environment Type | Deployment | The type of environment | [Deployment, Development] |
dbt Version | 1.4 (latest) | The dbt version used | Any dbt version in the dropdown |
Default to Custom Branch | ☑️ | Determines whether to use a branch other than the repository’s default | See below |
Custom Branch | dev | Custom Branch name | See below |
- dbt Cloud allows users to select any dbt release. At this time, environments must use a dbt version greater than or equal to v1.0.0; lower versions are no longer supported.
- If you select a current version with
(latest)
in the name, your environment will automatically install the latest stable version of the minor version selected. - Go Versionless, which removes the need for manually upgrading environment, while ensuring you are always up to date with the latest fixes and features.
Custom branch behavior
By default, all environments will use the default branch in your repository (usually the main
branch) when accessing your dbt code. This is overridable within each dbt Cloud Environment using the Default to a custom branch option. This setting will have slightly different behavior depending on the environment type:
- Development: determines which branch in the dbt Cloud IDE or dbt Cloud CLI developers create branches from and open PRs against.
- Deployment: determines the branch is cloned during job executions for each environment.
For more info, check out this FAQ page on this topic!
Extended attributes
Extended attributes are are currently not supported for SSH tunneling
Extended attributes allows users to set a flexible profiles.yml snippet in their dbt Cloud Environment settings. It provides users with more control over environments (both deployment and development) and extends how dbt Cloud connects to the data platform within a given environment.
Extended attributes are set at the environment level, and can partially override connection or environment credentials, including any custom environment variables. You can set any YAML attributes that a dbt adapter accepts in its profiles.yml
.
![Extended Attributes helps users add profiles.yml attributes to dbt Cloud Environment settings using a free form text box. Extended Attributes helps users add profiles.yml attributes to dbt Cloud Environment settings using a free form text box.](/img/docs/dbt-cloud/using-dbt-cloud/extended-attributes.jpg?v=2)
The following code is an example of the types of attributes you can add in the Extended Attributes text box:
dbname: jaffle_shop
schema: dbt_alice
threads: 4
username: alice
password: '{{ env_var(''DBT_ENV_SECRET_PASSWORD'') }}'
Extended Attributes don't mask secret values
We recommend avoiding setting secret values to prevent visibility in the text box and logs. A common workaround is to wrap extended attributes in environment variables. In the earlier example, password: '{{ env_var(''DBT_ENV_SECRET_PASSWORD'') }}'
will get a value from the DBT_ENV_SECRET_PASSWORD
environment variable at runtime.
How extended attributes work
If you're developing in the dbt Cloud IDE, dbt Cloud CLI, or orchestrating job runs, extended attributes parses through the provided YAML and extracts the profiles.yml
attributes. For each individual attribute:
-
If the attribute exists in another source (such as your project settings), it will replace its value (like environment-level values) in the profile. It also overrides any custom environment variables (if not itself wired using the syntax described for secrets above)
-
If the attribute doesn't exist, it will add the attribute or value pair to the profile.
Only the top-level keys are accepted in extended attributes
This means that if you want to change a specific sub-key value, you must provide the entire top-level key as a JSON block in your resulting YAML. For example, if you want to customize a particular field within a service account JSON for your BigQuery connection (like 'project_id' or 'client_email'), you need to provide an override for the entire top-level keyfile_json
main key/attribute using extended attributes. Include the sub-fields as a nested JSON block.
Create a development environment
To create a new dbt Cloud development environment:
- Navigate to Deploy -> Environments
- Click Create Environment.
- Select Development as the environment type.
- Fill in the fields under General Settings and Development Credentials.
- Click Save to create the environment.
Set developer credentials
To use the dbt Cloud IDE or dbt Cloud CLI, each developer will need to set up personal development credentials to your warehouse connection in their Profile Settings. This allows you to set separate target information and maintain individual credentials to connect to your warehouse.
Deployment environment
Deployment environments in dbt Cloud are necessary to execute scheduled jobs and use other features (like different workspaces for different tasks). You can have many environments in a single dbt Cloud project, enabling you to set up each space in a way that suits different needs (such as experimenting or testing).
Even though you can have many environments, only one of them can be the "main" deployment environment. This would be considered your "production" environment and represents your project's "source of truth", meaning it's where your most reliable and final data transformations live.
To learn more about dbt Cloud deployment environments and how to configure them, refer to the Deployment environments page. For our best practices guide, read dbt Cloud environment best practices for more info.