Dagster

Definition

Dagster is an open-source, Python-based data orchestration platform that lets teams define, schedule, test, and monitor data pipelines as software assets with built-in observability.

Overview

Unlike traditional task-based schedulers, Dagster is built around "software-defined assets" — the tables, files, and ML models a pipeline produces — rather than just the tasks that produce them. This asset-centric model gives teams a clearer picture of data lineage and makes pipelines easier to test and reason about, and it's often discussed as a more modern alternative to Apache Airflow. Dagster pipelines are defined entirely in Python, using ops and asset definitions that Dagster assembles into a dependency graph. A built-in web UI shows run history, asset lineage, and materialization status, while sensors and schedules trigger runs based on events or time. Dagster ships native integrations for the modern data stack, including dbt, Apache Spark, Databricks, and Snowflake, so it can orchestrate transformation, ingestion, and ML steps side by side. It's commonly adopted by teams building on top of a cloud warehouse who want dbt models, Python extraction scripts, and machine learning jobs orchestrated from one place, and it fits naturally alongside the practices covered in SkillVeris's Apache Airflow & Orchestration and dbt & Analytics Engineering courses.

Key Features

• Software-defined assets model pipelines around their data outputs, not just tasks
• Python-native API for defining ops, jobs, and asset graphs
• Built-in web UI for pipeline visualization, run history, and asset lineage
• Native integrations with dbt, Spark, Snowflake, Databricks, and pandas
• Data quality checks and materialization metadata attached to each asset
• Sensors and schedules for event-driven and time-based triggers
• Branch deployments for testing pipeline changes in isolation

Use Cases

Frequently Asked Questions

Related in DevOps