Get startedSign in

Installing Dagster

Guide to self-hosting Dagster with Plural

Description

Plural will install Dagster in a dependency-aware manner onto a Plural-managed Kubernetes cluster with one CLI command.

Installation

We currently support Dagster for the following providers:

plural bundle install dagster dagster-aws

Setup Configuration

  • dagsterBucket: s3 bucket for storing dagster logs
  • hostname: fqdn on which to deploy your dagster instance

Getting Started

First, create an account on https://app.plural.sh. This is to sync your installations and allow for the delivery of automated upgrades. You will not be asked to provide any infrastructure credentials or sensitive information.

Next, install the Plural CLI. If you're on Mac, the brew tap is the easiest way to do this, as it installs Helm, Terraform, and kubectl for you. For alternate installation methods such as curl and Docker, click here.

brew install pluralsh/plural/plural
Warning:

Before you proceed, make sure that your cloud provider CLI is properly configured and updated to the latest version. If it is not configured correctly, Plural will fail and won't be able to create resources on your behalf.

Plural uses a GitOps workflow to manage configuration files and deployment state, so you'll need a Git repository to store your Plural configuration. Plural supports GitHub, GitLab, and Bitbucket currently. If you're planning on using Bitbucket, follow this guide. This repository will contain the Helm charts, Terraform config, and Kubernetes manifests that Plural will autogenerate for you.

Navigate to a directory that you would like to create this Git repository in and run plural init. This will start a configuration wizard to configure your Git repo and cloud provider for use with Plural.

Installing Dagster

Run this command to find the bundle name associated with your cloud provider:

plural bundle list dagster

Now, to add it to your workspace, run the install command with the bundle name that you received in the previous step. This will guide you through some minor configuration of your Dagster installation, such as choosing the subdomain that your Dagster installation will be hosted at. If you are on AWS, it would look like this:

plural bundle install dagster dagster-aws

To generate the Helm charts, Terraform, and deployment YAML, run:

plural build

To deploy your infrastructure and commit the changes to your origin Git repository, run:

plural deploy --commit "deploying dagster"

Installing the Plural Console

Plural comes with a console that provides DevOps workflows, customized dashboards, and an interactive UI for managing your Kubernetes cluster. It's not required, but highly recommended. You can install the console the same way that you installed Dagster:

plural bundle install console console-aws
plural build
plural deploy --commit "deploying the console too"

Accessing your Dagster installation

Navigate to dagster.YOUR_SUBDOMAIN.onplural.sh to access the Dagster UI. If you set up a different subdomain for Dagster during the plural bundle install phase, use that instead.

Uninstalling Dagster

To bring down your Plural installation of Dagster, run:

plural destroy dagster

To bring down your entire Plural deployment, run:

plural destroy
Warning:

Only do this if you're absolutely sure you want to bring down all associated resources with this repository.

Troubleshooting

If you run into any issues with installing Dagster on Plural, feel free to join our Discord Community so that we can help you out.

If you'd like to request any updates or new features for our Dagster installation, feel free to open an issue here.