Installing Dagster
Guide to self-hosting Dagster with Plural
Description
Plural will install Dagster in a dependency-aware manner onto a Plural-managed Kubernetes cluster with one CLI command.
Installation
We currently support Dagster for the following providers:
plural bundle install dagster dagster-aws
Setup Configuration
dagsterBucket
: s3 bucket for storing dagster logshostname
: fqdn on which to deploy your dagster instance
Getting Started
First, create an account on https://app.plural.sh. This is to sync your installations and allow for the delivery of automated upgrades. You will not be asked to provide any infrastructure credentials or sensitive information.
Next, install the Plural CLI. If you're on Mac, the brew tap is the easiest way to do this, as it installs Helm, Terraform, and kubectl for you. For alternate installation methods such as curl and Docker, click here.
brew install pluralsh/plural/plural
Warning:
Before you proceed, make sure that your cloud provider CLI is properly configured and updated to the latest version. If it is not configured correctly, Plural will fail and won't be able to create resources on your behalf.
Plural uses a GitOps workflow to manage configuration files and deployment state, so you'll need a Git repository to store your Plural configuration. Plural supports GitHub, GitLab, and Bitbucket currently. If you're planning on using Bitbucket, follow this guide. This repository will contain the Helm charts, Terraform config, and Kubernetes manifests that Plural will autogenerate for you.
Navigate to a directory that you would like to create this Git repository in and run plural init
. This will start a configuration wizard to configure your Git repo and cloud provider for use with Plural.
Installing Dagster
Run this command to find the bundle name associated with your cloud provider:
plural bundle list dagster
Now, to add it to your workspace, run the install command with the bundle name that you received in the previous step. This will guide you through some minor configuration of your Dagster installation, such as choosing the subdomain that your Dagster installation will be hosted at. If you are on AWS, it would look like this:
plural bundle install dagster dagster-aws
To generate the Helm charts, Terraform, and deployment YAML, run:
plural build
To deploy your infrastructure and commit the changes to your origin Git repository, run:
plural deploy --commit "deploying dagster"
Installing the Plural Console
Plural comes with a console that provides DevOps workflows, customized dashboards, and an interactive UI for managing your Kubernetes cluster. It's not required, but highly recommended. You can install the console the same way that you installed Dagster:
plural bundle install console console-aws plural build plural deploy --commit "deploying the console too"
Accessing your Dagster installation
Navigate to dagster.YOUR_SUBDOMAIN.onplural.sh
to access the Dagster UI. If you set up a different subdomain for Dagster during the plural bundle install
phase, use that instead.
Uninstalling Dagster
To bring down your Plural installation of Dagster, run:
plural destroy dagster
To bring down your entire Plural deployment, run:
plural destroy
Warning:
Only do this if you're absolutely sure you want to bring down all associated resources with this repository.
Troubleshooting
If you run into any issues with installing Dagster on Plural, feel free to join our Discord Community so that we can help you out.
If you'd like to request any updates or new features for our Dagster installation, feel free to open an issue here.