Sharing Outputs with Continuous Deployment

It's frequently necessary to take infrastructure created in a stack and use the data in a Service being deployed to K8s. A few usecases where this can be important:

• exporting IAM role ARNs for access to various AWS services like S3 or SQS
• exporting DB connection strings to configure as k8s secrets for a webserver
• exporting a dynamically created S3 bucket name to use in your deployed service

This also facilitates end-to-end self-service, as you no longer need a human in-the-loop to apply that last mile configuration, plus you get continuous reconciliation in the event a terraform change implies a recreation of those resources.

End To End Example

We do this a lot in our service catalog, available here: https://github.com/pluralsh/scaffolds/tree/main/catalogs. A basic example would be something like our Airbyte setup, where the terraform stack has an outputs file like so:

output "access_key_id" {
  value = aws_iam_access_key.airbyte.id
}

output "secret_access_key" {
  value = aws_iam_access_key.airbyte.secret
  sensitive = true
}

output "postgres_host" {
  value = try(module.db.db_instance_address, "")
}

output "postgres_password" {
  value = random_password.password.result
  sensitive = true
}

output "oidc_cookie_secret" {
  value = random_password.oidc_cookie.result
  sensitive = true
}

output "oidc_client_id" {
  value = plural_oidc_provider.airbyte.client_id
  sensitive = true
}

output "oidc_client_secret" {
  value = plural_oidc_provider.airbyte.client_secret
  sensitive = true
}

Airbyte needs fixed aws access keys to communicate with S3 and also there's a dynamically generated OIDC client that's used for auth against it's webserver, alongside postgres credentials.

with a InfrastructureStack resource will like this:

apiVersion: deployments.plural.sh/v1alpha1
kind: InfrastructureStack
metadata:
  name: airbyte-data
  namespace: apps
spec:
  detach: false
  type: TERRAFORM
  approval: true
  manageState: true
  git:
    ref: main
    folder: terraform/apps/airbyte/data
  repositoryRef:
    name: infra
    namespace: infra
  clusterRef:
    name: mgmt
    namespace: infra

All of this data needs to be used by the service that is actually deployed to K8s, so it will explicitly "import" that stack with its imports declaration, like so: