Skip to main content

Control Plane

Monorepo for all Control Plane (CP) components.

[!NOTE] "Control Plane" is gradually evolving to mean everything upstream of the index workflows running in global AWS accounts (i.e. not partitioned cells). This repo currently contains the old "Controller" components which were never partitioned, and the old cell-based CP components will be migrated incrementally.

This doc will mostly explain the contents of the repo and how to interact with it. For more details about the Control Plane itself and development, see the Control Plane docs.

Prerequisites​

  • Python 3.11 (for now)
  • Pants

Setup​

  • Install Pants, a powerful build system that handles virtually all of our code-related tasks. Run pants help goals to see what it can do.

  • Clone the repo and open it in your IDE of choice. If you're working on a single component, you may prefer to open just that subtree of the file system to reduce the noise. Git and Pants commands will still work, but relatives paths will be different.

  • If you're doing development work, create a branch and PR on GitHub to trigger GitHub Actions which will typically take care of your deployment-related needs. Since the Control Plane is serverless, we spin up a new environment for each PR.

Structure​

The monorepo structure follows EDR-6:

  • /infra: Infrastructure as code (i.e. CDK) for deployments.
  • /components: Each of the independent components of the Control Plane system.
  • /.github/workflows: The CI/CD workflows that run on GitHub Actions.
  • /docs: Markdown docs for the Control Plane docs site (see mkdocs.yml)
  • /3rdparty: Pants-related dependency lockfiles (generally ignore these, unless adding or upgrading dependencies - see below).

There's also lots of top-level config files to configure various tools.

Development​

You can change one or more components and/or infra directories in the same PR.

For the time being, deployments are manual, so you can manage how the changes are rolled out after merging to main. However you should always ensure that the main branch is ready for deployment to production so as to not block the release of other changes. Whatever testing you need to feel confident should be automated on your branch.

Changing dependencies​

Most components share a single universe resolve. To add or update a dependency, edit 3rdparty/python/requirements.txt and regenerate the lockfile:

pants generate-lockfiles --resolve=universe

Deployment​

Development environments​

CI creates a new environment for each PR, so to create a dev environment, just create a PR!

[!NOTE] CI deployments currently require you to manually create a secret in Secrets Manager. Hopefully we will fix this soon.

Production environment​

For production deployments, you should first create a tag and release with release notes in GitHub, and announce the impending release on #prod-releases. This can be very lightweight for small changes.

Then manually trigger the Deploy Pipeline action in the GitHub Actions UI. This will deploy to and run smoke tests against each of the staging, preprod, and prod environments in turn (note that prod will require manual approval in GitHub).