Skip to main content

Platform Release Process

This document describes the release process for the Controller, Admin, and Ecommerce platforms.

Overview​

All platforms follow a continuous deployment model where changes merged to main are automatically deployed through staging and preprod environments before requiring manual approval for production. The release system automates changelog generation, GitHub release creation, and Slack notifications.

Deployment Pipeline​

Each platform has its own deploy pipeline workflow:

  • Controller: deploy_pipeline.yaml
  • Admin: deploy_pipeline_admin.yaml
  • Ecom: deploy_pipeline_ecom.yaml

Pipeline stages:

  1. ref-sha: Locks the commit SHA at pipeline start
  2. Staging: Automatic deployment and testing
  3. Preprod: Automatic deployment and testing after staging succeeds
  4. prepare-release: Creates a draft GitHub release (runs in parallel with prod)
  5. Prod: Requires manual approval, then deploys and tests

Concurrency and Draft Clobbering​

Each commit merged to main triggers its own pipeline. Staging and preprod are serialized (one at a time), but multiple pipelines can be waiting for prod approval simultaneously.

Critical behavior: prepare-release deletes any existing draft release for the platform before creating a new one. This means only the most recent pipeline to complete preprod has a valid draft. Earlier pipelines' drafts are clobbered.

When a user approves an earlier pipeline for prod deployment, publish-release handles two cases:

  1. Draft exists for this commit: Publishes the draft as-is (notes are already correct — prepare-release generated them for exactly the (last_release, this_commit] range).
  2. No matching draft (clobbered by a later pipeline): Generates the release from scratch — computes commits, generates changelog, and creates + publishes the release in one step.

In both cases, if the released commit is behind HEAD of main, a new draft release is created for the remaining commits (released_commit, HEAD] so the next deploy pipeline picks them up.

Release Preparation (prepare-release)​

After preprod succeeds, the prepare-release action creates a draft GitHub release.

What It Does​

  1. Finds the last published release for the platform
  2. Identifies commits in (last_release, target_sha] that triggered the platform's deploy workflow
  3. Generates a changelog from associated PRs
  4. Deletes any existing draft releases for the platform
  5. Creates a new draft GitHub release targeting target_sha
  6. Posts announcement to internal team Slack channel (if webhook is configured)

The target_sha is the specific commit being deployed (locked by ref-sha at pipeline start), not HEAD of main. This ensures the draft notes are always scoped to exactly the commits being released.

Release Tag Format​

<platform>.<version>

Examples: controller.5, admin.12, ecom.27

Release Notes Format​

Release notes include:

  • Changes: List of PRs with title, number, and author
  • Chores: PRs prefixed with chore: (case-insensitive) in title or description are grouped under a separate "Chores" section at the bottom
  • Deployment: Links to the deploy pipeline run and previous release

Example release notes:

## Changes

- #1551 - Add productTitle to merchandising fields (@author)
- #1546 - Add Lambda deployment monitoring (@author)

## Chores

- #1548 - chore: update dependencies (@author)

## Deployment

- [Deploy Pipeline Run](https://github.com/marqo-ai/cloud_control_plane/blob/445728d0a3ede69aeb9fa75d585b6d2547a443f9/docs/dev/link)
- Previous release: [controller.4](https://github.com/marqo-ai/cloud_control_plane/blob/445728d0a3ede69aeb9fa75d585b6d2547a443f9/docs/dev/link)
- [Full diff](https://github.com/marqo-ai/cloud_control_plane/blob/445728d0a3ede69aeb9fa75d585b6d2547a443f9/docs/dev/link)

Slack Notifications​

The release system sends Slack notifications at two points:

1. Draft Release (Internal Team)​

When a draft release is created during preprod deployment, a notification is sent to the internal team channel. This requires the RELEASE_DRAFT_SLACK_WEBHOOK secret.

Format:

📢 Controller Platform Release: controller.5

  • #1551 - Add feature X (@author)
  • #1546 - Fix bug Y (@author)

Approve production deployment

2. Published Release (Company-Wide)​

When a release is published (after production deployment succeeds), a notification is sent to the company-wide channel. This requires the RELEASE_SLACK_WEBHOOK secret.

Format:

📢 Controller Platform Release Published: controller.5

[Full release notes]

Required Secrets​

SecretDescription
RELEASE_DRAFT_SLACK_WEBHOOKWebhook URL for internal team channel (draft releases)
RELEASE_SLACK_WEBHOOKWebhook URL for company-wide channel (published releases)

Both secrets are optional. If not configured, the corresponding Slack notification is skipped.

Publishing a Release (publish-release)​

The publish-release action runs as a step inside the prod deploy job, before the actual deployment begins. It:

  1. Resolves the commit SHA being deployed
  2. Searches for a draft release whose target_commitish matches that SHA
  3. If a matching draft exists: publishes it by release ID (notes are already correct)
  4. If no matching draft exists (clobbered by a later pipeline): generates the release from scratch:
    • Finds the last published release for the platform
    • Computes commits in (last_release, commit] filtered by platform workflow runs
    • Generates changelog and creates a published (non-draft) release
  5. If the released commit is behind HEAD of main, creates a new draft for (commit, HEAD]
  6. Sends a company-wide Slack notification

Releasing a Non-Latest Commit​

When multiple pipelines are queued for prod and you approve an earlier one:

  • The earlier pipeline's draft was likely clobbered by a later prepare-release
  • publish-release detects the missing draft and generates the release from scratch
  • A forward draft is created for commits after the released commit, ensuring the next deploy pipeline has accurate release notes

Workflow Permissions​

The publish-release step requires these permissions on the deploy job:

  • contents: write — to create/publish GitHub releases
  • actions: read — to query workflow runs for commit filtering

Rollbacks​

To roll back a production deployment:

  1. Run the appropriate Deploy & Test workflow manually
  2. Select prod as the environment
  3. Enter the previous release tag as the ref input

Troubleshooting​

No commits found​

The workflow includes all commits from pipeline runs since the last release. If no deployments have occurred, the changelog will be empty.

PR not appearing in changelog​

PRs only appear if:

  • The commit triggered a pipeline run
  • The commit was merged via a PR (direct pushes won't have associated PRs)

Chore not categorized​

Ensure the PR title or description starts with chore: (case-insensitive). The prefix must be at the very beginning.