Skip to main content

Welcome to the cloud control plane CDK

This project is used to deploy the control plane to all marqo cells.

The cdk.json file tells the CDK Toolkit how to execute your app.

This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the .venv directory. To create the virtualenv it assumes that there is a python3 (or python for Windows) executable in your path with access to the venv package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually.

To manually create a virtualenv on MacOS and Linux:

$ python3 -m venv .venv

After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.

$ source .venv/bin/activate

If you are a Windows platform, you would activate the virtualenv like this:

% .venv\Scripts\activate.bat

Once the virtualenv is activated, you can install the required dependencies.

$ pip install -r requirements.txt

Now you need to update the submodules in your git repo by using(more on this below):

$ git submodule init
$ git submodule update

At this point you can now synthesize the CloudFormation template for this code.

AWS CELL

$ cdk deploy --app="python3 ${PWD}/aws_app.py --cell=staging|dev|preprod|prod_cell_1" --require-approval=never

GCP CELL

$ cdk deploy --app="python3 ${PWD}/gcp_app.py --cell=staging|dev|preprod|prod_cell_1" --require-approval=never

The config for each cell is stored in config/${cell_name}.json file. For new production cells, we simply need to define another config file and allowlist the cell_name in the TypedDict in config/config.py

Submodule setup​

This project uses gitsubmodules to fetch the assets for control plane like accounts_api app, index_workflows app and the billing app. Everytime you make a change to any of these repositories you need to create a pull request against this repository to include the new version of the app in the CDK and deploy it to staging.

For instance if there is a change in the cloud_accounts_api_lambda repository, you can add that to the CDK, by executing the following commands:

$ git checkout main
$ git fetch origin
$ git reset --hard origin/main
$ git checkout -b $USER/update_cloud_accounts_api_lambda
$ git submodule update --remote
$ git add submodules
$ git commit -m "update cloud_accounts_api_lambda"
$ git push -u origin $USER/update_cloud_accounts_api_lambda

Useful commands​

  • cdk ls list all stacks in the app
  • cdk synth emits the synthesized CloudFormation template
  • cdk deploy deploy this stack to your default AWS account/region
  • cdk diff compare deployed stack with current state
  • cdk docs open CDK documentation

Enjoy!

Managing Secrets​

Cloudflare API TOKEN​

Our index workflows requires Cloudflare API Token to be injected into containers as we provision and destroy Cloudflare infra on index creation and deletion. The API Tokens are stored in AWS Secrets Manager in the corresponding AWS account. In each environment config, we capture the ARN of this secret in INDEX_WORKFLOWS_CLOUDFLARE_API_TOKEN_AWS_SECRETS_ARN property. ECS Task Definition supports injecting secrets stored in AWS Secrets Manager, this is then made available to the application via an environment variable. See v3_workflows_stack.py.

Note that this is currently done manually by an operator, i.e. we create the API Token in Cloudflare, then manually create the AWS Secret and reference the ARN per config file.