Skip to main content

Runbook: marqo_api_env_var_hotfix

This runbook covers steps to hand-patch a Marqo API environment variable on a live index. The change updates the marqo-api-config ConfigMap and rolls over the marqo Deployment so the new pods pick up the new env.

Steps​

1. Get admin permissions via Escalator​

Request admin access through Escalator:

https://escalator.marqo-staging.com/

2. Copy admin credentials to local terminal​

Copy the admin credentials from Escalator and export them in your terminal.

3. Get EKS cluster credentials​

Substitute the cell that hosts the index (e.g. cell1, cell2). If unsure, look up the cluster on the per-index Grafana dashboard.

aws eks update-kubeconfig --region us-east-1 --name cell2-MultitenantEKSCluster

4. Find the index namespace​

Index namespaces are labeled with system_account_id and index_name. Resolve the namespace from those labels:

n=$(kubectl get ns \
  -l "system_account_id=<system_account_id>,index_name=<index_name>" \
  -o jsonpath='{.items[0].metadata.name}')
echo "$n"

Confirm the result is non-empty and points at exactly one namespace. If empty, the index is not on this cluster — check the per-index Grafana dashboard for the correct cell and re-run step 3.

5. Inspect the current ConfigMap​

The Marqo API reads its environment from a ConfigMap named marqo-api-config.

kubectl get configmap marqo-api-config -n "$n" -o yaml | less

Note the current value of the env var you intend to change so you can revert if needed.

6. Patch the env var​

Use kubectl edit for an interactive change, or kubectl patch for a scripted one.

Interactive:

kubectl edit configmap marqo-api-config -n "$n"

Scripted (set a single key under data):

kubectl patch configmap marqo-api-config -n "$n" \
  --type merge \
  -p '{"data":{"<ENV_VAR_NAME>":"<new-value>"}}'

Verify the patch landed:

kubectl get configmap marqo-api-config -n "$n" -o jsonpath='{.data.<ENV_VAR_NAME>}'; echo

7. Roll over the Marqo Deployment​

Pods read the ConfigMap at startup, so the patch only takes effect after a rollout:

kubectl rollout restart deployment marqo -n "$n"

Wait for the rollout to complete:

kubectl rollout status deployment marqo -n "$n" --timeout=10m

8. Rollback​

To revert, re-run step 6 with the original value, then step 7 to roll over again.

If the workflow already overwrote your change, the canonical config is back in place and no action is needed.