Manual Dev Worker Deployment Guide

This guide explains how to deploy your own personal dev instances of search_proxy and agentic_search Cloudflare Workers.

Prerequisites

1. Install dependencies in each component:

   cd components/agentic_search && npm install
   cd ../search_proxy && npm install

2. Login to Cloudflare:

   npx wrangler login

3. Get the required secrets from your team's secrets manager:

   • GOOGLE_API_KEY - Google AI Studio API key
   • AGENTIC_AWS_ACCESS_KEY_ID - Service account AWS credentials for DynamoDB
   • AGENTIC_AWS_SECRET_ACCESS_KEY - Service account AWS credentials for DynamoDB

Config Files

Create these config files (replace raynor with your name):

components/agentic_search/wrangler.dev.initial.toml

Initial deploy without search_proxy binding (to break circular dependency):

compatibility_flags = ["nodejs_compat"]
account_id = "3a8e992c9f607dcb3b401878264df92e"
main = "src/index.ts"
compatibility_date = "2024-09-23"
name = "dev-agentic-search"

[observability]
enabled = true

[[migrations]]
tag = "v1"
new_sqlite_classes = ["ConversationSqlDO"]

[env.dev-raynor]
name = "dev-raynor-agentic-search"
upload_source_maps = true

[env.dev-raynor.vars]
ENV = "staging"
FULL_ENV = "staging-raynor"
AWS_REGION = "us-east-1"
AGENTIC_CACHED_QUERIES_TABLE_NAME = "staging-AgenticCachedQueriesTable"
AGENTIC_USER_DATA_TABLE_NAME = "staging-AgenticUserDataTable"
TENANT_HELP_CENTER_TABLE_NAME = "staging-TenantHelpCenterTable"

[[env.dev-raynor.r2_buckets]]
binding = "AGENTIC_IMAGES_BUCKET"
bucket_name = "dev-raynor-agentic-images"

[[env.dev-raynor.durable_objects.bindings]]
name = "CONVERSATION_DO"
class_name = "ConversationSqlDO"

components/agentic_search/wrangler.dev.<yourname>.toml

Full config with search_proxy binding:

compatibility_flags = ["nodejs_compat"]
account_id = "3a8e992c9f607dcb3b401878264df92e"
main = "src/index.ts"
compatibility_date = "2024-09-23"
name = "dev-agentic-search"

[observability]
enabled = true

[[migrations]]
tag = "v1"
new_sqlite_classes = ["ConversationSqlDO"]

[env.dev-raynor]
name = "dev-raynor-agentic-search"
services = [
  { binding = "SEARCH_PROXY_WORKER", service = "dev-raynor-ecom-api" }
]
upload_source_maps = true

[env.dev-raynor.vars]
ENV = "staging"
FULL_ENV = "staging-raynor"
AWS_REGION = "us-east-1"
AGENTIC_CACHED_QUERIES_TABLE_NAME = "staging-AgenticCachedQueriesTable"
AGENTIC_USER_DATA_TABLE_NAME = "staging-AgenticUserDataTable"
TENANT_HELP_CENTER_TABLE_NAME = "staging-TenantHelpCenterTable"

[[env.dev-raynor.r2_buckets]]
binding = "AGENTIC_IMAGES_BUCKET"
bucket_name = "dev-raynor-agentic-images"

[[env.dev-raynor.durable_objects.bindings]]
name = "CONVERSATION_DO"
class_name = "ConversationSqlDO"

components/search_proxy/wrangler.dev.<yourname>.toml

main = "src/index.ts"
compatibility_date = "2024-09-23"
compatibility_flags = ["nodejs_compat"]
account_id = "3a8e992c9f607dcb3b401878264df92e"
name = "dev-ecom-api"

[observability]
enabled = true

[env.dev-raynor]
name = "dev-raynor-ecom-api"
routes = [{ pattern = "dev-raynor.dev-marqo.org", custom_domain = true }]
services = [
  { binding = "MARQO_WORKER", service = "global-worker" },
  { binding = "AGENTIC_SEARCH_WORKER", service = "dev-raynor-agentic-search" }
]
kv_namespaces = [
  { binding = "KV", id = "b89e4485d04b480fb3a6e1cfc9920c1b" },
  { binding = "KV_QCFG", id = "b89e4485d04b480fb3a6e1cfc9920c1b" },
]
upload_source_maps = true

[env.dev-raynor.vars]
ENV = "staging"
FULL_ENV = "staging-raynor"
SHOPIFY_API_KEY = "e8a915d783e8b9756174faa712e9361e"
ADMIN_SERVER_BASE_URL = "https://staging-admin.ecom.marqo-staging.com"
METRICS_QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/468036072962/staging-EcomMetricsQueue"
SHOPIFY_APP_SECRET = "your_app_secret_here"
SHOPIFY_SHOP_TO_CLIENT_ID = "{}"
SHOPIFY_CUSTOM_APP_SECRETS = "{}"
AWS_REGION = "us-east-1"

R2 Bucket Setup (Required for Image Uploads)

/api/v1/indexes/:index/agentic-search/images stores uploaded images in R2. Before deploying agentic_search, create your per-dev bucket and apply a 1-day lifecycle rule.

cd components/agentic_search

# Use your worker prefix
BUCKET_NAME="dev-raynor-agentic-images"

# Create bucket (safe to run again if it already exists)
npx wrangler r2 bucket create "${BUCKET_NAME}" || true

# Ensure 1-day expiry for uploaded images under v1/*
npx wrangler r2 bucket lifecycle remove "${BUCKET_NAME}" --name expire-images-1d >/dev/null 2>&1 || true
npx wrangler r2 bucket lifecycle add "${BUCKET_NAME}" expire-images-1d v1/ --expire-days 1 --force

Deployment Steps

The workers have a circular dependency (agentic_search ↔ search_proxy), so we deploy in 3 steps:

# Step 1: Deploy agentic_search WITHOUT search_proxy binding
cd components/agentic_search
npx wrangler deploy --config wrangler.dev.initial.toml --env dev-raynor

# Step 2: Deploy search_proxy (can now reference agentic_search)
cd ../search_proxy
npx wrangler deploy --config wrangler.dev.raynor.toml --env dev-raynor

# Step 3: Re-deploy agentic_search WITH search_proxy binding
cd ../agentic_search
npx wrangler deploy --config wrangler.dev.raynor.toml --env dev-raynor

Setting Secrets

Secrets are stored securely in Cloudflare and prompted interactively:

# agentic_search secrets
cd components/agentic_search
npx wrangler secret put GOOGLE_API_KEY --config wrangler.dev.raynor.toml --env dev-raynor
npx wrangler secret put AGENTIC_AWS_ACCESS_KEY_ID --config wrangler.dev.raynor.toml --env dev-raynor
npx wrangler secret put AGENTIC_AWS_SECRET_ACCESS_KEY --config wrangler.dev.raynor.toml --env dev-raynor

# search_proxy secrets
cd ../search_proxy
npx wrangler secret put AWS_ACCESS_KEY_ID --config wrangler.dev.raynor.toml --env dev-raynor
npx wrangler secret put AWS_SECRET_ACCESS_KEY --config wrangler.dev.raynor.toml --env dev-raynor

To list existing secrets:

npx wrangler secret list --config wrangler.dev.raynor.toml --env dev-raynor

Updating After Code Changes

After making code changes, just redeploy:

# Update agentic_search
cd components/agentic_search
npx wrangler deploy --config wrangler.dev.raynor.toml --env dev-raynor

# Update search_proxy
cd ../search_proxy
npx wrangler deploy --config wrangler.dev.raynor.toml --env dev-raynor

Testing

Your worker will be available at:

• https://dev-raynor.dev-marqo.org/

Example test:

curl -X POST "https://dev-raynor.dev-marqo.org/api/v1/indexes/YOUR_INDEX/agentic-search/chat-suggestions" \
  -H "Content-Type: application/json" \
  -H "x-marqo-index-id: YOUR_INDEX_ID" \
  -d '{
    "documentId": "YOUR_DOC_ID",
    "sessionId": "test-session",
    "userId": "test-user"
  }'

Viewing Logs

npx wrangler tail dev-raynor-agentic-search
npx wrangler tail dev-raynor-ecom-api

Cleanup

To delete your workers:

npx wrangler delete dev-raynor-agentic-search
npx wrangler delete dev-raynor-ecom-api
npx wrangler r2 bucket delete dev-raynor-agentic-images